OLS Day 4: Kroah-Hartman’s Keynote Address


Author: David 'cdlu' Graham

The fourth and final day of the 2006 Ottawa Linux Symposium saw the annual tradition of the closing keynote address, this year by Greg Kroah-Hartman, introduced by last year’s keynote speaker, Dave Jones, and the announcement of the next year’s speaker.

Dave Jones introduced Greg Kroah-Hartman of Novell’s SUSE Laboratories, noting in his introduction, among other things, that in his analysis of kernel contributors, sorted by volume, Greg was high on the list. He is responsible for udev, for which we should beat him later, Jones noted, adding Kroah-Hartman has spent two years working on a crusade to remove devfs from the filesystem (to great applause). Jones described Kroah-Hartman as approachable, and highly diplomatic, calling his approach to kernel communications diplomacy at its best. He punctuated this with a photo of Kroah-Hartman sitting at a table with his middle finger raised in a most diplomatic pose.

Kroah-Hartman began by noting that he is sure his daughter appreciates the photo.

Kroah-Hartman’s keynote was entitled “Myths, Lies, and Truths about the Linux Kernel”.

He started with a quote: “My favorite nemesis is that plug and play is not at the level of Windows.” Surely, he said, such a quote must come from someone not educated in the ways of Linux. It must be from someone unfamiliar with the system and its progress. He went on to the next slide, and the quote became attributed. It was said by Jeff Jaffe, the CTO of Novell. Surely, then, continued Kroah-Hartman, it must have been said a long time ago! Slide forward: Jaffe said it on April 3rd, 2006.

Linux, said Kroah-Hartman answering the charge, supports more devices out of the box than any other operating system ever has. Linux is often even ahead of the pack, being the first operating system to implement both USB2 and bluetooth.

Linux, he continued, supports more different hardware platforms than any other operating system. Someone shouted from the audience, ‘what about NetBSD!’ to which Kroah-Hartman retorted that Linux blew away NetBSD about three years ago.

Everything from cell phones to radio controlled airplanes to 73% of the top supercomputers in the world run Linux, he said. Linux scales.

Mr. Jaffe, he commented, should try his own product. We are doing something really good, he continued.

The “kernel has no obvious design”, or roadmap, Kroah said, citing the next fallacy about Linux he intended to attack. Marketing departments like roadmaps and design paths, he said, but Linux does not provide them. Linux, he said, has created something no-one else ever has.

“Open Source development violates almost all known management theories.” – Dr. Marietta Baba, Dean of the Department of Social Science at Michigan State University, he quoted.

He posted a slide showing a picture of a painting of a naked man from what appeared to be a religious context next to a squid-like animal with a number of weird anomalies, and a quote: “Linux is evolution, not intelligent design.” – Linus Torvalds.

Linux started off being (barely) supported on a single processor, Kroah-Hartman noted. Then someone offered to fix it to run on another processor, and the process of evolution was well under way. Linux evolves by current stimuli, not by marketing department requirements, he said.

The only way to help the evolution, he continued, is to provide code to the kernel. Ideas without code backing them won’t get far.

Linux implemented the POSIX standard six or eight years ago, he said. The evolution of the kernel is fast now, with around 6000 patches per major release. It is changing faster than ever, and becoming more stable than ever.

He moved on to the next myth, paraphrasing a common one: “the Kernel needs a stable API or no vendors will make drivers for Linux.” For those who don’t understand it, he said, an API is how the kernel talks to itself. He suggested reading Documentation/stable_api_nonsense.txt in the kernel source directory for more information on the topic.

Linus doesn’t want a stable API, he said. The USB stack, for one, has been re-implemented three times so far. Linux now has the fastest USB stack available, limited only by the hardware. Linux is lean and complex.

Windows, too, has rewritten the USB stack 3 times, he noted, but all three of them have to stick around in the system to support the various and uncontrolled old independent drivers kicking around. Linux’ native support for drivers means that independent drivers are not a problem and therefore that the API can be rewritten as needed without keeping older versions kicking around in the kernel. Windows has no access to the drivers and cannot adjust the API as a result.

The next myth he poked a hole in is the notion that, “my driver is only for an obscure piece of hardware. It would never be accepted into the mainline kernel.” We have an entire architecture, Kroah-Hartman countered, being used by just two users. There are lots of drivers, he said, that have but one user.

A company contacted Kroah-Hartman, he said, to find out about putting a driver into the kernel for an obscure task that they needed to do. The driver was put in, and several other companies that also had to do similar tasks no longer needed to maintain their own versions. The contribution became useful on a more widespread basis in a way that could not have been foreseen and is now used to support thousands of devices. Just get your code in, he said, people might actually use it.

He went on to address the problem of closed source and binary kernel modules. Every IP lawyer he has talked to, he said, regardless of who they work for, has agreed on one fundamental point: “Closed Source Linux kernel modules are illegal.” The lawyer’s can’t say it in public, he said, but he can.

He suggested not asking legal questions on the Linux Kernel mailing list, asking: would you ask the list about a bump on your elbow?

Kroah-Hartman explained how closed source modules included in Linux distributions cause problems and prevent progress, holding back the kernel included with the distributions. Closed source Linux kernel modules are unworkable, he said.

Companies that have intellectual property they say they want to protect, he said, should not use Linux. When you use Linux, you should follow its rules. You are saying that your IP, he said, trumps the entire Linux community, that you are more important than everyone else. Closed source Linux kernel modules are unethical, he said.

He suggested to companies that they read the kernel headers for who owns the copyright on various parts of Linux. He noted that you will find companies like AMD, Intel, and IBM represented there. Do you really want to skirt under these companies’ lawyers?

Novell, on February 9th, 2006 said in an official policy statement that it will no longer ship non-GPL kernel modules. It is as good a statement as is possible for any such company, he said, noting the lack of reference to the fundamental legal reason for avoiding it.

Someone shouted out asking about Nvidia. Nvidia, he said, and ATI, and VMWare all violate the GPL. But they do it cleverly. They write their code against the kernel source, but they don’t link it, the part that violates the GPL. They force the end use to do that, preventing them from redistributing their builds. VMWare, he commented, is not open source.

He moved on to the next myth: it is hard to get code into the main kernel tree.

If there are 6000 changes per release, someone is getting it in, he said. All that you need to do is read the Documentation/HOWTO file in the kernel tree and know what you are doing.

There are a number of ways to start working on the kernel, he explained. The first and easiest is to check out the Kernel Newbies project. It is available as a wiki and webpage at kernelnewbies.org. The second way, he said is to join the Kernel Newbies mailing list. He said it is virtually impossible to ask a stupid question on that list. Just read the recent archives so you don’t ask a question that has just been asked. The third and final way is to get on the Kernel Newbies IRC channel. He said there are around 300 users there and the channel is usually quiet, but not to worry, people will generally answer your questions.

But when seeking help, he cautioned, be prepared to show your code. People are not inclined to help people who are working on closed source code.

The next step up from Kernel Newbies is the Kernel Janitors project. This is a list of things that need doing. Check the list, see if you can knock some items of it. Getting a kernel patch accepted is a good feeling, he said. A lot of people started this way.

The next step up is to join the Linux Kernel Mailing List, which has around 400 to 500 messages a day. Don’t feel bad about not reading every message, he said, everyone filters. The only person in the world, he said, who reads all of the messages is Andrew Morton. Subscribe, and ask questions there, he suggested.

There are not very many people, he admitted, who review the code that comes in. But when someone reviews your code and gives you feedback, they are right. They are not the bad people, he warned, the people submitting the bad code are. Kroah-Hartman said he tried it for a week. He said it made him grumpy.

He suggested that people wishing to contribute should spend a few hours a week reading existing code. You must learn to read music, he analogized, before you can write it.

If you can’t contribute by writing code, what can you do, he asked.

It is not possible to do comprehensive regression tests on the Linux kernel, he said. You cannot test what happens if you add this device and remove that device in this order over this time. It just doesn’t work. The best test, he said, “is all of you”. Test Linus’ nightly snapshots, he urged.

If you find a problem, post it to bugzilla.kernel.org. Then bug him until he feels bad, so that it gets fixed.

Also, he said, try and test the -mm kernel tree.

In conclusion, he said, Linux supports more devices than anyone else. Linux progresses by evolution, not design. Closed source drivers are illegal. Linux can use help with reviews and testing.

And most importantly, he finished, total world domination is proceeding as planned.

With that, he moved on to taking questions.

The first question stated that the timeliness of device support is as important as the number of devices supported.

He responded that in order for drivers to come out quickly, it is important for hardware vendors to get involved.

The next person up to the microphone commented that if someone wants to learn how something works in the kernel, write a design document for it.

Kroah-Hartman responded that design documents need to go directly into the source code as it is the only way they will be kept up to date. There is a movement afoot, he said, to get OSDL to hire a full time documentation guy. He suggested that anyone present who works for a member of OSDL suggest to their employers that they contribute funds for this purpose.

Alan Cox came up to the mike to ask a question. Is Microsoft the Borg as the media likes to portray it, he asked, or is it really Linux that is the Borg?

Kroah-Hartman answered that Microsoft is buckling under its own load. By size, Microsoft is the Borg, but by function, it is Linux. But to him, he said, it is not a matter of us versus them. He doesn’t mind the competition.

To sum up

Following the Greg Kroah-Hartman’s keynote address, the annual door-prize draw took place. This year, the CELinux forum offered a development platform Linux PVR from Philips, and 3 Linux-based Nokia 7700s. Red Hat contributed 2 laptop bags and 3 red hats to the draw, and IBM threw in a couple of loaded Apple Powermac G5s. The hats were distributed by Alan Cox, with Greg Kroah-Hartman’s young daughter picking the numbers.

Conference co-organizer Craig Ross closed out the formal part of the conference with a series of announcements.

The first was that a US passport had been found, and would the owner please come forward to claim it. No-one did, but it led to lots of laughter.

Ross announced next year’s keynote speaker will be SCSI maintainer James Bottomley, without naming him. He held up a bow-tie and suggested that all attendees should wear one next year, as the charismatic Bottomley is known for, in his honor.

Ross continued, reminding all attendees that the closing reception at a nearby pub can only be entered with the help of official conference id. Guests and spouses can come if they have an event pass, available through registration, Ross said. Girlfriends, wives, and family met along the way between the conference center and the pub would not be welcome. People may not sleep in the fountain at the pub, and should not attempt to climb the fake balcony inside the pub. If you do misbehave in public, he grinned, at least take off your conference id!

During closing announcements, conference organizer Craig Ross asked the assembled crowd how many were attending OLS for the first time. The number of hands raised in response to this question was quite low, certainly well under a quarter of attendees. It was the fourth OLS that I attended and I am eagerly anticipating next year’s.

As for who came to the conference, contrary to what one might expect at an event called Ottawa Linux Symposium, the number of long-haired, unshaven, sandal-wearing geeks is actually very low, though the number of people meeting any one of these criteria on its own is somewhat higher. Judging by the name-tags, the number of people attending on their own tab as opposed to being sponsored by their employer is quite low. Most people, though certainly not all, have a company name on their tags. What conclusions can be drawn from this I leave to you to decide.

This year’s OLS saw approximately 128 sessions, BOFs, and formalized events, up sharply from 96 a year ago. Sessions started at 10am and, except for a break for lunch, went on until 9 pm most days split between 4 session halls. I took 43 pages of handwritten notes, up one from last year, from attending 26 sessions, up by three over last year. Of these, I covered a mere 12 in these summaries. Once again, I hope you enjoyed this taste of OLS and I hope to see you there next year!


  • Linux