Fireside Chat: GKH Talks Licensing, Email, and Aging Maintainers


No one aside from Linus Torvalds has more influence or name recognition in the Linux Kernel project than Greg Kroah-Hartman. More commonly known as GKH, the ex SUSE kernel developer and USB driver maintainer is now a Linux Foundation Fellow and the full-time maintainer of the -stable Linux branch and staging subsystem, among other roles. In a recent Fireside Chat with Kroah-Hartman at Embedded Linux Conference Europe, Tim Bird, Chair of the Architecture Group of the Linux Foundation’s CE Working Group, described him as the hardest working person he knows.

Not only does Kroah-Hartman review an endless series of kernel patchsets and explore new directions for Linux — he attends almost every Linux-related conference in the world, said Bird. This year, GKH will only reach about 100,000 miles of travel, down from last year’s 140K. This slacker schedule may in part be due to recently moving with his family from the Pacific Northwest to Paris, France.

But why Paris, asked Bird. “The food and wine are good,” said GKH. “My daughter thinks I’m having a midlife crisis. I claimed she’s on the coattails of it – she worked at LinuxCon last week.” For the record, GKH said his main goal in Paris was to collaborate with researchers at Pierre Marie Curie University on applied research in OS and system design.

The Keynote Fireside Chat at ELC Europe, held Oct, 11-13 in Berlin, focused primarily on two issues: whether older kernel maintainers should hand their jobs over to younger developers, and how to best bring open source scofflaws into compliance (see below). Meanwhile, here are a few other edited quick takes from GKH about issues ranging from patch review technologies to the role of Linux on microcontrollers.

On whether Linux has a role on microcontrollers…

GKH: A student of mine got Linux running on a Cortex-M3 with 4MB, which is great for Linux, but 2MB is pushing it. At LinuxCon in Toronto, some of us were drunk and found ways that we think we can get the kernel into 512KB — but it won’t do anything. Stripping Linux down for these chips would be awesome — I’d love to do that. But there are already so many good OSes for this. Zephyr is now a good alternative to Nuttx.

On whether email still makes sense as the basis for patch review, vs. say, Gerritt…

GKH: There’s nothing else that’s better, faster, or more widely used around the world than email. It’s free, and it works great for people who can’t use GUIs, who have intermittent Internet access, or who don’t speak English as their first language. You can also use tools on top of email such as Patchwork, which can tie into continuous integration and testing, and that’s what people use Gerrit for.

On whether there are too many aging kernel developers and maintainers…

GKH: Yes, we are getting old, but it beats the alternative. Age is a dual factor. David Miller has maintained the network stack for 21 years, and I’ve been maintaining USB since the 1990s. That’s knowledge, depth, and information. When USB 3 came out, Microsoft put a whole new team on it. We had one really good developer – Sarah Sharp – implement USB 3 for us in half the time. So knowledge is good.

But we also work on getting in new developers. We work with Outreachy and Summer of Code and lots of universities to bring in interns, some of whom are younger than Linux. We have tons work – if people want to do it. We have subsystems that nobody maintains. The Parallel port subsystem hadn’t had a maintainer for over 12 years because no one wanted to do it. A new developer came in and converted the Parallel ports to the driver model, and he did great, and he got a job. So youthful ignorance and blind ambition are great. I was there.

On the dos and don’ts of open source licensing compliance…

GKH: Amazon is an example of a company that perfectly complies with the license, but all they do is throw this random tarball up on some website. It’s the old SourceForge, ‘Let’s bury it somewhere’ crap, and that’s not good. It costs them money, and it’s a pain for us. So we just ignore them. They’re not receiving any of the benefits of being part of the community.

The biggest problem we have is the dumping of these huge patchsets. Look at Qualcomm’s 2.5 million lines of code crap in a git repository – okay, so it’s getting better, now it’s only 1.5 million lines. That’s crazy — it’s impossible to mine. They say, ‘Ooh, our new chips are based on kernel 3.18.’ Good job, guys. You’ll reach 4.4 just in time for me to obsolete that kernel. So all these embedded devices are running these crazy patchsets. It’s Linux ‘like’. There are entire SoCs and graphics drivers that nobody’s ever seen or touched.

On whether the Linux community is tough enough in enforcing open source compliance…

GKH: There’s been a lot of discussion lately about GPL enforcement. People have claimed that if we don’t enforce the GPL, it’s the same as BSD. That’s flat out false. Yes, people violate our license. That always happens. But it’s gotten a lot better. Back in the 1990s, people were shipping closed source Ethernet, SCSI, and controller drivers. It was crazy.

Intel used to be one of the biggest GPL violators, and now they’re our biggest supporter. And that happened due to us working with them. If you go into a company with lawyers, walls are going to come down, and you’re going to alienate everybody. It’s better if your developers contact the developers inside the company and say ‘What can we do to help you get your code merged into the kernel?’

Look at Microsoft, which is now an active contributor to Linux. That happened because Microsoft’s customers wanted open source, and because I knocked on their door nicely and asked if we could help with their kernel code. Their initial code dump was 12,000 lines of crap. We added it to the kernel staging directory, and after a year when we finally merged it out, it was only 7,000 lines, and supported four new device types. We showed them that if they worked with us, we could make their code smaller, make their stuff run better, and make their customers happier.

We’re in it for the long haul. We don’t just want an instant code dump — we want them to become part of our community. You may want to get one device working, but what you really want is to get them to join the community. The only way we’re going to survive is if we bring in more people. Make them realize that working with the community saves them time and money. One day, they will become so reliant on Linux that they have to invest. They will turn to their partners and ask why they aren’t doing the same. This has been proven again and again.

Watch the complete fireside chat with Greg Kroah-Hartman below:

Watch all 125+ sessions from ELC + OpenIoT Summit covering the latest on embedded Linux development and open source IoT. Watch now >>

Embedded Linux Conference + OpenIoT Summit North America will be held on February 21 – 23, 2017 in Portland, Oregon. Check out over 130 sessions on the Linux kernel, embedded development & systems, and the latest on the open Internet of Things. readers can register now with the discount code, LINUXRD5, for 5% off the attendee registration price. Register now>>