Report from FreedomHEC

SEATTLE — Every year, Microsoft holds a Windows Hardware Engineering Conference (WinHEC). This year, WinHEC was immediately followed by a small, informal two-day “unconference” dubbed FreedomHEC. Though not as large as WinHEC, FreedomHEC was a great chance for developers to get up close and personal with Linux kernel developers.

FreedomHEC was conceived by Don Marti, the former chief editor of Linux Journal magazine. It was a two-day mini-conference that started the day after WinHEC. Like WinHEC, it was held in Seattle.

Thanks to Pogo Linux, one of the sponsors, FreedomHEC was held in the Columbia Center skyscraper in downtown Seattle. With 76 floors, this is the tallest building West of the Mississippi; the top two floors are taken up by the Columbia Club. FreedomHEC was held on the 76th floor, in one of the Columbia Club rooms, which provided an incredible view for attendees.

First day

About three dozen people were present for the first day. The day opened with a discussion of what topics people wanted to discuss; Marti moderated, and quickly mapped out a schedule for the day.

Greg Kroah-Hartman, one of the top Linux kernel maintainers, was up first to explain sysfs and udev. Kroah-Hartman started out by explaining the /proc filesystem and its history; /proc has been kind of a shaggy mess, with a lot of files being stored there that don’t belong in /proc.

Now everything that is not process-related is being moved out of /proc and into the sysfs filesystem, /sys. Files in /sys are supposed to be trivial to parse. Ideally, they should have only one value on one line in the file, ASCII characters only. Kroah-Hartman emphasized that last point. “One value per file! If you don’t remember anything else, remember that. Please!”

Udev is a flexible dynamic hardware management system that lets the kernel expose the details of your hardware, and is easy to work with from user-space programs. Kroah-Hartman also demonstrated a “device manager” GUI system that was written in Python.

Next up was Randy Dunlap, who works at the Open Source Development Lab (OSDL) and is another top Linux kernel maintainer. Dunlap’s talk, “Linux Social Engineering,” was on how to work with the Linux kernel development community, which does many things differently than a corporation would. It’s not difficult to get contributions into the Linux kernel, if you understand how to work with the community. The process for Linux drivers isn’t really harder than for Windows drivers, it’s just different.

Greg Kroah-Hartman – click to enlarge

After lunch came a talk by James Bottomley, the Linux kernel SCSI driver maintainer, on SCSI in Linux and how the device driver model works. Linux has support for a broad variety of SCSI devices. He spoke on the three major sources of bugs: kernel bugs, device bugs, and transport bugs. Transport bugs, which affect communication between devices, are the worst. For example, you can make a SCSI cable unreliable just by stepping on it, and the cable doesn’t look any different; now you will get bugs no one else can duplicate. He says he has a large collection of cables that he has degraded (cutting little slices through the insulation, for example) so he can test the SCSI layer’s ability to cope with transport problems.

The next talk was on rockets, and the use of Linux in rocketry. The talk was presented by Sarah Bailey and Jamey Sharp of the Portland State Aerospace Society (PSAS) from Portland State University in Portland, Oregon. Many people build and fly rockets, but few build onboard flight computers for them; the rocketeers from PSAS are the exception. PSAS’s goal is to develop a rocket capable of putting a “nanosatellite” (about the size of a soda can) in orbit around the Earth. The primary problem is not so much to be able to build a rocket, but to build a rocket with accurate enough navigation and steering to fly a trajectory that will result in a stable orbit.

The most recent rocket launch successfully reached an altitude of 21,000 feet (6,400 meters) but unfortunately the landing was somewhat less successful. The black powder charges to open the landing parachutes burned only partially, probably because of the cold and low pressure at that altitude. The parachutes didn’t deploy, and the rocket crashed while going more than 560 miles per hour (900 km/hour). Fortunately, the navigation system, running open-source software, kept broadcasting right up to the crash, so they didn’t need to search for the crash site; they just drove right to it and found the remains of their rocket. PSAS will build another rocket and fly it in the future; check their web site to find out more.

Randy Dunlap – click to enlarge

The final session of the first day was a question and answer session with Kroah-Hartman on getting drivers accepted into the Linux kernel. It was a lively session, touching on many areas of kernel development. Kroah-Hartman assured the attendees that kernel developers are interested in their drivers. “People always say, ‘Oh, they won’t want my driver; we only ship a few hundred devices per year that use it.’ I always tell them that we have device support in the Linux kernel for hardware with only one or two known users. Really, we’ll take your driver!”

Second day

Again about three dozen people showed up, but some people who had been present the previous day were gone, and some new people were present. My estimate is that about 42 people in total attended FreedomHEC. The day started with a quick discussion of what topics people wished to discuss.

First up was the session I had been eagerly waiting for: Kroah-Hartman’s “Write Your Own Linux Device Driver” session. We installed his driver source code on our Linux laptops and followed his directions. Really, a more accurate title for the session would be “Compile and Understand a Simple Linux Device Driver.”

We didn’t really write any code, but we did build, install, and test drivers of increasing complexity and utility. The goal was to finish with a driver for a USB temperature probe; the driver would create a file in /sys, and reading from the file would give the temperature value returned by the USB thermometer. The first driver was a simple “Hello, world” driver that placed a hello message in the kernel log. The next driver created a file in /sys. Subsequent versions detected and initialized the thermometer, and the final version put it all together. The driver code was straightforward and easy to understand. This session was a great way to learn how kernel device drivers work.

Before we tried out our own drivers for the USB thermometer, we unloaded the official Linux driver that manages it: ldusb. Kroah-Hartman finished up by saying a few words about the ldusb driver. He told us that this one small driver supports data acquisition devices from four different manufacturers, and it’s a solid success. The manufacturers are happy that they can just add a few lines of code to the driver and their new devices are supported; they don’t have to do very much work to be supported under Linux. Kroah-Hartman also discussed a few simple techniques for debugging Linux drivers.

After lunch was a talk on graphics hardware and the future of X11, given by PSAS’s Sharp. Sharp is not only a rocketeer with PSAS, but has also spent the past five years working on the xcb library, intended to replace the venerable Xlib.

Sharp says that the X server does far too much with the hardware. Historically, the Linux kernel hardware support wasn’t sufficient for graphics devices, and instead of improving the kernel hardware support, X developers managed the graphics hardware from inside X. The trend now is to take hardware management out of X and put it in the kernel where it belongs.

X.org is improving at a rapid pace. It’s easier than ever to build X, and to add new devices to X. New hardware acceleration libraries are being developed, and slick eye-candy effects will be available in the major Linux distributions soon. The X developers are also working on a hotplug system. For example, if you set up a laptop and plug in a projector, it should be autodetected and “just work.”

The next talk was by Bottomley, who not only is the lead SCSI device driver maintainer but is also the CTO of a company called SteelEye Technology. For this talk Bottomley says he was wearing his CTO hat, not his SCSI maintainer hat. “When I go to trade shows, I’m the CTO. When I go to Linux conferences, I’m the SCSI maintainer. This is the first time I’ve been both at the same event.” His talk was on “Why Linux can help Make Windows Sales (and Vice Versa)”.

Bottomley pointed out that almost all companies have a mix of Windows and Linux. They might have Linux on some servers and Windows on the desktop, they might have Unix in a research lab but use Exchange to schedule workers. Bottomley says that a company that supports Windows and Linux is in a better position to support a mixed company. “Companies trust suppliers who respect their choices. Any company who goes with a bigoted sales message tends to get kicked back.”

He also says that offering a Windows product can help sell a Linux product. A company might be a bit nervous about adopting a Linux-based technology, and the availability of the Windows-based version can lessen the perceived risk. “In practice, they never switch from the Linux version to the Windows version; but knowing that they could can help close the sale.”

Steeleye Technologies sells a product called LifeKeeper Data Replication (LKDR), and has a version for Linux and one for Windows. On a T3 line, the Linux product would push 90% of the maximum bandwidth of the line; the Windows product, a completely different code base, only used 20% of the bandwidth. To increase the Windows port’s throughput, Bottomley put a Windows developer and a Linux developer together to solve the problem. “The APIs are totally different, but the skills transfer,” he told us. The expertise of the Linux developer was instrumental in revamping the Windows product, and now the Windows product can also push 90% of the maximum bandwidth of the T3 line.

The last discussion was an informal question and answer session with Kroah-Hartman. Many topics were covered, including a discussion of why Linus Torvalds’s Git source control system works so well for kernel development.

At the end of the day, Marti led a brief discussion of the future of FreedomHEC. Next year, expect FreedomHEC to be much bigger. FreedomHEC was conceived a relatively short time before WinHEC this year, and as a result few people were able to change their plans and attend.

With a full year of lead time, FreedomHEC 2007 will be announced earlier, and people can plan their travel up front to include both WinHEC and FreedomHEC. No dates have been set yet for next year’s FreedomHEC, but it will most likely once again start the day after WinHEC.

I learned a lot at FreedomHEC and I had a great time. It was also really neat to see the faces of famous kernel developers. If you will be at WinHEC next year, or just have an interest in kernel development, consider attending FreedomHEC as well.