March 28, 2006

Interview: Theo de Raadt of OpenBSD

Author: Manolis Tzanidakis

Theo de Raadt is the project leader for OpenBSD, a Unix-like operating system. We spoke with Theo about the upcoming release of OpenBSD, 3.9, the financial state of the project, and about companies that profit from free software without contributing back.

NewsForge: Hello Theo. Could you tell us a few things about yourself and your involvement in the OpenBSD project?

Theo de Raadt: I have been the project leader for OpenBSD now for more than 10 years, and along the way I have had some good adventures with the developers in the group. We've developed some side projects as well, which are heavily used by everyone in the Unix world, such as OpenSSH.

NF: How many developers contribute to OpenBSD at the moment?

TdR: Inside the project, the count has slowly grown. It was 40 in the early years, and now it is about 80. Of course, that is just counting internal developers. There are many more people on the outside who send us bug reports, fixes, or new code contributions. We also are able to take pieces of code from other sources if they are sufficiently free. But since internal developers have more responsibility -- they have really maintained the areas they are in -- the people on the outside really have an easier job, and should not envy the people on the inside. Instead, they should find a bug, write a fix, and send it in. When someone on the outside sends us many (good) bug fixes, we invite them to become a developer.

NF: You regularly organize events called hackathons. What exactly is a hackathon?

TdR: This is something we started many years ago. A bunch of us would fly to one location (typically before or after a conference) and we would sit down and code. These events really are about getting tasks done; there is very little chatter, as we already know basically what needs to be done. They are not meetings, no one presents talks, nor are they so-called summits. They are for taking action in the source tree, knowing that the guy you need to ask a question of really quickly is sitting at a table a meter away.

NF: OpenBSD is considered one of the most secure operating systems currently available. What approaches do you take towards security?

TdR: We've had 10 years of nearly fanatical devotion to anything which can make OpenBSD more secure. A very important part of that is that we have not been afraid to completely overhaul anything even if it breaks backward compatibility. Secondly, when we have found a flaw in any part of the system we have assumed that the same mistake was made elsewhere, and gone on a hunt to fix them all. Thirdly, we have developed and incorporated a collection of methods that make software flaws very difficult to attack.

The important detail is that in all three of these areas we have not only been fanatical, but pretty much first. Other vendors are not treating their source code the way we treat ours -- with distrust, knowing that we should always actively churn it, so that it can slowly evolve into a better state.

Someone on wikipedia has gone through a lot of effort to identify some of our security efforts, and there is the Exploit Mitigation Techniques paper which I have presented at a number of conferences.

NF: Why should someone use OpenBSD instead of another operating system, besides security?

TdR: I don't really take any position of advocacy. People should use what they want to, and I am not the right person to say anyone "should" do anything. But hey, if someone is adventurous, check it out.

NF: A new stable version, 3.9, is about to be released on May 1. A complete changelog between 3.8 and 3.9 is available; would you comment on some of the new features of this release? Start with G5-based Mac support on macppc architecture. How well does it work at the moment?

TdR: It works on some of the models. For some of the machines we have a strange bug in the Serverworks SATA chipset that we have not been able to fix yet. There is no documentation for that chipset, of course.


Hardware sensors support (ESM, IPMI, IIC) -- a useful feature, especially on servers.

TdR: This has been a significant effort this release. These are three major subsystems that provide temperature, voltage, and fan sensor data. We have a unified system above that, that takes all this and makes it available to a daemon that can alert you when things go wrong.

Regarding specifically the "i2c" subsystem: in the Linux world there is the lm-sensors package, which requires all sorts of hand-configuration for each specific machine. In OpenBSD, we carefully probe for the devices, and it should just work, on every single PC, without any configuration. Thus, pretty much every OpenBSD 3.9 machine will have some sort of sensor now.

We have more work to do now that 3.9 is released, since the sensor daemon is a bit weak for reporting events. We want to make it fantastic.


The new ftp-proxy -- why write a new FTP proxy daemon when the previous one worked fine?

TdR: FTP is a nasty protocol to begin with, and trying to proxy it perfectly is a very difficult task. The new daemon just has a better design, and IPv6 works as well.


NFE, the Nvidia nForce MCP Ethernet adapter. How did you manage to write this driver? Is it reverse-engineered?

TdR: Nvidia did not give anyone documentation. Instead, they expect people to load a gigantic blob of binary code into their kernel, and just be happy with that. Some Linux people in Germany reverse-engineered the driver years ago, but the rough story I heard is that Nvidia asked them to stop, and they did. This just astounds me! In any case, Jonathan Gray (who started this effort) asked for their help with a few problematic technical details, and they refused. I could not believe that, so I asked as well -- and they refused again. These are Linux developers, basically placing the community in a situation where they have to run a binary blob of unknown code from a vendor, instead of sticking to their guns about open source? I must admit, I just don't understand some people. They must have much more flexibility to their belief systems than I have.

Damien Bergamini joined Jonathan toward the end and got all the bugs out of the driver. We are happy to say that it appears to be working better than the Nvidia binary blob. It is also significantly smaller, and it is very clean source code.

NF: In the past there was a movement in the OpenBSD community to press hardware vendors to release documentation about their products (Ethernet and wireless network adapters, RAID controllers, etc.) so that drivers could be written for OpenBSD or other open source projects. Some vendors did release documentation, but others didn't. Why do you think vendors do that? They don't want their products to be supported on OpenBSD?

TdR: There are always at least a few efforts in the project to get more documentation out of vendors. But some vendors are still incredibly resistant. We often run into vendors who have signed NDA agreements with Linux developers, who will then happily write a Linux driver filled with magic numbers, which only one developer in the world understands. Having signed the NDA ensured that Linux got a working driver, sure, but the internals are indistinguishable from magic. It cannot be fixed by anyone else, because it is full of secrets. It is a source code version of a blob.

There are many reasons why vendors will not give information out. I believe that all their reasons are a lie to the customer. I can get nearly complete data books for the parts that are in my car, and I should be able to get them for the parts in my computer.

Once in a while we hear from vendors out of the blue, and they offer us hardware and software without us having to ask. It is happening more -- mostly from Asian hardware manufacturers eager to have their hardware supported by all systems. On the other hand, American companies in particular are becoming increasingly insular, and sometimes we think twice before wasting our time trying to contact them. As a result, our support for a few high-end or very new American products is lagging, because there just isn't documentation available. That is a problem, but it should not be overstated, because 99% of the world is buying these Asian products. For instance, Asian 802.11 vendors accounted for perhaps 1% of the market five years ago, but within a year or two the market is likely to be split between Intel (because of how they tie their wireless chipset into their laptop Centrino brand) and the Asian vendors, such as RAlink and Zydas.

NF: Now that OpenBSD's user base seems to have increased a bit, do you have more success convincing vendors to release documentation for hardware?

TdR: We are having more success getting documentation, but I am not sure if it is due in any way to our user base size. Part of it might be that many more products are coming from Asia (where business sense still applies -- the customer gets the documentation he wants). I think that the Asian businesses are just being smarter about this. When it comes to documentation requests, an Asian company that says no is rare. An American company that says yes is rare.

NF: I understand that OpenBSD is financed from CD sales and donations. Does this money pay for all the projects needs?

TdR: Our income varies year to year. Donations rise and fall, and so do the sales of our products. Meanwhile, our FTP servers just keep getting busier.

We have built up some savings to deal with a rainy day, but our basic operation is perhaps falling behind slowly, or at least slowing our growth. We want to hold more hackathons, since that is where many amazing developments come from. If we had more money, we would also want to pay the travel expenses of some of the poorer developers, since we have some smart developers who are students or live in poorer countries. But with the finances we have, it is difficult to justify these things now. I want us to do much more, but we are constrained.

Donations make the most difference, since our project does not get taxed on them. We have investigated becoming a non-profit organization, but the margins and savings really do not make sense for our project, especially since most of our donations do not come from the country where we operate. Also, there are numerous other constraints and rules. So for now we are sticking to clear cash donations, without tax receipts.

NF: Lots of hardware vendors use OpenSSH. Have you got anything back from them?

TdR: If I add up everything we have ever gotten in exchange for our efforts with OpenSSH, it might amount to $1,000. This all came from individuals. For our work on OpenSSH, companies using OpenSSH have never given us a cent. What about companies that incorporate OpenSSH directly into their products, saving themselves millions of dollars? Companies such as Cisco, Sun, SGI, HP, IBM, Siemens, a raft of medium-sized firewall companies -- we have not received a cent. Or from Linux vendors? Not a cent.

Of course we did not set out to create OpenSSH for the money -- we purposely made it completely free so that the "telnet infrastructure" of the 1980s would die. But it sure is sad that none of these companies return even a fraction of value in kind.

If you want to judge any entity particularly harshly, judge Sun. Yearly they hold interoperability events, for NFS and other protocols, and they include SSH implementation tests as well. Twice we asked them to cover the travel and accommodation costs for a developer to come to their event, and they refused. Considering that their SunSSH is directly based on our code, that is just flat out insulting. Shame on you Sun, shame, shame, shame.

I will say it here -- if an OpenSSH hole is found that applies to SunSSH, Sun will not be informed. Or maybe that has happened already.


  • BSD