May 22, 2006

DebConf6: Hot, spicy, and working hard to satisfy Debian users

Author: Robin 'Roblimo' Miller

OAXTEPEC, MEXICO -- The 7th annual Debian Developers Conference (DebConf6) brought close to 300 Debian developers, package maintainers, and other interested parties to Mexico's government-owned Oaxtepec Vacational Center last week.

Admission was free, including lodging and food (and in some case travel expenses) for those who could not afford to pay their own way, which meant that almost any active Debian participant who could get time off from work and was willing to travel to Mexico was welcome. There was a fair amount of partying, but even more serious work, much of which was aimed at making Debian more user-friendly. There was also plenty of discussion about how "Debian derivatives" should interact with the primary Debian organization. Much of this discussion centered on Ubuntu, and on differences in how Debian and Ubuntu handle technical and social challenges, especially after Ubuntu founder/benefactor Mark Shuttleworth delivered his Ubuntu annual report to the assembled Debian masses on Monday, May 15.

Soon after he finished his informal discussion with about 110 Debianistas, Shuttleworth flew to San Francisco to stand on a stage and share some love with Sun Microsystems CEO Jonathan Schwartz Tuesday at Sun's JavaOne conference in front of a much larger audience.

But Shuttleworth was back in Oaxtepec Wednesday, hanging out and chatting with individuals and small groups, checking his email, and otherwise behaving like most other DebConf attendees. It was obvious that even if not all of them loved him -- or Ubuntu, or his for-profit company, Canonical Ltd. -- he considered this small group important enough to be worth more of his time than the 14,000 people who went to JavaOne.

Sun, Intel, Hewlett-Packard, and other Linux-using corporations also had people at DebConf, and the backs of the conference t-shirts listed 24 corporate sponsors. Indeed, Sun people told DebConf attendees about the company's latest Java licensing strategy on Monday, a day before it was announced at Sun's own JavaOne conference.

Why is Debian important?

Novell and Red Hat have significant PR and ad budgets and use them to make lots of noise in the trade press. Gentoo has devoted followers who post pro-Gentoo messages in almost every online discussion about GNU/Linux distributions. Fedora has Red Hat support, plus a fine reputation for Free Software purity. In DistroWatch's default "last six months" rankings, Debian is number seven in popularity, well behind Ubuntu (number one), SUSE, Fedora, Mandriva, MEPIS, and Damn Small Linux.

But Debian has an unparalleled reputation for stability and security. The faults that make it less than totally popular, including a conservative approach to adding new software or new software upgrades to its "stable" releases, and installation and administration tools designed more for command line-savvy sysadmins than for casual desktop users, are virtues in its devotees' eyes.

Debian also has a unique Social Contract that sets it apart from other GNU/Linux distributions. To become a Debian developer, packager or maintainer, you must agree to this Social Contract and become part of a rigid (but emphatically democratic) organizational structure.

In many ways, Debian is more of a social movement than a free software distribution. One of the greatest tenets of this movement is that quality control is more important than release schedules, feature requests, and even usability. If a free software package is accepted as one of the 15,000+ currently supported as part of the main Debian distribution, it is a virtual guarantee that it is stable, does what it is supposed to do, and interacts correctly with other Debian-endorsed software packages. A free software developer -- individual or corporate -- whose work becomes part of Debian can rightfully point to that inclusion with pride.

The Ubuntu challenge

Shuttleworth is handsome, charismatic, a strong free software advocate, and rich. While Debian is maintained entirely by volunteers (many of whom, admittedly, get paid by their employers for at least some of the time they spend working on Debian), Shuttleworth can and does hire developers to work directly on Ubuntu full-time. Many of the developers he's hired are either current or former Debian volunteers. He says that in many cases the people he hires (through Canonical) to work on Ubuntu keep maintaining the same packages for Debian; that Ubuntu improvements and bugfixes are fed directly back to Debian.

But there are grumblings in the Debian ranks about many of Shuttleworth's claims. Long-time Debian volunteer/activist Joey Hess worries that Debian is starting to become a supermarket of ingredients, in the form of packaged software binaries and utilities, while Linspire, Xandros, MEPIS, Progeny, Ubuntu and other Debian-derived Linux distributions are busily turning those ingredients into the equivalent of prepackaged "convenience food" meals that consumers in a busy world often prefer to the time-consuming process of making their own, individualized meals (or Linux installations). Hess is not at all pleased by the possibility of Debian being relegated to a background role in the Debian universe while more user-palatable Debian variants get all the public approbation.

Debian Project Secretary Manoj Srivastava worries about Ubuntu's tendency to treat "upstream providers" poorly. In a private conversation over breakfast, he spent at least half an hour telling me that when Shuttleworth claimed bug reports and patches from Ubuntu were always fed back to Debian, he was not telling the whole truth. Yes, Srivastava said, perhaps a whole stack of patches and bugs are sent to Debian or posted somewhere, but the right way to deal with an upstream developer or project is to send specific patches and bug reports directly to the person who could do something about them, along with direct contact information for the person submitting them rather than having all the ones coming to Debian from Ubuntu being "from Ubuntu."

The discussion about dealing with streams both down and up was a recurring DebConf theme because in many ways, and to many users, a GNU/Linux distribution's primary purpose is to serve as an intermediary between end users and software developers. If, for example, I run Ubuntu on my laptop, and I believe the default GNOME desktop it incorporates should have a "Logout" icon in its control panel instead of forcing me to click System -> Logout, the logical place for me to go with that suggestion is to the Ubuntu bug reporting system, which also handles feature requests. As an end user, I may not know (or care) whether this change requires work from Ubuntu, on a Debian GNOME package from which the Ubuntu one is derived or needs action from the GNOME project. I just put in my bug report (or feature request) and let things happen from there.

I'm using this specific example because there has been some online discussion about a default logout icon for the Ubuntu GNOME desktop, not because I had any trouble adding such an icon through the basic "Add to Panel" dialog I brought up by right-clicking on the panel in my not-the-latest Ubuntu/GNOME installation. I'm also using this example because several people at DebConf told me the person primarily responsible for GNOME package maintenance in Ubuntu -- as a paid Canonical employee -- is also the primary Debian GNOME package maintainer, and added the most recent version of GNOME to Ubuntu several months before he got around to adding it to Debian's unstable branch. This led to a certain amount of ill will toward Ubuntu from Debian believers, especially since other, similar tales of divided loyalty by developers who work on both Ubuntu and Debian were making the rounds, too. One DebConf attendee summed it up to me this way: "Nobody wants to do Ubuntu packaging all day at work, then do Debian packaging at night on his own time. That's just too much."

Politics aside, Ubuntu has diverged enough from Debian that you can no longer assume a Debian package will install and run in Ubuntu or vice versa. And there are other factors that separate Ubuntu from its Debian parent:

  • Canonical is not a non-profit. Shuttleworth may be plowing a substantial percentage of his fortune into Ubuntu development now, but he eventually hopes to make money from this venture by offering commercial Ubuntu support, customization, and other services.
  • Shuttleworth was a Debian person long before he got rich, but he is by no means a "free software or die" guy. Indeed, one of Canonical's larger projects, Launchpad, is not free software. The site's FAQ says some or all of it may be free someday, "... though it will take some time (potentially, years) before that happens."
  • Ubuntu loves Python. Warren Woodford of MEPIS has switched the MEPIS base from Debian to Ubuntu, and has some add-on control panel feature he plans to submit to both Ubuntu and KDE as an act of good open source citizenship, but he says, "They'll be rejected automatically because they're in C, not Python." Warren isn't the only programmer who prefers (fill in language here) over Python.
  • Ubuntu's tight release schedule can compromise quality. Debian is famous for being "behind the times" when it comes to having the latest packages available, especially in its stable branch. But you never see bug reports like these Ubuntu bugs for Debian.

This last factor -- the quality thing -- is what bothers Brazilian Debian developer Otavio Salvador most about Ubuntu and keeps him loyal to tried and true, stodgy Debian. He told me the Ubuntu emphasis on releases every six months has led to "some ugly hacks." He used the code that powers Ubuntu's boot splash screen as an example. It looks good to users, he said. The problems "are not something a user sees, but they caused problems with LTSP," (the Linux Terminal Server Project) which is one of Salvador's prime interests.

Salvador said he recently sent six patches for LTSP-related bugs to Debian maintainers, and that those bugs were fixed almost immediately, but that those same bugs have not yet been fixed in Ubuntu. "Users trust you," he said. "You have to give them things that are stable." And he is not sure Ubuntu is committed to doing this. He also personally distrusts Shuttleworth to the point where he bluntly told me, "What he says and what he does are different," and went on to say that the "relationship is terrible" between Ubuntu and Debian, with the quality vs. release schedule problem creating much of the friction.

Because of what he called "a lot of mistakes" in Ubuntu packages, Salvador said he believes Ubuntu should not be used on production servers. He noted, though, that not everyone agrees with him on this, and pointed out that "I don't say to people directly, 'Don't use it,' but I don't use it."

Meanwhile, Ubuntu seems to work fine on the ThinkPad T43 I'm using to write this article, and seemed to work fine for plenty of other people who attended DebConf6. At one point I looked around me while I worked in the "hacklab" room where the wireless access point's signal was strongest, and noticed that five out of 12 laptop screens in my line of sight were running Ubuntu.

Debian is moving forward

Thursday morning former Debian Project LeaderBdale Garbee led a discussion about Debian governance and how it might need to change to meet new challenges. One metaphor he used that especially caught my ear was the way he likened Debian to a city government in the middle of a metropolitan area where a lot of the growth is taking place beyond the city limits, beyond direct control of the city government.

The city still needs to maintain a close relationship with the growing suburbs. In return -- for their own sake -- the suburbs need to make sure the city stays healthy so that the metro area remains vital as a whole.

Garbee asked this question: "What can we do to make the relationship with our derivatives be more active and proactive?"

He also mentioned the possibility of changes in the way Debian is governed, especially in light of the growth in the Debian community (and its derivatives) since the current governing rules were drawn up, and suggested participating in, which at the time this article was written only linked to slides from a talk about Debian governance Garbee gave back in 2005.

Another, more direct, Debian coding issue several people talked to me about was making the Debian installer more user-friendly. One said, "It got to where it was great for the average sysadmin-type Debian user, and development on it slowed, but now we need to think about making it more useful for beginners."

Someone else nearby chimed in with, "Debian was never meant for Linux beginners."

Salvador, the Brazilian, said he now has a "pure" Debian desktop put together that is suitable for beginning and/or non-technical computer users and is being used in a growing number of Brazilian hospitals. He promised to give me a link for it once we both got home, along with a warning that it's still only in Portuguese.

The upshot is that a number of people within Debian -- and we're talking about people inside the Debian "city" itself -- are actively working to advance the project and to help it move faster, develop more efficiently, and become more user-friendly, all while maintaining Debian's overall devotion to stability, security, and quality control.

DebConf6 was more than hacking and discussion

NewsForge isn't the kind of scandal sheet that dwells on events like the (female) European Debian person who reportedly decided to go skinny-dipping in one of the Oaxtepec resort's pools well after closing time and got chased out by a security guard who was so unnerved by her nakedness that he found himself unable to look directly at her even as he ordered her out of the pool. We'd rather give you a nice link to the (non-porn) DebConf6 Photo Gallery instead.

This blog shows some of the tremendous effort the organizers -- especially Gunnar Wolf -- put into getting everything together, including the sweat it took to bring Internet access to a resort facility that had none by begging and borrowing equipment, then installing it themselves.

Wolf's blog (the second link in the last paragraph) also mentions some of the visa and travel problems a group organizing an international conference can face. Wolf talks about the problems in the context of Mexican officialdom, which can be maddeningly slow and inefficient, but others told me that trying to get visas for the same group of people from the US government would have been far worse. One person (from Germany) who has helped organize other technical events told me visa problems make it virtually impossible to hold a truly international scientific or technical conference in the US these days.

Barry Hawkins posted some excellent DebConf6 photos here and on Flickr. Meike Reichle kept an excellent personal blog that focused on the social rather than formal side of DebConf6. Hess posted this one-page summary of his DebConf6 experience.

For more insight into the hassles Wolf and his crew faced in bringing Internet connectivity to the conference, here's some technical detail about the cabling they installed. And beyond this, there were people with all kinds of power requirements -- especially Europeans whose equipment wasn't set up for Mexican "standard" 110 VAC outlets, which are the same as US ones -- so the place rapidly turned into a veritable jungle of plug-in adapters.

The missing link(s)

Holger Levsen and several others shot HD (high definition) video of most DebConf6 presentations. Levsen's blog includes several posts about the frustrations he faced shooting the videos and trying to stream them. We'd give you a link to Levsen's videos online, but it's going to be a while before they're processed, edited, transcoded, and uploaded. When they finally are, we hope we can give you a link (or links) to them.

Meanwhile, despite video and connectivity problems, DebConf6 has been both enlightening and entertaining, and I'm sorry that it's time to go home.

For me, the conference's defining moment came Thursday morning. I was sitting on the veranda outside the hacklab area, waiting for my email to download through the overloaded wireless network, listening to the two guys sitting across the table from me.

One looked at the other's laptop screen and said, "Oh! So that's how you get that to work. Thank you for showing me." A moment later the other guy looked at the first one's laptop screen and said, "That is a really cool trick. How did you figure it out?"

They went on like this for the hour-plus it took me to read and respond to my overnight email, and they were still at it when I left. I didn't get their names because I didn't want to disturb them.

This is the kind of peer-to-peer networking that makes open source conferences like DebConf6 worthwhile. It wasn't a teacher/student relationship. It was two hackers sharing knowledge with each other. Perhaps this pair had talked before in IRC or by email, but it looked to me as if this was their first face-to-face meeting, and knowledge was flying back and forth between them at a rate not even the most sophisticated online conference software could have handled.

The knowledge they shared will, no doubt, improve Debian -- and by extension will help improve GNU/Linux for everyone, whether they use Debian (or a Debian derivative), SUSE, Fedora, Gentoo, Mandriva, or any other distribution.

This is the nature of free and open source software, and is why I prefer attending smaller conferences over going to huge, commercial conventions with "show floors" full of corporate displays.

I hope my schedule and my employer's budget make it possible for me to go to DebConf7 next year, wherever it is held. And if you are interested in Debian, I hope you manage to get there, too.

Click Here!