Software companies have long realized the economic and strategic value of using and contributing code to external open source projects. But they’re much slower to understand and apply the same open source methods of collaboration to their own projects internally, said Phil Odence, vice president of business development at Black Duck Software in a Collaboration Summit presentation today.

Companies that take the practices, processes, culture and methods from the open source world and apply them to internal software projects can see the same competitive advantages that participating in open source projects provide, including lower development costs, faster time to market, better code, increased innovation, improved collaboration, and better retention, Odence said.

“The most important thing about open source may not be the technology, but the methods it’s bringing inside the firewall of corporations,” Odence said. “It’s really hard to do, for some companies more than others, but the payoff can be huge.”

With the creation of its open source group last year, Samsung began working to transform its company culture to this “inner source” approach, in addition to vastly increasing its open source participation, said Guy Martin, open source strategist at Samsung Research America, in the presentation. That involves getting developers who work across teams and geographies to communicate better, as well as adopt common tools and methods for software development. The company’s approach to user involvement in the development process and employee staffing and incentives are also being revised.

“This is a big cultural change,” Martin said.

To developers accustomed to working in a more traditional, siloed and hierarchical management structure, adopting a peer review process for code, for example, can be a difficult adjustment, he said. “Telling coders that you’re not going to lose face because your patch was rejected… It’s hard not to take that feedback personally.”

How to get started

To successfully transition to an open source collaborative process, companies must first have a vision for what they want to accomplish and establish some processes for participation, Odence said. Then they can introduce the tools and technologies for collaboration.

Start by defining clear community goals, then identify seed collaborators who can act as mentors in the open source culture, Martin said. Then choose a small project to start implementing the new methods and deploy the inner source platform he said. Some essential components of an inner source collaboration model include:

– Open communication – via IRC, mailing lists and wikis and open archives of the decision-making process. The benefits are greater visibility and open planning between teams, faster resolution of dependencies and an enhanced documentation process.

– User involvement. In an open source model source code is available outside of the development team, requirements are published, and bug tracking is open. This empowers early users to find and fix bugs.

– Peer review. Patches are reviewed in small chunks and contributors and reviewers aren’t necessarily part of your product team. There’s a consistent review cycle and commit privileges are centralized. All code is reviewed before it’s committed. This improves code quality and you attract a much wider range of contributors.

“We had to put together a job description for what a project maintainer was,” Martin said. “It wasn’t a familiar job in Samsung because everyone has access to a project repository and so we don’t find bugs before integration.”

– New staffing procedures. Each project has a clear owner or owners, but if there’s something you can contribute to and there’s justification for how it helps your role, you can contribute to that and there are incentives built into participation.

Linux kernel developers Greg Kroah-Hartman, Jens Axboe, Dave Chinner, Matthew Garrett, and Mel Gorman participated in a panel discussion, moderated by LWN Editor Jon Corbet, at Collaboration Summit on Wednesday. Here are some of the highlights. For the full session, view the video, below.

For more information on what each developer is working on now, see our Linux kernel panel preview. 

On why companies hire kernel developers:

“We’re finding out more and more there’s actually a commercial imperative to give your software away to help others, because they help you and then everyone benefits… This is the way to do business.” – Dave Chinner, Red Hat.

“As Facebook has demonstrated with their Open Compute platform initiative, it actually makes economic sense to develop this in the open and not just improve your own data center but also improve data centers elsewhere … I view this as pretty much the same thing, just on the software side.” – Jens Axboe, Facebook.

On Facebook’s participation in kernel development:

Jon Corbet: “So do we need a certain number of likes to get a block patch merged now?”
Jens Axboe: “That’s the intent. I’m setting up a review group on Facebook now.”

On working with developers at competing companies:

“For the most part there’s very good collaboration between companies… and these people are my friends… Even when someone is working for competitor companies it’s hard to think of them that way.” – Matthew Garrett, Nebula.

“Even though the other companies are competitors, there’s also a symbiotic relationship. It reached the point a long time ago that there’s no single company that has access to enough expertise in all areas to completely go it alone.” – Mel Gorman, SUSE.

On their relationship with userspace developers:

“There are some cases when userspace developers come to us and say, ‘this kernel behavior is causing problems,’ and our immediate reaction is ‘no, the kernel is fine, you’re using it wrong.’ And sometimes that’s true and sometimes in fact we’ve created a user interface that’s impossible to interact with. But we tend to lean a bit too far to the side that userspace programmers know less about the kernel than we do, which is probably true, and therefore userspace programmers are usually wrong which may not be true. It would be nice if we didn’t always jump to that conclusion quite so quickly.” – Matthew Garrett.

On the pace of change within the kernel:

“We are blazing new trails. We’ve been doing that for a long time. We’re way past what Unix could ever do and that’s hard to do.” Greg Kroah-Hartman, The Linux Foundation.

“The Linux kernel is one of the largest collaborative software projects in the history of the world and has almost nothing in the way of formalized management structure. We have people who have a strong operating systems background who have been contributing code, and then we have people like me. I have a background in fruit fly genetics and yet someone lets me get close to the Linux kernel; this seems wrong. And then we have people who are genuinely kids in their bedroom. It’s a miracle it works as well as it does. We should be astonished that we’re able to get it so right so much of the time.” – Matthew Garrett.

“We still do it better than anybody else ever has, so give us credit for that.” Greg Kroah-Hartman.

On bug fixing and automated testing:

“We’re less reliant on eyeballs to find hard-to-find bugs – they just don’t get out of a developer’s machine because they’re found (in testing) before review is even considered. The bar has gone up and the code coming out from the developers has improved; it’s better quality. We still need better test coverage, (but) it’s better than it was.” – Dave Chinner.

Greg Kroah-Hartman: The tests we have, even in the kernel tree don’t work.
Mel Gorman: We need tests for the tests.
Greg Kroah-Hartman: No we need tests that work.

On UEFI secure boot and Matthew Garrett’s recent award from the Free Software Foundation:

Corbet: How many of you are using systems with secure boot enabled now? (A few people raise their hands.)
Audience member: It’s the first thing I turn off on a new machine.
Garett : You could just try it, except I guess that most of you probably haven’t bought new hardware for years because you’re kernel developers.
Audience member: I’m on my fifth motherboard thanks to UEFI, so…
Corbet: Well the three of you who are using it can thank Matthew.