Home News Featured Blogs Libby Clark CentOS Project Leader Karanbir Singh Opens Up on Red Hat Deal

CentOS Project Leader Karanbir Singh Opens Up on Red Hat Deal

In the 10 years since the CentOS project was launched there has been no board of directors, or legal team, or commercial backing. The developers who labored to build the community-led version of Red Hat Enterprise Linux (RHEL) worked largely unpaid (though some took a few consulting gigs on the side.) They had a few hundred dollars in their bank account to pay for event t-shirts and that was it. And the project's direction was decided based on the developers' immediate needs, not a grand vision of future technology.

Karanbir SinghThat's all changing with the news last week that CentOS will join Red Hat, said Karanbir Singh, CentOS project leader and one of four CentOS developers going to work for Red Hat as part of the new collaboration. For the first time, developers will work on CentOS professionally and that's created a “paradigm shift” for the project and its contributors, he said.

Red Hat will bring resources and structure to the project. And in return, CentOS will provide a stable testing ground for new technologies – for RHEL and many other projects as well, Singh said.

“We'll be working on infrastructure other projects need to be successful with us,” Singh said. “We won't be delivering the features. We'll make it as trivial as possible to come in and do the build you need, without needing to learn everything about building CentOS images.”

Here Singh discusses how CentOS and Red Hat joined up; what to expect from the project now; Red Hat's role in the community; opportunities for more partnerships; and his own feelings on working on CentOS professionally for Red Hat.

How did this partnership come about? Did you really just get a call from Red Hat lawyers one day (as you joked on Twitter)?

The big challenge was when Red Hat called us and said, “who are your lawyers? Who do we speak to?” We don't have a commercial interest or a legal team, so there's no entity to partner with. Red Hat is coming in to incubate the project, provide some of the resources and help us meet goals. So it's more accurate to say we're joining forces. It's a bit of legalese, but it's a better description because some of the things they came up with we thought were too corporate.

What enabled you to “join forces?” Has your relationship with Red Hat improved?

A lot of people seem to think the relationship with Red Hat is brand spanking new but it's not, we've always used them as upstream. We were taking what they were putting out and making it available for everybody.

But one of the challenges we have was isolating people from legal issues that might come up. Say someone forgot to remove the appropriate trademark somewhere. By law (Red Hat) is required to protect trademarks. The only way we could protect that is by having structure, working as an organized group privately. One of my biggest gripes was CentOS couldn't be worked on out in the public because if we did it opened us up to possible legal action.

As part of the relation with Red Hat now we can work with them to make sure that (trademark) work is done upstream. That allows us to be a lot more open.

How will CentOS be more open?

In the past the way to get into the CentOS build process was to be nominated out of the community into the QA team. You had to have a few years of history, lots of forum posts, and not have a commercial interest, then you could come into the QA. Then those were elevated into privileged positions with access to servers.

That entire infrastructure just goes away. We go from a build team of 6 to15 people, with 60 to 70 people in the QA team, and the big win – what I'm most excited about – is other people in other projects trying to do their own thing can come and work with us to do what they need to do.

How can other projects start working with CentOS?

We're still working out the details but we want to call them CentOS variants. They can add layers and components and configure the system to make their own whether they want to do VOIP, hosting, cloud, storage, (etc.)

When we were talking to Red Hat originally we thought it would be ideal to have one project that steps forward and model the variant projects on that relationship. We wanted six projects to come through in first 6 months. So far, we've had seven come through in the first 24 hours and three more since then.

One requirement, though, is every project needs a nominee from an existing development team to make sure we set the right expectations. There are a bunch of conversations going on.

What are you hearing from projects that want to be involved?

Going back 4-5 years, a lot of people found CentOS to be a stable OS they could plan against, but because there's no SLA, it's a best-effort set up. They'd take it back to their own ecosystem and do their own customizations in-house. They have to own every bug, feature and patch.

The variant process will allow them to come within the CentOS project. They only have to maintain the stuff they care about. It's like if 40 different projects are using Apache to do something, only one needs to maintain Apache, the others can focus on their own bit.

How will CentOS benefit from the relationship?

People can come in and we can expand the ecosystem together. There's so much stuff we've had to be very pessimistic about by keeping things private, by not allowing people we didn't personally know to get involved. Those barriers are coming down.

How do you feel about being in a structured set up?

We have had levels of structure, but not to the extent of having a proper board. The great thing about Red Hat people is they understand how some of these communities work. They meet everybody on IRC every day, for example. We will firm up processes and procedures, like what infrastructure we'll use, how code will flow, over the next few months and everybody can pitch in and make the decisions together.

We haven't been a feature-led community, we've been a problem-led community. Eighty-five percent of the time, it's the same thing. It's that 15 percent when it's not the same thing that a developer-led community is important.

The new things we've brought into the project weren't because someone had a big idea of what's coming in the future. It's been to address problems we already have. For example, everyone out there is trying to build cloud instances to adapt for what's coming down the road. Our approach to that right now is if they're smart enough to install it, they don't need our help anyway.

The CentOS project as it was is going to stay intact, but we'll be working on infrastructure other projects need to be successful with us. We won't be delivering the features. We'll make it as trivial as possible to come in and do the build you need, without needing to learn everything about building CentOS images. People will come together to work on the same problems.

Will CentOS become a proving ground for new technologies in RHEL as some commenters have speculated?

I think it should. But it would be wrong to say it will be a proving ground for things into RHEL. It's more appropriate to say it's a proving ground for anything. Because the platform is resilient you can build on CentOS without things caving in. There's nothing in what we're doing that limits ourselves to what Red Hat is doing. Our biggest (potential) variant is Xen Project and Red Hat isn't involved in that at all.

We do expect our work to feed back to RHEL. But we're pretty serious that the platform we're building isn't specific to RHEL.

CentOS 6.5 was released in December – what's on tap for the next release? How will the new partnership affect that?

In spite of the fact that I work for Red Hat now, I'm not privy to their release schedules. So over the next few months the board will publish our own road map. The mainline, the core, will stay as it is. But the variants will be able to set up their own road maps (based on their own release schedule).

The big thing we're trying to get to is to have an installer for CentOS users to select what variants they want. But that's a wish-list item that needs to be specced out before it's developed and designed.

You are personally going to Red Hat – what do you think you'll be able to do in house that you wouldn't be able to do otherwise?

Ten years ago when some of us were getting together to start the project, the aim was to get 300 people to use it, that was fantastic. From our perspective it's been fairly successful. How we define success is to build something we would use and that comes back to the user-driven approach. We cared about how things worked, where they worked, and overall it worked out well having that user perspective.

I've never worked for a big open source company before but I hope to bring that user perspective to Red Hat and what I'll take a way is a large approach to user communities and hopefully manage that better.

Otherwise, not much has changed. They sent me a phone and a laptop and that's how it's going to go. I feel quite privileged to have this opportunity to focus on the CentOS larger ecosystem side of things.

Will you be able to relax a little now? Maybe enjoy your work more?

There's been no money involved in the project. We have a bank account that's never had more than a couple hundred dollars for printing t-shirts for events. This is the first time there's a group of people professionally working on CentOS as a platform.

How CentOS used to happen was some of us would go to work and then work another 40 hours a week on CentOS. You can't sustain 80 hours a week. The reason I did it wasn't for compensation, it was because I wanted to.

You wrote on Twitter, “I think this is also a great opportunity for RH to prove it cares about community in an open, user focused, transparent way.” How can they do that?

That's a hard question. Here you have a total community-led project suddenly going from a place of not accepting financial donation to being backed. We have access to certain resources we can rely on – people available during working hours and machines available, for example. There's a paradigm shift in the project, but we want to keep the project as it is.

There's nothing we're doing right now that should make you feel it will change. If there is make sure you make those feelings public and judge us in six months. The onus for proving that, the entity to make that point and make it visible, is Red Hat.

What do you think this partnership will do for Red Hat's image in the Linux community?

I think it should be good. They already contribute so much. The second day there I made the mistake of looking through the address book of people I could call and I was intimidated. These are the people I've looked up to and suddenly he's two extensions away from me. But having them on board now it just expands what they're doing. I think it should get them some good will.

A lot of feedback I get from people at Red Hat, from senior managers to junior developers, is that they appreciate what we do, and that ethos of being open and a community is quite strong within Red Hat itself. People outside the developer world don't really get how engrained they are in the open source community mindset.

Anything else you'd like to share with the community?

Come join us.  



Subscribe to Comments Feed
  • Microlinux Said:

    Hi Karanbir, Since now you have all the RH member extensions, can you please prank call Lennart Poettering and tell him the engineer board decided to replace systemd by runit? :o)

  • Scott Dowdle Said:

    It's hard sometimes... to accept that people can read the same things and get completely different ideas of what they meant. So, the "CentOS distro" isn't going to change. It still is going to be be a RHEL clone. What is changing is that there will be optional builds by related projects that add on top of CentOS to offer easier access to the software their project provides. Those are the varients. So I guess you could say that varients are a proving ground or a test bed... or whatever... but the core isn't changing. This is basically expanding what CentOS already offers... and having additional spins or remixes with a lot of the third-party software end users were already wanting to add to it anyway.

  • jake Said:

    "CentOS will provide a stable testing ground for new technologies" Opps. So much for Centos being a free clone of red hat. Very Machiavellian of Red Hat. By turning Centos into a fedora-like test bed instead of an enterprise OS, they eliminate the competition while appearing to be the good guys.

  • Code Ghar Said:

    It feels more like Debian's relationship with Vyatta, Kali, Ubuntu, CrunchBang, etc. RHEL will be Debian in this case and CentOS a gratis clone which serves a base for other projects to build on top within the community (if they so desire). Only time will tell. But I don't fear the worst as a CentOS user because there are other clones, e.g. Scientific Linux, Springdale, etc.

  • Pepa Said:

    If it is indeed no longer going to be a RHEL clone at the basis, then it is a great loss and a dastard move by RH. But I keep hoping it will still be a clone, but with different variants being provided.

  • pat Said:

    That's shocking, I thought that's why we had Fedora in the first place, for testing...

  • DevGuy Said:

    My thoughts exactly! R.H could have done this on it's own.

  • mankou0 Said:

    So basically Centos is becoming something between Fedora and RHEL? I hope this doesn't kill Centos as we know it, because Cento's userbase was essentially people or businesses that wanted RHEL without paying the fee. I certainly hope I'm reading this interview wrong. On the bright side, another team of Linux devs will finally start getting paid for their hard work.

  • Scott Dowdle Said:

    Opps, my comment was for all of those who were "shocked" and thought CentOS was becoming more like Fedora... but obviously I clicked on the wrong [Reply] button.

  • Erinn Said:

    What does "opps" mean? Is that l33tsp34k for operations?

  • Steve Said:

    it seems like the best option for business and the community. I don't see why people are griping. this is win win. let it play out.

  • Ward Mundy Said:

    Paradigm shift indeed. We're just starting to see the long-term damage to the CentOS project. Here's the rest of the story:

  • Pepa Said:

    Yes, it is very bad. :-( Read the link above... All the myriad of distro's using CentOS are in trouble, and only official CentOS distributions are allowed to use CentOS.

  • Linux Said:

    As new concepts they are explained in, making it easy for anyone to understand.

  • Mick Russom Said:

    I begged and argued for YEARS. I used to be into whitebox and Caos-linux. And CentOS and the folks that run it have collected all know-how and destroyed all other white-box-RHEL-rebuilds by attrition. Now even Scientific Linux 7 is knuckling under the dictatorship of CentOS. Now we will have only one place that builds CentOS and they keep all the knowledge, the know how, essentially all the make files under lock and key. The "C" in centos is no longer community, more like commoners or serf's copy of RHEL when they feel like it.

Who we are ?

The Linux Foundation is a non-profit consortium dedicated to the growth of Linux.

More About the foundation...

Frequent Questions

Join / Linux Training / Board