If you’re using open-source software, you’ve probably come across a bug that you want to fix but don’t have the expertise to do it yourself — and the original author isn’t all that interested in fixing it. With Git Bounty, which was dreamed up by a team of French Canadians (and one Frenchman) from Montreal at our Disrupt SF hackathon this weekend, you can incentivize open-source programmers to fix those bugs for you. Git Bounty lets you pick a bug you need fixed, set a reward and then publicize it.
The hackers (Angus MacIsaac, Adam Burvill, Anton Shevchenko, Nathan Boiron, Martin Coulombe), which all work together at Montreal development shop Osedea, told me that they came up with the idea on Friday night before the event.
Isabel Jimenez will graduate from her master in Computer Science this month. She spent three months contributing to Apache Mesos with the Twitter team thanks to Gnome’s Outreach Program for Women.
MesosCon started with a keynote from Benjamin Hindman, one of the creators of the project and Mesos team lead. One of the major news items in this keynote was the release of Apache Mesos version 0.20 later that day. This release answered a popular request from Mesos users: Docker integration.
Although it was already possible to launch Docker containers with Mesos, thanks to the external containerizer mechanism, Docker has become a first-class citizen. With Mesos 0.20 it is now possible to use Docker containers without any external installation.
The history of Mesos with containers goes back to the start of the project, in 2011. The Mesos default containerizer used LXC. Later for better integration Mesos decided to use cgroups directly.
With Docker’s popularity growth, Mesos started supporting Docker with some work like mesos-docker, an executor, and later with Deimos, an external containerizer.
But to fully enjoy all of Docker’s features, this year it became fully integrated into Mesos. This means that a specific containerizer was created for Docker inside Mesos.
A new component, ContainerInfo was added as the base protobuf message along with a DockerInfo that helps to lend parameters to Docker container specifics. The ContainerInfo base was added to TaskInfo and ExecutorInfo also to allow users to launch Docker as a task or an executor.
What the Docker containerizer does is simply translate Task/Executor Launch and Destroy calls to the Docker CLI commands.
In my opinion, this announcement is a immense step towards what could be the future of Docker containers orchestration. Other new tools are coming to the surface to answer this huge demand but they are no more than a Docker API smart overlay and still lack lots of improvements.
One of the speakers at MesosCon was Victor Vieux, a Docker core maintainer, who talked about how easy it was to use this new integration to build new Mesos frameworks. He demonstrated Volt, a framework I had the chance to work on, that serves as perfect proof-of-concept for Mesos Docker support and its easy and quick application.
Although not every Docker feature is supported yet in Mesos, Volt allows a user to launch tasks inside Docker containers, letting the user benefit at the same time from Mesos resource management.
The Marathon Mesos framework, by mesosphere was also quickly updated to support this new Docker integration.
The community had a quick positive reaction to all these changes and in my opinion, is eager to see the integration fully developed. Especially about letting Docker containers be linked and communicate with each other.
The event was also a great chance to talk with all Mesos users and developers. As part of the organizing committee I had the occasion to participate in the speaker dinner where I witnessed the great energy behind different Mesos use cases.
The hackathon day was a great opportunity to hack on the Mesos ecosystem and implement the newly announced features, for example JenkinsOnMesos, which added Docker support during the hackathon.
It was also an opportunity for me to continue some work on Volt and get firsthand feedback on it.
Of course MesosCon was about much more than this fantastic news. All the talks were rewarding, the open-mindness of the Apache Mesos community lead to very interesting talks on different cluster management approaches and how to make Mesos better. Benjamin Hindman let us know that 1.0 release was as near as some API upgrades and we are all looking forward to MesosCon 2015!
Amazon Web Services first launched in 2006 with one instance and one operating system: Amazon Linux. The cloud computing giant has since expanded to offer customers the option of running on more than 30 instance types and more than 10 operating systems, but Linux, Xen and other open source projects remain the core technologies behind AWS.
“We view open source as a companion to AWS’s business model,” said Chris Schlaeger, director of software development at Amazon Web Services and managing director of the Amazon Development Center Germany GmbH.
Schlaeger, who leads the team that develops Amazon’s cloud computing service, EC2, as well as Amazon Linux, will give a keynote presentation at CloudOpen Europe, Oct. 13-15 in Dusseldorf, Germany. Here he discusses how AWS uses Linux, the Amazon Linux operating system, the company’s new development offices in Germany, and what he’ll cover in his keynote presentation.
Linux.com: What are the building blocks of AWS? What role does Linux play?
Chris Schlaeger: Amazon Web Services is built on two fundamental services: S3 for storage services and EC2 for compute services. These were the first services AWS launched. Since launching in 2006 we have added more and more services that build on top of each other every year. In 2011 we added 80 new features and services, in 2012 it was 160. In 2013 it was 280 and we are already close to hitting this number for this year. Linux, in the form of Amazon Linux as well as Xen are fundamental technologies for AWS.
Since first launching in 2006 with one instance type, and one operating system, Amazon Linux, Amazon EC2 has now grown to more than 30 Instance types and more than 10 Operating Systems, both Linux and Windows, available from 10 infrastructure Regions in seven countries around the world. Customers now use Amazon EC2 for everything from media and web hosting to development and test environments, High Performance Computing to running ERP systems, running complex enterprise applications such as Oracle, SAP, IBM and Microsoft to Big Data analysis, and all using the same pay-as-you-go model with no long-term commitments.
What is Amazon Linux? How has Amazon customized the OS for its own purposes?
Amazon Linux is AWS’s own flavor of a Linux operating system. Customers using our EC2 service and all the services running on EC2 can use Amazon Linux as their operating system of choice. Over the years we have customized Amazon Linux based on the needs of AWS customers. We do still strive to be RHEL compatible to ease the customer transition to Amazon Linux. As my teams also develop the EC2 instance types we can ensure that Amazon Linux is the most optimized version of Linux to use within EC2.
AWS has a new development presence in Germany, can you tell us what you’re up to there?
Since we launched the business in 2006 we have seen rapid growth in the services and AWS now has hundreds of thousands of customers in over 190 countries. Many of the most well-known start-ups in Germany, such as Wooga, OneFootball and Soundcloud use AWS as well as many well-known German enterprises such as Kaercher, Talanx AG and Siemens Healthcare.
In terms of our physical presence, we have offices in Munich, Berlin and Dresden. In May last year we opened the Amazon Development Center Germany GmbH. It has currently 2 major development sites in Berlin and Dresden. The teams in Berlin are focusing on Machine Learning while the Dresden teams are working on the Linux Kernel, Xen, EC2 instances and Amazon Linux. I’m very excited about how rapidly the Dresden site has grown over the last year or so. We’ve just added another floor to host even more Linux and Xen developers.
How does Amazon contribute to the Linux kernel and open source in general?
We view open source as a companion to AWS’s business model. We use open source and have built many AWS services on top of open source technologies, like the Linux Kernel, Xen and MySQL. We also contribute to several open source projects.
If we find a bug in the Linux Kernel we usually send the patch upstream. It’s much easier for us to ingest such fixes with the next Linux Kernel release than having to maintain our own patch sets with bug fixes.
Can you give us a short overview of what you’ll discuss in your keynote at LinuxCon and CloudOpen Europe?
In the Keynote, I will provide an overview of some key AWS services and how they use Linux and Xen. I’ll provide some insight into Amazon Linux and its development. This will include, among other things, an introduction to our Germany-based development activities.
The Apache Infrastructure team has gotten Git migrations down pat. Just ask the Apache Hadoop project, which moved from Subversion to Git in less than 10 days.
Hadoop committer Karthik Kambatla says that getting from the request to Infra to a usable repo “took less than 10 days. The actual migration itself took 3-4 days. Daniel (Gruno, an Infrastructure team member) smartly suggested we start the process on a weekend to minimize disruption… he updated us fairly frequently on the status and made useful suggestions. There were a couple of follow-up items that were fixed fairly quickly as well.”
Chris Douglas, PMC Chair for Hadoop, says that Git wasn’t an option when Hadoop spun out of Lucene. “At least, there was no discussion about it. Git wasn’t available, in any case.”
In fact, some projects may not even be aware that it’s an option today. Git is still labeled “work in progress” (WIP), and it may not be obvious that Infra can switch projects to git. David Nalley, VP of Infrastructure at the Apache Software Foundation (ASF), explains that the “WIP” label is just that. “With git used by so many projects, it’s just a label at this point. Git is fully supported and in production at the ASF.”
“Historically, the original git.apache.org service was a svn-to-git mirror, and today that also provides the mirroring capabilities to Github,” says Nalley. “There is a project slated to merge what is git-wip and git.apache.org into a single service to rid us of the WIP moniker.”
Kambatla says that the Hadoop project wanted to switch for a number of reasons. First, he says, “most users and developers were using git for development for a number of reasons: local commits, easy patch updates against latest committed versions, sharing code with others, etc. SVN, in my opinion was being used primarily only to commit code. Using git would avoid this duplication.”
Secondly, Kambatla says that working on feature branches “is easier with git, to keep up with the work on main branches.” And finally, Git provides the “potential for better code review tools.”
Douglas agrees that Git was a better fit for Hadoop. “Hadoop often has several active development branches that require backporting features. I don’t know if subversion added better support for this workflow subsequently, but git made it much easier to manage multiple patches, branches, and review. Because Hadoop uses review-then-commit (RTC), one often has multiple patches in flight that require quick context switches. So most developers had already switched to git for their work on Hadoop.”
Git does introduce some feature trade-offs, though. Douglas notes that the authorization is less fine-grained with git. However, he expresses little concern there. “Our experiments with branch committers suggest that we won’t regret relaxing the strict authorization we exercised with Subversion.”
Daniel Gruno, who performed Hadoop’s migration, says that code provenance remains intact but git isn’t SVN. “There are a few things like property settings that are lost, and the overall structure of a repository changes, which can make it somewhat difficult to browse what happened in the past – until you suddenly learn how it’s all set up, and then it gets easy.”
Though Kambatla and Douglas make it sound simple, the actual migration from SVN to git isn’t trivial. First, says Gruno, it involves “a lot of paperwork. We have to be very careful when we migrate, and so we have a set of rules we always follow.”
The process involves locking the SVN repository, “so someone doesn’t commit something while we migrate and it gets lost or messes up the migration,” says Gruno. “Once a project decides to move to Git, the Subversion repository is basically voided.”
The physical migration requires “a big ol’ Perl-based system” that “takes ages to complete” says Gruno. Ages, as in “1.5 hours per 1,000 commits.” Hadoop took more than two days to migrate, and he says that OpenOffice took more than a week. Hence the weekend migration plan.
After the migration, the repo is put into read-only mode so the PMC can “inspect” it. After the project’s PMC signs off, write access is allowed. And, finally, the old SVN repository is “partially opened” for write access so the project can update its website.
Gruno says that quite a few projects are currently using git as their Version Control System (VCS) of choice. Out of 151 Top-Level Projects (TLPs), 69 are using Git as the primary VCS (Gruno points out that all Web sites are using SVN), as well as 19 podlings in the Apache Incubator (out of around 30 incubating projects). So, projects at the ASF are about evenly split in terms of which projects use Git and which use Subversion.
Furthermore, the git repositories are pretty active, says Gruno. “When commits are concerned, Git usually outnumbers subversion commits by 1.5/2 to 1, but I suspect this is mainly due to the nature of git commits. In April, 2014, we hit the magic mark where we had more Git commits than Subversion commits.”
Finally, Gruno encourages projects to make the most out of Git if they’re going to switch (or even if they don’t). “If projects do move to git, they really really really should take advantage of our GitHub integration! It opens up ASF to a whole new bunch of wonderful people and for a lot of projects, it results in a ton of new ideas, pull requests, comments, you name it. We can enable GitHub integration for any project (even Subversion projects) quite quickly.”
Want to learn more about using Git at the ASF? There will be a session on Git at ApacheCon Europe for attendees to learn more about using Git with Apache projects.
