Home Blog Page 1365

How Amazon Web Services Uses Linux and Open Source

Chris Schlaeger, AWSAmazon Web Services first launched in 2006 with one instance and one operating system: Amazon Linux. The cloud computing giant has since expanded to offer customers the option of running on more than 30 instance types and more than 10 operating systems, but Linux, Xen and other open source projects remain the core technologies behind AWS.

“We view open source as a companion to AWS’s business model,” said Chris Schlaeger, director of software development at Amazon Web Services and managing director of the Amazon Development Center Germany GmbH.

Schlaeger, who leads the team that develops Amazon’s cloud computing service, EC2, as well as Amazon Linux, will give a keynote presentation at CloudOpen Europe, Oct. 13-15 in Dusseldorf, Germany. Here he discusses how AWS uses Linux, the Amazon Linux operating system, the company’s new development offices in Germany, and what he’ll cover in his keynote presentation.

Linux.com: What are the building blocks of AWS? What role does Linux play?

Chris Schlaeger: Amazon Web Services is built on two fundamental services: S3 for storage services and EC2 for compute services. These were the first services AWS launched. Since launching in 2006 we have added more and more services that build on top of each other every year. In 2011 we added 80 new features and services, in 2012 it was 160. In 2013 it was 280 and we are already close to hitting this number for this year. Linux, in the form of Amazon Linux as well as Xen are fundamental technologies for AWS.

Since first launching in 2006 with one instance type, and one operating system, Amazon Linux, Amazon EC2 has now grown to more than 30 Instance types and more than 10 Operating Systems, both Linux and Windows, available from 10 infrastructure Regions in seven countries around the world. Customers now use Amazon EC2 for everything from media and web hosting to development and test environments, High Performance Computing to running ERP systems, running complex enterprise applications such as Oracle, SAP, IBM and Microsoft to Big Data analysis, and all using the same pay-as-you-go model with no long-term commitments.

What is Amazon Linux? How has Amazon customized the OS for its own purposes?

Amazon Linux is AWS’s own flavor of a Linux operating system. Customers using our EC2 service and all the services running on EC2 can use Amazon Linux as their operating system of choice. Over the years we have customized Amazon Linux based on the needs of AWS customers. We do still strive to be RHEL compatible to ease the customer transition to Amazon Linux. As my teams also develop the EC2 instance types we can ensure that Amazon Linux is the most optimized version of Linux to use within EC2.

AWS has a new development presence in Germany, can you tell us what you’re up to there?

Since we launched the business in 2006 we have seen rapid growth in the services and AWS now has hundreds of thousands of customers in over 190 countries. Many of the most well-known start-ups in Germany, such as Wooga, OneFootball and Soundcloud use AWS as well as many well-known German enterprises such as Kaercher, Talanx AG and Siemens Healthcare.

In terms of our physical presence, we have offices in Munich, Berlin and Dresden. In May last year we opened the Amazon Development Center Germany GmbH. It has currently 2 major development sites in Berlin and Dresden. The teams in Berlin are focusing on Machine Learning while the Dresden teams are working on the Linux Kernel, Xen, EC2 instances and Amazon Linux. I’m very excited about how rapidly the Dresden site has grown over the last year or so. We’ve just added another floor to host even more Linux and Xen developers.

How does Amazon contribute to the Linux kernel and open source in general?

We view open source as a companion to AWS’s business model. We use open source and have built many AWS services on top of open source technologies, like the Linux Kernel, Xen and MySQL. We also contribute to several open source projects.

If we find a bug in the Linux Kernel we usually send the patch upstream. It’s much easier for us to ingest such fixes with the next Linux Kernel release than having to maintain our own patch sets with bug fixes.

Can you give us a short overview of what you’ll discuss in your keynote at LinuxCon and CloudOpen Europe?

In the Keynote, I will provide an overview of some key AWS services and how they use Linux and Xen. I’ll provide some insight into Amazon Linux and its development. This will include, among other things, an introduction to our Germany-based development activities.

Register now for CloudOpen Europe.

Linux-Ready Modules Support Range of Xilinx FPGAs

Avnet released a series of four Linux-ready “PicoZed” COMs based on Xilinx’s Zynq-7000 SoCs, which integrate dual ARM Cortex-A9 cores along with FPGAs. We’ve flip-flopped between calling Avnet’s MicroZed boards computer-on-modules or single board computers. In fact, the compact boards reflect aspects of both, and are much more COM-like than Avnet’s original Zynq-based ZedBoard. Now, […]

Read more at LinuxGizmos

HP’s Bill Hilf Discusses Helion

HP’s Senior VP for HP Cloud chats about what his group is doing and where it is going.

Apache Hadoop Transitions to Git

Hadoop logo The Apache Infrastructure team has gotten Git migrations down pat. Just ask the Apache Hadoop project, which moved from Subversion to Git in less than 10 days.

Hadoop committer Karthik Kambatla says that getting from the request to Infra to a usable repo “took less than 10 days. The actual migration itself took 3-4 days. Daniel (Gruno, an Infrastructure team member) smartly suggested we start the process on a weekend to minimize disruption… he updated us fairly frequently on the status and made useful suggestions. There were a couple of follow-up items that were fixed fairly quickly as well.”

Git is an Option?

Chris Douglas, PMC Chair for Hadoop, says that Git wasn’t an option when Hadoop spun out of Lucene. “At least, there was no discussion about it. Git wasn’t available, in any case.”

In fact, some projects may not even be aware that it’s an option today. Git is still labeled “work in progress” (WIP), and it may not be obvious that Infra can switch projects to git. David Nalley, VP of Infrastructure at the Apache Software Foundation (ASF), explains that the “WIP” label is just that. “With git used by so many projects, it’s just a label at this point. Git is fully supported and in production at the ASF.”

“Historically, the original git.apache.org service was a svn-to-git mirror, and today that also provides the mirroring capabilities to Github,” says Nalley. “There is a project slated to merge what is git-wip and git.apache.org into a single service to rid us of the WIP moniker.”

Reasons to Switch

Kambatla says that the Hadoop project wanted to switch for a number of reasons. First, he says, “most users and developers were using git for development for a number of reasons: local commits, easy patch updates against latest committed versions, sharing code with others, etc. SVN, in my opinion was being used primarily only to commit code. Using git would avoid this duplication.”

Secondly, Kambatla says that working on feature branches “is easier with git, to keep up with the work on main branches.” And finally, Git provides the “potential for better code review tools.”

Douglas agrees that Git was a better fit for Hadoop. “Hadoop often has several active development branches that require backporting features. I don’t know if subversion added better support for this workflow subsequently, but git made it much easier to manage multiple patches, branches, and review. Because Hadoop uses review-then-commit (RTC), one often has multiple patches in flight that require quick context switches. So most developers had already switched to git for their work on Hadoop.”

Git does introduce some feature trade-offs, though. Douglas notes that the authorization is less fine-grained with git. However, he expresses little concern there. “Our experiments with branch committers suggest that we won’t regret relaxing the strict authorization we exercised with Subversion.”

Daniel Gruno, who performed Hadoop’s migration, says that code provenance remains intact but git isn’t SVN. “There are a few things like property settings that are lost, and the overall structure of a repository changes, which can make it somewhat difficult to browse what happened in the past – until you suddenly learn how it’s all set up, and then it gets easy.”

Mechanics of Switching

Though Kambatla and Douglas make it sound simple, the actual migration from SVN to git isn’t trivial. First, says Gruno, it involves “a lot of paperwork. We have to be very careful when we migrate, and so we have a set of rules we always follow.”

The process involves locking the SVN repository, “so someone doesn’t commit something while we migrate and it gets lost or messes up the migration,” says Gruno. “Once a project decides to move to Git, the Subversion repository is basically voided.”

The physical migration requires “a big ol’ Perl-based system” that “takes ages to complete” says Gruno. Ages, as in “1.5 hours per 1,000 commits.” Hadoop took more than two days to migrate, and he says that OpenOffice took more than a week. Hence the weekend migration plan.

After the migration, the repo is put into read-only mode so the PMC can “inspect” it. After the project’s PMC signs off, write access is allowed. And, finally, the old SVN repository is “partially opened” for write access so the project can update its website.

Git by the Numbers

Gruno says that quite a few projects are currently using git as their Version Control System (VCS) of choice. Out of 151 Top-Level Projects (TLPs), 69 are using Git as the primary VCS (Gruno points out that all Web sites are using SVN), as well as 19 podlings in the Apache Incubator (out of around 30 incubating projects). So, projects at the ASF are about evenly split in terms of which projects use Git and which use Subversion.

Furthermore, the git repositories are pretty active, says Gruno. “When commits are concerned, Git usually outnumbers subversion commits by 1.5/2 to 1, but I suspect this is mainly due to the nature of git commits. In April, 2014, we hit the magic mark where we had more Git commits than Subversion commits.”

Finally, Gruno encourages projects to make the most out of Git if they’re going to switch (or even if they don’t). “If projects do move to git, they really really really should take advantage of our GitHub integration! It opens up ASF to a whole new bunch of wonderful people and for a lot of projects, it results in a ton of new ideas, pull requests, comments, you name it. We can enable GitHub integration for any project (even Subversion projects) quite quickly.”

Want to learn more about using Git at the ASF? There will be a session on Git at ApacheCon Europe for attendees to learn more about using Git with Apache projects.

Linus Torvalds Says Linux Binary Packages Are Terrible, Valve Might Save the Desktop

The fact that most of the major distributions use different binaries is a well know problem, one that is not going to be solved anytime soon. The number of problems that need to be overcome in this regard is simply too great, but it’s not impossible.

Linus Torvalds was invited to a Q&A at DebConf14 (Debian Developer Conference), which took place in Portland, USA. The Linux kernel developer and maintainer had a talk with the participants and made some interesting remarks about the current state of the binaries and the future of this thorny issue.

Read more at Softpedia.

Why Did Docker Catch on Quickly and Why is it so Interesting?

Docker has rapidly become “the new thing” for computing in 2014. Why did that happen so quickly, and how has a Linux application container become so popular? Broadly, I think Docker is a great example of how to build a viral, developer-oriented product.

The rapid adoption rate took everyone by surprise, and now it’s too late to build a competitor. So everyone is having to cooperate around a single container solution. This is great for the developers and end users, but means that several platform as a service (PaaS) vendors have lost control of their destiny somewhat as Docker did an end-run around their strategy.

Read more at The New Stack.

systemd vs sysVinit Linux Cheatsheet

LinOxide: systemd is the new init system, started with Fedora and now adopted in many distributions like Red Hat, Suse and Centos.  Here’s a cheat sheet comparing commands in systemd to sysVinit.

Read more at LinOxide.

Xen & Docker: Made for Each Other!

By Olivier Lambert, Creator of Xen Orchestra Project

Containers and hypervisors are often seen as competing technologies – enemies even. But in reality the two technologies are complementary and increasingly used together by developers and admins. This recent Linux.com article talked about this supposed battle, noting however that developers are using Docker in traditional VMs to bolster security. Containers allow users to develop and deploy a variety of applications with incredible efficiency, while virtualization eliminates any constraints and/or exposure to outside attacks.

Uniting these technologies helps developers and system administrators be even more efficient. Let’s take a closer look at how to achieve this with Docker and Xen Project virtualization, and why we expect more and more organizations to use them together in the near future. This will also be a key topic at the September 15 Xen Project User Summit at the Lighthouse Executive Conference Center in New York City. Register today to learn more about enabling Docker in Xen environments for a truly open infrastructure.

Who’s Who: What is Xen Project Virtualization?

Xen Project Hypervisor is mature virtualization technology used by many of the world’s largest cloud providers like AWS, Verizon Terremark, Rackspace and many more. Founded in 2003, Xen Project virtualization is proven as a highly reliable, efficient and flexible hypervisor for arange of environments, running perfectly from x86 to ARM.

It’s now completely integrated in the Linux upstream and is hosted by the Linux Foundation. The same big cloud users mentioned above also contribute regularly to the project along with many of the world’s largest technology companies, including Citrix, Cavium, Intel, Oracle and more.

Feature updates and broader community collaboration are on the upswing too: more commits, more communication, better integration, new use cases and simpler and more powerful modes, such as PVHVM then PVH, as outlined in this recent blog.

The core Xen Project team takes security seriously. The technology has also been battle-tested by many in the defense industry including the NSA. Xen Project users have benefited from this for years, and developers building, shipping and running distributed applications will profit as well.

What is XenServer and Xen Orchestra?

XenServer is a packaged product consisting of the Xen Project Hypervisor and the Xen Project Management API (XAPI) toolstack within a performance tuned CentOS distribution. It’s free and can be installed in just a few minutes; click here to download it: http://xenserver.org/open-source-virtualization-download.html.

Xen Orchestra (XO) is a simple but powerful web interface working out-of-the-box with XenServer, or any host with Xen and XAPI (the most advanced API for Xen). Take a look on the project website to learn more. Both of these tool are of course free software.

What is Docker?

In its own words, Docker defines itself as an open platform for developers and sysadmins to build, ship, and run distributed applications. Consisting of Docker Engine, a portable, lightweight runtime and packaging tool, and Docker Hub, a cloud service for sharing applications and automating workflows, Docker enables apps to be quickly assembled from components and eliminates the friction between development, QA, and production environments.

Main Advantages:

  • fast (boot a container in milliseconds)
  • simple to use, even in complex workflows
  • light (same kernel)
  • container density on one host

The other side of the coin:

  • all containers rely on the same kernel (isolation, security)
  • less maturity than traditional hypervisor (Docker is still young)
  • containers are using the same OS on the host (less diversity than hypervisors)
  • some friction between developers and admins about its usage: not Docker’s fault, more a classical friction when you bring new toys to your devs.  We’ll see why and how to cope with just that below.

Best of Both Worlds

An ideal world would:

  • Let admins do their admin stuff without constraints and/or exposure to dangerous things.
  • Let developers do their developer stuff without constraints and/or exposure to dangerous things.

Fluid Workflow

In other words, they’d be able to create really cool workflows. For example:

  • An admin should be able to easily create a Docker ready VM running in a hypervisor, with the exact amount of resources needed at a given point in time (he knows the total amount of resources, e.g a VM with 2 CPUs and 4GB of RAM.
  • He should delegate (with the same simplicity) this Docker-ready VM to the dev team.
  • Developers can use it and play with their new toy, without any chance of breaking stuff other than the VM itself. The VM is actually a sandbox, not a jail; developers can create their containers as they need in this scenario.

Now you can easily imagine other exciting things such as:

  • An admin can delegate rollback snapshot control to a developer. If he breaks the VM, he can rollback to the “clean” snapshot — without bothering the admin staff. Live, die, repeat!
  • Need to clone the same container for other tests? One click in a web interface.
  • Need to extend the resources of this current VM? One click, in live.
  • Ideally, let a developer create its container from the same web interface.

Xen Orchestra: A Bridge Between Docker and Xen Project Hypervisor 

So how do we do all this without creating a brand new tool? As you may guess, the answer is Xen Orchestra, which today achieves much of this. Updates planned for later this year and 2015 will deliver even more efficiencies between the two technologies.

What XO Does Today

  • Adjust Resources In Live: You can reduce/raise number of CPUs, RAM, etc., while the VM is running! Doing this, you can grow or reduce the footprint of your Docker VM, without interrupting the service. Check it out in this short video.
  • Snapshots and Rollback: Snapshots and rollback in XO are totally operational since XO 3.3. Check out how this works in this feature presentation. Coupled with Docker, this is very helpful. When your fresh Dockerized VM is ready, take a snapshot. Then you can rollback when you want to retrieve this clean state. All with just a few clicks and in a few seconds.

Coming Soon

  • Docker-Ready Templates in One Click: This feature will be released this year. In a few words, you can request our template directly from your XO interface, it will be downloaded and operational in your own infrastructure with a Docker listening and ready for action,Iin the resources you choose to allocate (CPU, RAM, Disk). No installation: It works out of the box. Read more in this article.
  • ACL and Delegation: The perfect workflow rest upon integration of ACLs in Xen Orchestra is our current priority. In our case, it allows VM delegation for your team using Docker; the VM can be rollbacked or rebooted without asking you. More info.here.
  • Docker Control from XO: Because we can get the IP of a VM thanks to its Xen tools, we should be able to send command to the Docker API directly through XO. In this way, you’ll just have to use one interface for Docker AND Xen (at least, for simple Docker operations). And take the best of XO for both: ACLs, visualization etc. This last feature is not in our current roadmap, but will probably pop up early in 2015! 

Conclusion

Docker is a really promising and growing technology. With Docker and Xen on the same team, the two technologies work in tandem to create an extremely efficient, best-of-breed infrastructure. Finally uniting them in one interface is a big leap ahead!

Any questions or comments? Go ahead!

By Olivier Lambert, Creator of Xen Orchestra Project

FSF and Debian Join Forces to Help Free Software Users Find Hardware

The Free Software Foundation (FSF) and the Debian project today announced cooperation to expand and enhance h-node [1], a database to help users learn and share information about computers that work with
free software operating systems.

While other databases list hardware that is technically compatible with GNU/Linux, h-node lists hardware as compatible only if it does not require any proprietary software or firmware. Information about hardware that flunks this test is also included, so users know what to avoid.

Read more at Debian Weekly News

Rackspace: Can it Outrun Customer Uncertainty as CenturyLink, Others Eye Acquisition?

The Rackspace “strategic alternatives” process needs to be resolved as soon as possible or landing deals may become more difficult.