Home News Enterprise Computing Cloud Computing Containers vs Hypervisors: The Battle Has Just Begun

Containers vs Hypervisors: The Battle Has Just Begun

xen architectureLet me make one thing clear up front: I like Docker. I think it is good tech. I can see all sorts of scenarios when a lightweight, containerized deployment mechanism could be really nifty, even if it doesn't have the advantage of a hypervisor under it to protect its host operating system. I get it. It's cool.

But there are a few loud, but lonely, voices in the crowd who proclaim, “The battle is over! Containers have won! Hypervisors are obsolete!”

I'm sorry, that's just wrong.

That's like a championship fighter who upon landing a couple good blows on his opponent in the first round yells, “Stop the fight! I've won! Crown me champion!”

We all know it doesn't work that way. Ultimately, the championship goes to the one who wins the battle. And, in this case, the battle isn't over; it's just beginning.

The Case for Containers

The concepts behind Docker and other Linux containers are solid:

  • Very small VMs that allow for much higher server density by removing redundant or unnecessary operating system elements from the VMs themselves.

  • Nicely packaged VM stacks, which can easily be transferred, replicated, and controlled, ensuring high levels of portability.

  • VM software stacks that are small, removing the problem and tedium of building a large stack of version-specific operating systems and tools that need lots of care and feeding to replicate and maintain.

  • Extremely fast startup times that can facilitate a more flexible infrastructure, allowing greater latitude to respond to the needs of the moment – literally.

The challenge, however, is that the security attack surface of a “shared kernel” strategy has its weakest link in that “shared kernel” itself. If one malicious hacker manages to violate that shared kernel, all instances that employ that shared kernel are potentially compromised.

Certainly, a similar argument can be made of traditional hypervisors – if you can violate the hypervisor, you might be able to violate the VMs it powers – but the industry has had many years of experience hardening hypervisor installations. While it is not a task for slackers, it is not rocket science either. Hundreds of successful virtualization and cloud providers in the world attest to the manageability of the task; from giants like Amazon, Rackspace, Verizon, and Huawei, down to smaller local and boutique-style service providers, the names of which are far less well known.

Some voices asserting the supremacy of containers cite Google as the proof case. By architecting the Google Docs application stack to work safely on containers, Google has shown the way, they say.

Except Google is Google. They can afford to hire thousands of the best and brightest to do intelligent things that few others can do. After 30 years in this industry, and two decades dealing with customers on site, I doubt that most organizations could readily do what Google has done. If they could, they'd be Google, too.

The ideal solution is to provide the benefits of containers while actually reducing the attack surface. In the age of the cloud, systems need a higher degree of security than ever before. The shared kernel scenario simply doesn't provide that.

The situation is so critical, in fact, that multiple Docker power users announced at LinuxCon North America in Chicago that they use Docker in traditional VMs to bolster security. Even the recent announcements from new Docker partner VMware highlight the value of running Docker in VMware's VMs to bring “security capabilities to container environments.” Suddenly, the announcements of the death of the hypervisor seem entirely premature.

The Unikernel, Power for Security-Sensitive Cloud

However, there is a technology on the rise which is far more promising where security is needed: the unikernel.

Sometimes called “cloud operating systems” or “library operating systems,” unikernels combine many of the advantages of Docker-like container systems with the security footprint of hypervisors and a much smaller attack surface within each VM.

Unikernel systems create tiny VMs. Mirage OS from the Xen Project incubator, for example, has created several network devices that run kilobytes in size (yes, that's “kilobytes” – when was the last time you heard of any VM under a megabyte?). They can get that small because the VM itself does not contain a general-purpose operating system per se, but rather a specially built piece of code that exposes only those operating system functions required by the application.

There is no multi-user operating environment, no shell scripts, and no massive library of utilities to take up room – or to subvert in some nefarious exploit. There is just enough code to make the application run, and precious little for a malefactor to leverage. And in unikernels like Mirage OS, all the code that is present is statically type-safe, from the applications stack all the way down to the device drivers themselves. It's not the “end-all be-all” of security, but it is certainly heading in the right direction.

The number of unikernel-type systems today is still small, and their names are not yet well known. In addition to Mirage OS from Xen Project, there are others like OSv from Cloudius Systems, HaLVM from Galois Systems, NEC's ClickOS, and LING (formerly known as “Erlang on Xen”). They provide environments which bind to languages such as Java, C, OCaml, Haskell, and Erlang. And they greatly simplify both the development and deployment processes surrounding the unikernel solution stack.

They are young – and they are the beginning.

I fully expect that five years from now we will look back at the unikernels of 2014 and see these as the seedlings of what will be a growing forest of unikernel-type systems. Unikernels raise the bar on the notion of a containerized VM by matching the touted packaging and deployment benefits while improving software development workflow and resource efficiency. And, perhaps most importantly of all, it does all this while reducing the external attack surface through operating environment specialization. Frankly, I can't wait to see what will develop in this space.

“The battle is over?” No chance. We're not even done with Round 1. It’s just beginning to get to really interesting.  



Subscribe to Comments Feed
  • James Mak Said:

    So, isn't a unikernel on top of Xen equivalent to a process on top of Linux? In both cases you have a control plane that if exploited would allow access to the physical machine. I'm struggling to understand the difference. Wouldn't we be better off reducing the effects of a root level exploit via capabilities or other security enhancing techniques?

  • Russell Pavlicek Said:

    James, The attack surface of a Type 1 hypervisor like Xen Project is MUCH smaller than the attack surface of a multi-user operating system like Linux. This is verified by inspection and by history. George Dunlap did a nice talk about this at LinuxCon NA in Chicago earlier this month showing that a security review of compromises encountered in the past year for Xen Project, KVM, and Qemu were less than the potential container compromises found in the last 2 months. Also, the attack surface of unikernel VMs is EXCEPTIONALLY small. Since there is no shell, no generalized operating environment, and (in most cases) no exposure of library functions except those required by the payload, the ability to penetrate the VM and then exploit it are greatly reduced. About securing containers: A main selling point of Docker is its easy deployment model, which is quite good. If you have to manually secure each instance you deploy with all sorts of security methods, then the advantage is gone -- especially since a slip-up may cost you every container instance on that host. I've been told that Google deploys externally-facing containers in VMs for security (someone said they heard that in a session at LinuxCon NA). Docker's new partner VMware says to deply Docker in VMs for security. I had someone else tell me that one of the Docker engineers is on record saying that deployment in VMs is recommended for security (but I don't have a link to any such statement or I would have used it in the article). The Docker people seem to have a good view of their tech; it's these other voices who make loud and unjustified prognostications about the impact of Docker which cause the problem. Use Docker when it makes sense. Use Docker in VMs when you need security. And definitely explore the new world of unikernels for large-scale, small footprint, higher security VMs where they make sense. But, whatever you use today, I bet it will all look very different 5 years from now as the various Open Source projects innovate in this space. More fun to come!

  • Jon Forrest Said:

    Nice article but you manage to confuse the issue by using the term "VM" when you're talking about the Case for Containers. For example, you said: "Very small VMs that allow for much higher server density by removing redundant or unnecessary operating system elements from the VMs themselves." You should have said: "Very small containers that allow for much higher server density by removing redundant or unnecessary operating system elements from the containers themselves." The same is true for the other statements in that section that mention "VMs". There's enough confusion in the world as it is. Why add to it? Jon Forrest

  • Russell Pavlicek Said:

    Jon, My error. I should have been clearer.

  • Lennie Said:

    It's obviously a bit more complicated. Google deploys their whole infrastructure on containers. All the services like gmail, search, etc. Recent Linux kernel can run containers in a much more secure way. But the Docker developers are not using all the features because they want people to be able to Docker on a very large number of existing Linux installations and distributions. Because of that and because there is a lot of other things that are in being developed they are also not in a hurry to make it available to people on the newer kernels, because when they've done so they'll probably be stuck with the interface they choose. Also if they say it's secure a whole lot of people will start to offer Docker services in a multi-tenant way. Then if they are wrong they will get a bad name. They would rather be pragmetic. Here is are the slides of a talk that explains all the (if you don't like reading: slide 56 has a dummary) Article on namespaces: User namespaces is the last part that was needed. Only starting to look secure in very recent kernels. Like since the latest kernel release 3.16, maybe it was 3.15. Now to get back to Google, when you buy Google services to run your own code on the Google infrastructure. For example as a Docker container, they'll keep it inside a VM. But that VM is launched from within a container like everything else in their infrastructure. So if you run Docker on Google App Engine. You are actually running container in VM in container. It's all about the layers. Better to be save then sorry.

  • Sandeep Puri Said:

    I have to say, polyglot programming is a big deal these days.. the unikernels that you talk about are mostly single language frameworks, which may not see much uptake.

  • Russell Pavlicek Said:

    Folks in the New York area can learn more about unikernels at Xen Project User Summit on September 15. We have two unikernel talks on the schedule -- "Unikernels: Who, What, Where, When, and Why?" and "Deploying C and Java applications directly on Xen, with OSv". You don't need to be a current Xen Project user to attend; people who are looking at virtualization options are welcome to join us. readers can use code "Xenuser25" for 25% off.

  • Russell Pavlicek Said:

    Folks in the New York area can learn more about unikernels at Xen Project User Summit on September 15. We have two unikernel talks on the schedule -- "Unikernels: Who, What, Where, When, and Why?" and "Deploying C and Java applications directly on Xen, with OSv". You don't need to be a current Xen Project user to attend; people who are looking at virtualization options are welcome to join us. readers can use code "Xenuser25" for 25% off.

  • Hamid Said:

    Russel, your title "Containers vs Hypervisors: The Battle Has Just Begun " is concentrated to just some value points of using hypervisors / containers. In most of use scenarios, you need more of better performance and resource usage and comfortable execution rather than just some more theoretical security strengths. Personally I'd rather to use some combination of container facilities of Linux , e.g namespaces, suitable to my work and use abilities of kernel for all my instances of workloads instead of just replication of OS capabilities in essence and under shadow of of more security specially in theory. In your context of theorem (running under containers which are run under hypervisors), it is more better to have nested virtualization in multiple layers more and more, to have better security. But the question is until when? And the potential answer could be till we pay all of performance to have *near* all security.

  • BillDStrong Said:

    Xen is not a type 1 hypervisor, because it requires an external kernel, i.e. Linux, to run. It is a type two hypervisor, or at most a type 1.5. This make it a poor choice when it comes for the very security applications you are promoting their use in. A better choice is a true hypervisor, such as the seL4 kernel that has less than 20K Loc, and is mathematically verified at the code level for all systems, and to the binary level for certain arm system. Compare this to Linux with multiple million Loc, with no verification, then add the Xen code to that, and you see that this is not the best starting platform. However, there is nothing stopping anyone from running some of these unikernel systems on top of the much more secure seL4 systems, and this may indeed be the future of high security systems.

  • Russell Pavlicek Said:

    I know of no definition of a "Type 1" hypervisor which Xen Project does not fulfill. It does not use a host kernel to run, like KVM does. It runs on bare metal, and only gets driver support from either Domain 0 or from disaggregated driver domains. If you look at one of the overview slide decks on, you can see diagrams of how the Xen Project architecture works. It is entirely possible to have a Xen Project host which has no Linux presence whatsoever.

  • Edelweiss Said:

    There is a very good presentation which lays out what good use-cases for unikernels are ... for emerging technologies, such as NFV, IoT and other emerging areas they look like a great fit. In those segments, applications such as LINC & LINCX (see are actually starting to appear. OSv - see - is taking a different approach, targeting existing applications rather than the likes of MirageOS, ErlangOnXen, ...

Upcoming Linux Foundation Courses

  1. LFS201 Essentials of System Administration
    12 Jan » 30 Mar - Online Self-Paced
  2. LFS220 Linux System Administration
    18 May » 21 May - Virtual (GUARANTEED TO RUN)
  3. LFS230 Linux Network Management
    25 May » 28 May - Virtual

View All Upcoming Courses

Become an Individual Member
Check out the Friday Funnies

Sign Up For the Newsletter

Who we are ?

The Linux Foundation is a non-profit consortium dedicated to the growth of Linux.

More About the foundation...

Frequent Questions

Join / Linux Training / Board