Steering Kubernetes Through Uncharted Territory

By

-

October 25, 2016

Taylor Thomas is a Cloud Software Engineer for the Software Defined Infrastructure team at Intel working on Kubernetes, CI/CD, and the Snap open telemetry framework. The team also uses Kubernetes to run a large part of their services, and Thomas will describe this work in his upcoming talk “Off the Beaten Path: An Explorer’s Guide to Kubernetes ” at KubeCon. In this article, however, he provides a preview of some challenges that the team has encountered.

Taylor Thomas, Cloud Software Engineer at Intel

Linux.com: What are some examples of uncharted territory that your team has encountered?

Taylor Thomas: My team works in research and development, and we use Kubernetes to run a large part of our team’s services. Because of this work, we have found many examples of uncharted territory, and I’ll cover most of these in my presentation at KubeCon. However, there are two really good examples we encountered recently.

The first is how to use readiness probes with services that have both a “ready to synchronize with peers” and a “ready to accept requests” state. Readiness probes are a feature in Kubernetes that allow you to control when services are in a ready state based on customizable call-outs. If the container a service points to is not ready, the service will not route traffic to a container. By default, containers are considered to be in a ready state as soon as the process is running. In practice, this doesn’t account for the fact that services are not always ready to accept requests immediately on start because there is some initial configuration that needs to complete first. Our Cassandra instance is a perfect example of how this can be problematic. When we put in a readiness probe, it created a race condition where each node couldn’t be ready until it synchronized with its peers, but couldn’t synchronize with its peers until they were in a ready state.

ConfigMaps are Kubernetes objects that allow users to specify configuration data that can be mounted into a container at runtime. We ran into specific use cases where we wanted to mount multiple ConfigMaps into the same directory or when we wanted to place ConfigMaps into a directory with other config files. When we updated to Kubernetes 1.3, all of a sudden, we found that all of our files were no longer mounted. This prevented Jenkins from starting with the proper configuration.

Linux.com: How did you address these issues?

Taylor: For the readiness probes, after some digging, we found you can add an annotation to the pod like this:

service.alpha.kubernetes.io/tolerate-unready-endpoints: “true”

When we created a separate service for the Cassandra inter-node communication used for initialization with this annotation, everything started up again and the readiness probes worked perfectly.

Our workaround for the mounted file involved a little imagination, but it has worked extremely well. In our Jenkins container startup script, we created a dead symlink to a separate path in the container where we will mount the config file.This allowed us to put each config file into it’s own directory as the symlink points to the correct location.

Linux.com: How can the documenting of solutions ease future pain points?

Taylor: Both of the problems I mentioned did not have any documentation. In the readiness probe example, we had to search around until we found a GitHub issue with the same problem. This then pointed to a PR where the annotation had been merged. In both examples, it took time to diagnose the issue (which can be difficult with all the moving parts of Kubernetes) and then search for a solution to the problem. Had these been documented, both issues would take less than 10 minutes to fix. Documentation helps everyone reduce the time spent troubleshooting and searching.

Linux.com: What should companies consider before installing/deploying Kubernetes?

Taylor: Kubernetes abstracts away many of the pain points of having to orchestrate containers. However, it creates new problems to solve (and document!) and forces you to adopt a new paradigm. It comes down to the common problem of the value you gain from it. In our case, it has sped up our ability to iterate and release features without impacting users and has made it easy to spin up something as complex as Cassandra or Jenkins in a matter of seconds. It has come at the expense of a lot of learning and researching. So, if you are just hosting a webapp or other simple application, the added complexity may not be worth it. If things are bigger though and you need the ability to scale, I would highly recommend giving Kubernetes a try!

Linux.com: Have the complexities of Kubernetes increased with its popularity?

Taylor: With more users always comes more feature requests, so the simple answer to this is yes, it has. Even so, I don’t find that a problem. Adam Jacob at Chef recently gave a great talk I loved in which he talked about embracing complexity as long as it brings ease. So the question I like to ask is “Does Kubernetes make things work with ease?” To that, I would say yes. It is still growing and there are still issues to be solved, but it has given us a scalable system we wouldn’t have had otherwise.

Registration for this event is sold out, but you can still watch the keynotes via livestream and catch the session recordings on CNCF’s YouTube channel. Sign up for the livestream now.

What are the Top NFV Risks for Carriers?

By

SDx Central

-

October 25, 2016

What are the risks of network functions virtualization (NFV)? As with any emerging technology, moving fast or picking the wrong components can do more harm than good. Let’s spend some time breaking down the NFV risks in building a virtual network. I have spent the few months gathering feedback from various service providers to get their view on whether NFV and its cousin software-defined networking (SDN) are ready for prime time. Even though many service providers expressed optimism that NFV technology is moving toward maturity, there are definitely cautionary tales on what to look out for.

This article serves as an introduction to the challenges of NFV component selection – later articles will refer in more detail to the challenges in selecting NFV hardware and software components such as OpenStack and Open vSwitch.

5 More Reasons to Love Kubernetes

By

Sebastien Goasguen

-

October 25, 2016

In part one of this series, I covered my top five reasons to love Kubernetes, the open source container orchestration platform created by Google. Kubernetes was donated to the Cloud Native Computing Foundation in July of 2015, where it is now under development by dozens of companies including Canonical, CoreOS, Red Hat, and more.

My first five reasons were primarily about the project’s heritage, ease of use, and ramp-up. The next five get more technical. As I mentioned in part one, choosing a distributed system to perform tasks in a datacenter is much more complex than looking at a spreadsheet of features or performance. And, you should make your decision based on your own needs and team dynamics. However, this top 10 list will give you my perspective, as someone who has been using, testing, and developing systems for a while now.

#6 Rolling Updates and Rollbacks

Rolling updates and rollback are absolutely necessary features in application management. Especially when we start embracing extremely quick development cycles and continuous innovation. Having a system that has these features not only built-in but also thought out as part of how the system works is huge.

Deployment resources are new in Kubernetes — at first Kubernetes had Replication Controllers (RC). These were resources that defined a declarative state of your Pods — meaning, which containers and how many do I want in my system at all times. Rolling updates were implemented with replication controllers but were client side, which was problematic when the client shut down.

Therefore, Kubernetes introduced the Deployment resource and replaced replication controllers with replica sets (part of a bigger renaming of various resources). Every time a deployment is modified it creates a new replica set. Scaling up and down the replica sets of that Deployment gives you rolling update and the ability to roll backs. Indeed, old replica sets are not deleted but remain in the system and are part of the history of a Deployment.

The scaling and deployment strategy of a Deployment can be tweaked in the Deployment manifest.

And all of this, of course, is triggered by simple HTTP calls to a REST API.

Let’s check the history of a simple deployment:

```

$ kubectl rollout history deployment ghost

deployments "ghost":

REVISION    CHANGE-CAUSE

1        kubectl run ghost --image=ghost --record

2        kubectl edit deployment/ghost

```

Here it shows that one update was made. We can rollback that change with rollout undo. This will increment the revisions in the history.

```

$ kubectl rollout undo deployment ghost

deployment "ghost" rolled back

$ kubectl rollout history deployment ghost

deployments "ghost":

REVISION    CHANGE-CAUSE

2        kubectl edit deployment/ghost

3        kubectl run ghost --image=ghost --record

```

Bottom line, edit a Deployment and you get a rolling update. Rollback with a rollout undo, and, yes, you can rollback to a specific revision.

#7 Quotas

In open source, there have been lots of business model variants during the past 15 years. One of them has been and still is to make certain features commercial add-ons/plugins. Quotas that limit the use of resources in a system are sometimes paid add-ons.

In Kubernetes, quotas are built in. They can be used to limit the number of API resources or limit the use of physical resources like cpus and memory. Similarly to Pods and Deployments, quotas are API resources in k8s. You define them in yaml or json files and create them in your cluster using kubectl.

For example to limit the number of Pods to 1 in a specific namespace, you would define a ResourceQuota like this:

```

apiVersion: v1

kind: ResourceQuota

metadata:

 name: object-counts

 namespace: default

spec:

 hard:

   pods: "1"

```

You will be able to see and modify the quota like any other resource with the kubectl get and kubectl edit commands:

```

$ kubectl get resourcequota

NAME            AGE

object-counts   15s

```

With a single Pod running, if you try to create a new one, k8s will return an error saying that you are over quota:


```

$ kubectl create -f redis.yaml

Error from server: error when creating "redis.yaml": pods "redis" is forbidden: exceeded quota: object-counts, requested: pods=1, used: pods=1, limited: pods=1

```

Quotas are built in and first class citizen in the k8s API. Amazing!

#8 Third-Party Resources

This one is a bit more challenging to grasp as it is a new concept in most systems.

The philosophy of Kubernetes is that it should consist of a core set of API resources that are needed for managing containerized applications. It is foreseen that in the very short term, this core should be stable, stable as in no need for anything extra. Any additional API resource that users might need will not be added to the core, instead users will be able to create those resources on the fly. Kubernetes will manage them and the client will dynamically be able to use them. This technique is used at Pearson to manage databases.

The example I used in my LinuxCon talk, is to create a new API object called pinguin. You define it via a so-called ThirdParty Resource object. Like any other k8s resource it has metadata, apiversion, kind plus a set of versions. The metadata contains a name which defines a new resource group. Here it is:

```

metadata:

 name: pin-guin.k8s.linuxcon.com

apiVersion: extensions/v1beta1

kind: ThirdPartyResource

description: "A crazy pinguin at Linuxcon"

versions:

- name: v1

```

Let’s create this new resource:

```

$ kubectl create -f pinguins.yml

$ kubectl get thirdpartyresources

NAME                        DESCRIPTION                   VERSION(S)

pin-guin.k8s.linuxcon.com   A crazy pinguin at Linuxcon   v1

```

With this in place, you are now free to create a pinguin (to keep with the LinuxCon theme):

```

$ cat pinguin.yml

apiVersion: k8s.linuxcon.com/v1

kind: PinGuin

metadata:

 name: crazy

 labels:

   linuxcon: rocks

$ kubectl create -f pinguin.yml

```

And, dynamically, kubectl is now aware of the pinguin you created. Note that is only available in the latest version of k8s: v1.4.0.

```

$ kubectl get pinguin

NAME      LABELS           DATA

crazy     linuxcon=rocks   {"apiVersion":"k8s.linuxcon.com/v1","kind":"PinGui...

```

Now you will wonder what to do with that, and the answer is that you will need to write a controller. A piece of code that will watch for pinguins and perform actions when they get created, deleted, or modified.

This is purely awesome as it means that the Kubernetes API server is now fully extendable by each user.

#9 Role-Based Access Control (RBAC)

In addition to quotas, role based access control is a must have in enterprise systems. Similarly to quotas, it is also quite often an after-thought in data-center solutions when it is not a commercial add-on.

With Kubernetes, we now have fine-grained access control via roles (RBAC), and the best part is that, of course, it is 100% API driven. By that, I mean that roles and bindings are API resources that an administrator can write and create on the cluster like you would create Pods, deployments, etc.

It was first introduced in v1.3.0; it is an alpha API feature and still considered experimental, but couple more releases and I fully expect that it will be a stable API.

Roughly speaking, you create roles, API resources of kind role, and define some rules for each of these roles:

```

kind: Role

apiVersion: rbac.authorization.k8s.io/v1alpha1

metadata:

 namespace: default

 name: pod-reader

rules:

 - apiGroups: ["api/v1"]

   resources: ["pods"]

   verbs: ["get", "watch", "list"]

   nonResourceURLs: []

```

Then you associate users to these roles, by creating bindings resources of kind RoleBinding. A binding, takes a list of users — aka subjects — and associates them with a role. Once you create that binding, any users of the system will inherit the access rules defined in the role.

```

kind: RoleBinding

apiVersion: rbac.authorization.k8s.io/v1alpha1

metadata:

 name: admin

 namespace: default

subjects:

 - kind: ServiceAccount

   name: default

roleRef:

 kind: Role

 namespace: default

 name: admin

 apiVersion: rbac.authorization.k8s.io/v1alpha1

```

There is a great demo of this from Eric Chiang of coreOS on YouTube.

Built-in RBAC, fully API driven. What a joy!

#10 Cluster Federation

Last but not least is cluster federation.

Going back to the Borg paper and our first reason to love Kubernetes, you probably noticed that a single k8s cluster is in fact the equivalent of a single Borg “cell” or availability zone. Now in Kubernetes 1.4.0 is the ability to federate multiple clusters through a single control plane. That means that we now have Borg lite.

It is a key reason to like Kubernetes, as it will bring an Hybrid cloud solution for containers. Imagine having a k8s cluster on-premise and one in a public cloud (e.g., AWS, GKE, Azure). With this federated control plane, you will be able to launch microservices that could span multiple clusters. Scaling will automatically balance the replicas across clusters but still provide a single DNS endpoint, with load balancing also federated.

I find this super exciting, because we should be able to quickly migrate an app between on-prem and public cloud and vice-versa at will. Yes you heard that right, and this is all built-in, not a commercial add-on.

To get started with federation, Kelsey Hightower wrote a very detailed walk-through that is worth a try.

[Picture courtesy of coreOS]

And that’s it, you have it, my top 10 reasons to like Kubernetes. I am sure that others will have a different list as there is so much to like about this system. I have been using, testing and developing data-center solutions for a while now, and I feel like Kubernetes is really a well thought-out system, extremely stable, extensible and with some key features built-in that we have grown to expect as commercial add-ons. Truly a game changer.

So you’ve heard of Kubernetes but have no idea what it is or how it works? The Linux Foundation’s Kubernetes Fundamentals course will take you from zero to knowing how to deploy a containerized application and manipulate resources via the API. Sign up now!

Manage Disk Image Files Wisely in the Face of DevOps Sprawl

By

Tech Target

-

October 25, 2016

There are strategies to prevent obsolete disk images from clogging up valuable space, especially in a rapid-change DevOps shop. However, not all VMs will benefit from the same plan.

Disk images are fast and easy to create, but they’re not free and they cause administrative problems. Each image file, such as a .vhdx or .vmdk file, requires storage space. Virtualization or IT administrators must constantly manage disk image files to ensure each image is appropriate and relevant to the enterprise. They must archive or delete obsolete or unneeded disk image files to ease management and recover costly storage space.

These problems are multiplied in a modern DevOps environment. DevOps teams rely on continuous delivery models to speed development and experiment with creative and competitive capabilities that could yield the business an advantage.

The Point Of Docker Is More Than Containers

By

Forbes

-

October 25, 2016

Spending time with Docker during Cloud Field Day about a month ago opened my eyes to the larger ecosystem that Docker is building, and that others are building around it. There is so much more to Docker than just the idea of immutable containers.

For a start, Docker made using containers easy. That’s no small feat for a tricky piece of technical infrastructure. Making it easy, and specifically easy for developers, to use removed a lot of friction that was no small contributor to the pain of other, earlier methods. It gave developers are really simple way to create a fully functional development environment, isolated from all other dependencies, with which to work.

Using Wearables and Machine Learning to Help With Speech Disorders

By

D Zone

-

October 25, 2016

A team from MIT and the Massachusetts General Hospital believe that machine learning can play a part in better understanding speech disorders.

In a recent paper, they describe using a wearable device to collect accelerometer data to detect differences in people with Muscle Tension Dysphonia (MTD) and a control group. After such individuals with MTD had received therapy for the condition, their behaviors appeared to converge with that of the control group.

“We believe this approach could help detect disorders that are exacerbated by vocal misuse, and help to empirically measure the impact of voice therapy,” the authors say. “Our long-term goal is for such a system to be used to alert patients when they are using their voices in ways that could lead to problems.”

The Blockchain Created By Ethereum’s Fork is Forking Now

By

Coin Desk

-

October 25, 2016

A blockchain that was born out of the rejection of a contentious technical change is on the cusp of making a decision some argue contradicts its core values.

That’s the situation the developers behind ethereum classic face ahead of a hard fork expected to be enacted on its blockchain on 25th October (should network participants approve the upgrade). Originally formed in reaction to a decision by the ethereum community to edit its “immutable” ledger, the fork caused an ideological schism among its enthusiasts.

5 Basic curl Command Examples You Can Run on Your Linux Server

By

RoseHosting .com

-

October 25, 2016

cURL can be used in many different and interesting ways. With this tool you can download, upload and manage files, check your email address, or even update your status on some of the social media websites or check the weather outside. This article will introduce you to cURL with five basic command examples.

cURL is very useful command line tool to transfer data from or to a server. cURL supports various protocols like FILE, HTTP, HTTPS, IMAP, IMAPS, LDAP, DICT, LDAPS, TELNET, FTP, FTPS, GOPHER, RTMP, RTSP, SCP, SFTP, POP3, POP3S, SMB, SMBS, SMTP, SMTPS, and TFTP.

In this article will cover five of the most useful and basic uses of the cURL tool on any Linux VPS.

Click here to get the 5 cURL command examples that will get you more familiar with cURL.

CloudNativeCon Unites Leaders in Open Source, Container and Cloud Native Tech

By

Cloud Native Computing Foundation

-

October 24, 2016

Today’s cloud native ecosystem is growing at an incredibly rapid pace – as new technologies are continuously introduced and current applications are ever-evolving.

Taking the lead in bringing together the industry’s top developers, end users, and vendors, the Cloud Native Computing Foundation (CNCF) hosts critical components of the cloud native software stacks including Kubernetes, Prometheus, and OpenTracing and serves as a neutral home for collaboration.

To help spread cloud native practices and technology across the world, CNCF is hosting CloudNativeCon to bring together leading contributors in cloud native applications and computing, containers, microservices, central orchestration processing, and more November 8-9 in Seattle.

By co-locating with KubeCon and PrometheusDay, this conference will provide a platform for showcasing a full range of technologies that support the cloud native ecosystem and help bring associated communities together. Keynote speakers include Box’s Sam Ghods, Comcast’s Erik St. Martin, LightStep’s Ben Sigelman, Prometheus Expert Fabian Reinartz and Kubernetes Experts Kelsey Hightower and David Aronchick.

During KubeCon, attendees will have the opportunity to meet with leading Kubernetes technologists who work with one of the highest velocity open source projects – bringing together this quickly growing community to advance their knowledge of Kubernetes use with containers and cloud native architectures.

Similarly, PrometheusDay will feature technical talks covering major Prometheus adopters, leading expert contributor insights, and a full range of technologies that support open source monitoring technology in the cloud native ecosystem. From Percona to Shuttlecloud, Prometheus is helping these companies scale and the project itself is quickly growing as well. PrometheusDay will bring together this active and developing community to advance their knowledge of Prometheus use with cloud native architectures.

The event will feature sessions from companies like eBay, Twitter, Midokura, The Walt Disney Company, Ticketmaster.com, Home Depot, NBCSports, Samsung SDS, Golfchannel.com, Bloomberg, and more.

Registration for this event is sold out, but you can still watch the keynotes via livestream and catch the session recordings on CNCF’s YouTube channel. Sign up for the livestream now.

For a sneak peek at a few notable speakers and their presentations, read on.

UniK: Unikernel Runtime for Kubernetes

UniK is an open-source tool written in Go for compiling applications into unikernels and deploying those unikernels across a variety of cloud providers, embedded devices (IoT), as well as a developer laptop or workstation. To demonstrate the value of cluster management of unikernels, EMC implemented a UniK runtime for Kubernetes – making Kubernetes the first cluster manager to support unikernels. In her featured KubeCon session, Idit Levine – CTO of EMC – will dive into this integration and explain how her team allowed UniK to take advantage of core Kubernetes features like horizontal scaling, automated rollouts and rollbacks, storage orchestration, self-healing, service discovery, load balancing and batch execution.

Kubernetes + Prometheus = Love

K8s needs a system capable of monitoring all individual units across the entire stack while enabling users to drill down from a global view to individual instances. Prometheus was designed with exactly this goal in mind. In his CloudNativeCon keynote – titled “Monitoring Kubernetes Clusters with Prometheus” – Fabian Reinartz, software engineer at CoreOS, will explain common challenges when monitoring large scale infrastructure, how Prometheus provides high-level observability without giving up low-level insight, and why this makes Kubernetes and Prometheus a match made in open source heaven.

Comcast’s Kubernetes Journey

Comcast is working to build a geographically distributed system for streaming linear video to millions of cable customers. Its infrastructure consists of approximately 1,000 physical locations, within 27 regional networks. Currently, its team aims to conserve bandwidth across its cable backbones by encoding IPTV streams to QAM streams at the last mile – eliminating duplicate video streams across the corporation’s backbone while supporting existing QAM infrastructure. In his keynote, aptly named “Kubernetes: As Seen On TV,” Erik St. Martin, a systems architect at Comcast, will walk attendees through how the company leveraged existing Kubernetes components and developed custom ones to build this growing system.

Taking the Challenge out of Service Level Agreement for Networking

With the proliferation of cloud services and the development of fine-grained virtualization techniques, the Network Service Agreement (SLA) is required to manage network resources efficiently for the large-scale, high-density computing units. In their session, “Network Service Agreement (SLA) Systems for Container Networks,” Yan Sun and Xuefeng Han, staff research engineers from Huawei, will discuss the role SLA Monitoring plays in classical SLA management model and a proposed solution called Networking Service Level Agreement system that targets a general and robust design, keeps minor modification to the Cloud Management System and is verified by prototype implementation.

How Bad Is Dirty COW?

By

Konstantin Ryabitsev

-

October 24, 2016

“Dirty COW” is a serious Linux kernel vulnerability that was recently discovered to have been lurking in the code for more than nine years. It is pretty much guaranteed that if you’re using any version of Linux or Android released in the past decade, you’re vulnerable. But what is this vulnerability, exactly, and how does it work? To understand this, it’s helpful to illustrate it using a popular tourist scam.

The con

Have you ever played the game of shells? It’s traditionally played with a pea and three walnut shells — hence the name — and it is found on touristy street corners all over the world. Besides the shells themselves, it also involves the gullible “mark” (that’s you), the con artist (that’s the person moving the shells), and, invariably, one or several scammer’s assistants in the crowd pretending to be fellow tourists. At first, the accomplices “bait” the crowd by winning many bets in a row, so you assume the game is pretty easy to win — after all, you can clearly see the ball move from shell to shell, and it’s always revealed right where you thought it would be.

So, you step forward, win a few rounds, and then decide to go for a big bet, usually goaded by the scammers. At just the right time you’re momentarily distracted by the scammer’s assistants, causing you to look away for a mere fraction of a second — but that’s enough for the scammer to palm the pea or quickly move it to another shell. When you call your bet, the empty shell is triumphantly revealed and you walk away relieved of your cash.

The race

In computing terms, you just experienced a “race condition.” You saw the ball go under a specific shell (checked for required condition), and therefore that’s the one you pointed at (performed the action). However, unknown to you, between the check and the action the situation has changed, causing the initial condition to no longer be true. In real life, you were probably only out of a couple of hundred bucks, but in computing world race conditions can lead to truly bad outcomes.

Race conditions are usually solved by requiring that the check and the action are performed as part of an atomic transaction, locking the state of the system so that the initial condition cannot be modified until the action is completed. Think of it as putting your foot on the shell right after you see the pea go under it — to prevent the scammer from palming or moving it while you are distracted (though I don’t suggest you try this unless you’re ready to get into a fistfight with the scammer and their accomplices).

The COW

Unfortunately, one such race condition was recently discovered in the part of the Linux Kernel that is responsible for memory mapping. Linux uses the “Change on Write” (COW) approach to reduce unnecessary duplication of memory objects. If you are a programmer, imagine you have the following code:

a = ‘COW’

b = a

Even though there are two variables here, they both point at the same memory object — since there is no need to take up twice the amount of RAM for two identical values. Next, the OS will wait until the value of the duplicate object is actually modified:

b += ‘ Dirty’

At this point, Linux will do the following (I’m simplifying for clarity):

allocate memory for the new, modified version of the object
read the original contents of the object being duplicated (‘COW’)
perform any required changes to it (append ‘ Dirty’)
write modified contents into the newly allocated area of memory

Unfortunately, there is a race condition between step 2 and step 4 which tricks the memory mapper to write the modified contents into the original memory range instead of the newly allocated area, such that instead of modifying memory belonging to “b” we end up modifying “a”.

The paydirt

Just like any other POSIX system, Linux implements “Discretionary Access Controls” (DAC), which relies on a framework of users and groups to grant or deny access to various parts of the OS. The grant permission can be read-only, or read-write. For example, as a non-privileged user you should be able to read “/bin/bash” in order to start a shell session when you log in, but not write to it. Only a privileged account (e.g. “root”) should be able to modify this file — otherwise any malicious user could replace the bash binary with a modified version that, for example, logs all passwords or starts up a backdoor.

The race condition described above allows the attacker to bypass this permissions framework by tricking the COW mechanism to modify the original read-only objects instead of their copies. In other words, a carefully crafted attack can indeed replace “/bin/bash” with a malicious version by an unprivileged user. This vulnerability has been assigned both the boring name (“CVE-2016-5195”), and the now-customary branded name of “Dirty COW.”

The really bad news is that this race condition has been present in the kernel for over 9 years, which is a very long time when it comes to computing. It is pretty much guaranteed that if you’re using any version of Linux or Android released in the past decade, you’re vulnerable.

The fix

Triggering this exploit is not as trivial as running a simple “cp” operation and putting any kind of modified binary in place. That said, given enough time and perseverance, we should assume that attackers will come up with cookie-cutter exploits that will allow them to elevate privileges (i.e. “become root”) on any unpatched system where they are able to freely execute arbitrary code. It is imperative that all Linux systems are patched as soon as possible — and a full reboot will be required, unless you have some kind of live patching solution available to you (if you don’t already know whether you can live-patch, then you probably cannot, as it’s not a widely used technology yet).

There is a fix available in the upstream kernel, and, at the time of writing this article, the distributions are starting to release updated packages. You should be closely monitoring your distribution’s release alerts and apply any outstanding kernel errata as soon as it becomes available. The same applies to any Android devices you may have.

If you cannot update and reboot your system right away, there are some mitigation mechanisms available to you while you wait (see this Red Hat Bugzilla entry for more details). It is important to note that the STAP method will only mitigate against known proof of concept exploits and is not generic enough to be considered a good long-term fix. Unfortunately, “Dirty COW” is not the kind of bug that can be prevented (much) by SELinux, AppArmor or any other RBAC solution, nor is it mitigated by PaX/GrSecurity hardening patches.

The takeaway

As I said earlier, in order to exploit the “Dirty COW” bug, the attacker must first be able to execute arbitrary code on the system. This, in itself, is bad enough — even if an attacker is not able to gain immediate root-level privilege, being able to execute arbitrary code gives them a massive foothold on your infrastructure and allows them a pivot point to reach your internal networks.

In fact, you should always assume that there are bad bugs lurking in the kernel that we do not yet know about (but the attackers do). Kees Cook in his blog about security bug lifetimes points out that vulnerabilities are usually fixed long after they are first introduced — many of them lurking in the code for years. Really bad bugs the caliber of the “Dirty COW” are worth hundreds of thousands of dollars on the black market, and you should always assume that an attacker who is able to execute arbitrary code on your systems will eventually be able to escalate their privileges and gain root access. Efforts like the “Kernel Self Protection Project” can help reduce the impact of some of these lurking bugs, but not all of them — for example, race conditions are particularly tricky to guard against and can be devastating in their scope of impact.

Therefore, any mitigation for the “Dirty COW” and other privilege escalation bugs should really be considered a part of a comprehensive defense-in-depth strategy that would work to keep attackers as far away as possible from being able to execute arbitrary code on your systems. Before they even get close to the kernel stack, the attackers should have to first defeat your network firewalls, your intrusion prevention systems, your web filters, and the RBAC protections around your daemons.

Taken altogether, these technologies will provide your systems with a great deal of herd immunity to ensure that no single exploit like the “Dirty COW” can bring your whole infrastructure to its tipping point.

Learn more about how to secure Linux systems through The Linux Foundation’s online, self-paced course Linux Security Fundamentals.