Fix Bugs, Go Fast, and Update: 3 Approaches to Container Security


Containers are becoming the central piece of the future of IT. Linux has had containers for ages, but they are still maturing as a technology to be used in production or mission-critical enterprise scenarios. With that, security is becoming a central theme around containers. There are many proposed solutions to the problem, including identifying exactly what technology is in place, fixing known bugs, restricting change, and generally implementing sound security policies. This article looks at these issues and how organizations can adapt their approach to security to keep pace with the rapid evolution of containers.

During his talk at the Cloud Foundry Summit, Justin Smith, Director at Pivotal Software, Inc., who is also involved with Cloud Foundry security mentioned the 2015 Worldwide Threat Assessment which now lists cyberattacks as the number one threat. Ahead of terrorism! “Devices, designed and fielded with minimal security requirements and testing, and an ever-increasing complexity of networks could lead to widespread vulnerabilities in civilian infrastructures and US Government systems,” the report said.

The tech community is less worried about an Armageddon-style cyber attack. “We’re more worried about a bunch of moderate attacks against a bunch of companies. It just devastates the economy. Death by a thousand paper cuts,” said Smith.

Security is fixing bugs

Companies with stakes in the container landscape adopt different approaches towards security.  Lars Herrmann, general manager of Red Hat told me in an interview, “Containers define a different way for the organization to collaborate inside the organization. That’s really the disruptive potential but, in order to do this, we need a certain kind of technology to enable that transformation.”

He said that security is an important aspect of it because we cannot go into production with lots of different applications without having a solid understanding as to how do we manage security in this environment.

As Linus Torvalds once said: security means bugs. And, no software is immune to bugs. Herrmann said that even the developers’ own code might have security issues. So companies need to think about all aspects of the process.

To protect organizations, Red Hat has partnered with Black Duck so that customers can identify and detect which open source technology is in which version, and then correlate that with their back-end database based on what they know about the technology in that version. “We’ll see more solutions that are driving more insights and are making a more fine-grained risk assessment,” Herrmann said.

Scanning is only one part of the picture; the second part is dealing with it. Herrmann said developers could implement policies such that if there is a container in an environment, on a registry that doesn’t meet security criteria, they can automatically trigger workflow and rebuild that container with the latest runtime components to address that security issue.

That’s just one piece of the security jigsaw puzzle.

Security means going faster

Pivotal’s Smith is critical of the way organizations approach security. He said “…to get safer, you go faster. That’s the exact opposite of how organizations think today. What I want in my time at Pivotal and my time as part of the Cloud Foundry community is to build the system that gives no quarter for malware in the data center.”

According to Smith, organizations need to make it harder for attackers to compromise their systems. It should be like playing a video game for the malware author, where you have get to level 100, but you can never get past level five because there’s not enough time.

“What if every server inside my data center had a maximum lifetime of two hours?” Smith said. That situation will frustrate attackers, because it limits the time needed to exploit known vulnerabilities.

When I asked Herrmann about this approach, he partially agreed that going faster may sound safer, but it heavily depends on the nature of attack. There is no single factor. “How long a system is potentially exposed is one factor. How badly it is exposed is another factor. How many layers of protection you have is certainly another relevant factor, just to name a few. “

Herrmann added that he didn’t actually believe that just because a given container only lives for a couple seconds, nobody would be able to compromise it. “I wouldn’t assume that because the attackers have the same technologies available as you have,” he said.

Another problem with the “going faster makes you safer” approach, Herrman said, is that “just because I do everything in a continuous integration fashion, I’m always pulling the latest from everywhere… that also means constantly pulling a lot of different risks from lots of different places. Yeah, sometimes that might work well, but what if it doesn’t? What do you do then? Well, just rebuilding the whole thing with broken components is probably not going to fix your problem. Eventually, somebody has to deliver a fix.”

Build systems that can be updated

Leading kernel developer Greg Kroah-Hartman is of the opinion that companies should build systems that are able to be updated. A major security hole is that of unpatched systems running in production. Kroah-Hartman added that, although the Linux community works quickly to fix the bugs so that vendors can push them to their users, vendors don’t always do a good job.

“We have a very bad history of keeping bugs alive for a long time. Somebody did a check of it; most known bugs live for 5 years in systems. These are things that people know and know how to exploit. They’re not closed. That’s a problem in our infrastructure,” said Kroah-hartman.

Another issue involves the mindset that once a system is set up, you shouldn’t touch it. Herrmann defended that approach and explained that traditional security processes have very much relied on the principle of restricting change. You should work under the assumption that, if you can minimize the risk in every single change, you can minimize the risk in your resulting system. That leads to the idea of “don’t touch a running system,” and all that. There are whole philosophies built around it: Don’t a let a bad change happen. Instead, have guardrails on every change so you can separate the bad change from the good change.

There is no silver bullet

The bottom line is that there is no silver bullet. Container technologies are evolving fast. As they become mature and continue to evolve, the nature of threats will change over time, involving the security of computers, the security of storage, and the security of networking, for example.

Organizations need to be proactive about security instead of hoping for the best. Organizations need to have a holistic approach towards containers, and they need to borrow from different philosophies: Restrict changes, go fast, and have systems that update quickly. It’s less about software and more about culture.