Talking security with Red Hat’s Mark Cox

IT professionals spend a lot of time thinking about security, and ways to make sure their systems are patched as quickly as possible. However, what goes on before they hear about a vulnerability is mostly a mystery. To get a clearer picture, we talked to Mark Cox, director of Red Hat’s security response team, about trends in Linux security, who discovers vulnerabilities, how they’re rated, and what’s being done to minimize security problems.

Cox says that his interest in security started about 12 years ago when he discovered security issues in the National Center for Supercomputing Applications (NCSA) HTTPd Web server. “Soon after that I founded the European operation for C2Net, where we developed the Stronghold Web server specifically outside of the USA so we could distribute full-strength SSL encryption worldwide. Red Hat purchased C2Net and here I am!” In addition to his security work with Red Hat, Cox says he also participates in the security teams for the Apache Software Foundation and the OpenSSL group in his spare time.

What does the job of a security response team member entail? Cox says that the team is accountable for ensuring that vulnerabilities are addressed. “This involves a large number of things; from working with researchers and monitoring to find out about issues that affect us, through triage and investigation, to oversight during the release process.”

In addition, Cox says that team members are tasked with writing advisories and answering customer questions about vulnerabilities. “In the last year, we managed a 98% response within one business day.”

One might wonder exactly how many people work on security issues for a large vendor like Red Hat. Cox didn’t say how many people are directly employed to work on security, but he did say that they often pull in different engineers depending on the vulnerability.

“We’ve a number of full-time engineers distributed globally to give us good coverage of different timezones, and we’re always looking for more good people. When responding to an issue, we’ll bring in specific domain expertise from the development and quality engineering teams; so the number of Red Hat employees working on security vulnerabilities varies week to week.”

Security teams often have access to vulnerability information before the general public. In fact, Cox says it’s “about half and half,” for Red Hat’s security team — meaning that Red Hat often has time to address a vulnerability before it’s common knowledge. “Where we know about an issue in advance it gives us the ability to work with our peers to get the right patch, look for similar issues in similar code bases, do more testing, and coordinate a release with other affected distributors.”

However, Cox says that even when a vulnerability isn’t public knowledge, they try to address the issue as quickly as possible. “In fact, over the last two years, of vulnerabilities we’ve known about in advance, the median notice before public was just seven days. For the subset of issues with critical severity the median notice was two days.”

Daddy, where do vulnerabilities come from?

How, exactly, are most security vulnerabilities discovered? On his blog, Cox broke down the sources of vulnerability information for Red Hat from March 2005 through March 2007. The note also looks at whether Red Hat received information prior to the vulnerability being made public.

According to Cox’s breakdown, the leading source of information is from other FLOSS vendors, followed by reports from upstream vendors such as Firefox. In many cases, Red Hat is informed by other FLOSS vendors or upstream projects before the general public.

But how do the reportees stumble on the vulnerabilities in the first place? Cox says that the vulnerabilities are found in a number of ways, sometimes by proactive audits, and sometimes as a result of bug reports for other issues. “Upstream projects and distributors sometimes perform audits or run code analysis tools, some of the security innovations have discovered issues proactively, customers sometimes report bugs that end up having a security consequence.”

Stopping problems before they happen

Just fixing security issues as they arise, essentially putting out fires as they happen, is a losing proposition, Cox says. This is why Red Hat and other Linux vendors have been working on technologies that mitigate security problems. For example, SELinux and Exec-Shield can protect systems by blocking certain types of exploits.

Cox released a paper in April that addresses mitigation technologies. In the paper, Cox describes various security features shipped in Red Hat Enterprise Linux (RHEL) 3 and 4, and what sorts of exploits they address.

For instance, Cox discusses two “double-free” vulnerabilities. A double-free is a condition where the free() function is called twice on the same memory address, which might lead to a buffer overflow. According to the paper, the flaw was responsible for “high-profile exploits of services such as wu-ftpd and CVS pserver” in 2003, prior to “hardening” of glibc against double-free vulnerabilities.

In 2004, a double-free flaw was found and announced in the MIT Kerberos 5 Key Distribution Center. However, because glibc in RHEL 4 was hardened against the flaw, “we were able to downgrade the severity of this flaw from critical, as the glibc hardening totally prevented the exploitation of this double-free flaw.”

Not all vulnerabilities are created equal

When Red Hat issues a vulnerability notice, it also issues a threat rating to go along with it. The issues are listed along a four-point scale ranging from serious flaws that could allow a remote attacker to compromise a system without any user interaction (“Critical”) to vulnerabilities that have some kind of security impact, but where it’s thought to be very difficult to execute an exploit against a system and/or when an exploit would have negligible consequences (“Low”).

Red Hat does not change the security classification depending on whether an exploit is actually available in the wild or not. Cox says that is, in part, in order to remain in step with the way that other organizations report vulnerabilities. “We define the risk to a user as being a function of both the vulnerability and the threat. The ratings we give on our advisories are based solely on the vulnerability and are designed to match other vendors like Microsoft and Apache. Since the threat changes over time and relies heavily on the end users’ configuration and use, we don’t report that part.”

Is Linux seeing more exploits?

Conventional wisdom a few years ago was that as Linux became more popular, there would be more exploits crafted specifically for Linux. Cox says that Red Hat has just finished looking at exploits that might have affected RHEL 4 in the first two years of its release, and that doesn’t seem to be the case. “There were public exploits for 37 vulnerabilities. Many of them are caught by the various security innovations, some need unlikely configurations, and many don’t work as published. What you are left with are some kernel exploits that would allow a local user to gain root privileges on an unpatched machine, and some exploits that would work against unpatched Web browsers.

“Aside from Web browser flaws there are not actually that many remote vulnerabilities these days for attackers to write exploits for. A default installation of Enterprise Linux 4 AS, for example, was vulnerable to only three critical security issues in its first two years.”