Operating system security stats hard to compare, but more Linux vulnerabilities being reported

Linux security experts take issue with recent reports from vnunet.com and from WinInformant.com that suggest Windows is more secure than Linux, based on statistics from SecurityFocus. But one Linux security guru says he’s seeing more Linux security vulnerabilities reported in the last six months, mostly due to greater awareness on the part of Linux vendors.

Both Vnunet and WinInformant later backed away, at least partially, from their analysis of last year’s SecurityFocus stats, with the raw numbers saying Windows 2000 reported only 24 security vulnerabilities while Debian 2.2, Red Hat 7.0 and Mandrake 7.2 each had 26 or more security vulnerabilities.

Like others,
Dave Wreski argues that the SecurityFocus stats tell an incomplete story. Wreski, corporate manager of Guardian Digital, which supports the EnGarde Linux project, says the problem with the numbers is that many Linux distributions contain the same Open Source programs, so a vulnerability in xchat, for example, would show up as a vulnerability in several Linux distributions, unlike a single report for Windows.

Wreski suggests a healthy skepticism about statistics. “They can be interpreted in any number of ways,” he says. “As we’ve seen, the Windows pundits
choose to interpret them as an aggregate, while Linux advocates correctly point out that nearly all distributions contain the same programs, resulting in duplicated and skewed results.

“The issue goes beyond the numbers,” he adds. “The numbers neglect to point out the
sometimes thousands of packages that comprise a Linux system, nearly none of which are included in Windows … It’s somewhat akin to if we were talking about cars. If all Linux car manufacturers purchased their tires from Goodyear, while Microsoft
purchased theirs from Bridgestone, a defect in Goodyear tires wouldn’t
affect Microsoft, but instead could affect all Linux car companies equally.”

As for security vulnerabilties in the Linux kernel itself, there have only been about a half dozen in the last two years, Wreski says.

Jay Beale, the lead developer of Bastille Linux and a security consultant, comments on the stats another way. “The most obvious problem … is that they’re using the statistics for a purpose they are not intended for. The statistics, from SecurityFocus.com, were accompanied by the following statement, in bold-face type: ‘The numbers presented below should not be considered a metric by which an accurate comparison of the vulnerability of one operating system versus another can be made.’ “

Increase in Linux vulnerabilities?

Oops. Still, Wreski says he sees at least a 10% increase in Linux security vulnerabilities in the last six months or so. He attributes this to a greater awareness of security issues and their implications over the past year and a half, partly due to budget concerns in businesses, partly due to technology security debates following the Sept. 11 terrorist attacks on the United States.

“Small companies and enterprises demand solutions that will protect their corporate assets,” he says, addressing economic issues. “There is no time to have to deal with viruses, unauthorized access, or even policy considerations. There often is no budget for on-staff security professionals or an administrator that needs to understand the weakest link that a cyber-vandal would find to compromise their system.”

Wreski suggests that Microsoft and Linux vendors have “different attitudes about security. “Microsoft assumes that because there is a vulnerability in a product
that no one knows about, there is little danger because no one would
know how to exploit it,” he says. “There is no market incentive for Microsoft to release all information about potential security vulnerabilities and no repercussions when they
don’t. Scott Culp’s essay from some months ago indicated to us that Microsoft
feels security issues will go away if they are ignored or not publicly announced.”

Although Wreski says Microsoft’s security reporting has improved in the last couple of years, he calls it a “facade,” based partly on Culp’s objection to Web sites reporting security flaws. “If they had it their way, they would squelch any ability by external individuals or companies from announcing that they’ve found a vulnerability, because they make no money from security,” he says. “If their marketing group can convince the public-at-large that they’ve taken new security measures with their latest versions, they
have no incentive to fix or audit old versions or give exact details on what they’ve done to improve security in their latest versions.”

Many eyes see many holes

Beale says he hasn’t tracked the data close enough to confirm Wreski’s observation of more Linux security vulnerabilities lately, but he notes that the Linux community’s “many eyes” practice of reporting security problems may affect the numbers.

“With some vendors, it is definitely the case that they don’t pay attention to a vulnerability advisement until someone makes an exploit, theoretically raising the danger level,” Beale says. ” With particularly bad vendors, they don’t pay attention until the exploit is released publicly and they’re pressured by tons of customers and reporters to actually fix the hole. Linux vendors don’t ever fall into this category because so much of the development is open to public review and discourse.”

Problems with the stats

As for security statistics, one problem is that reports often happen in “clumps,” Beale adds. A program gets audited, and it creates a kind of snowball effect, with more people paying attention to security problems in that application. “The one point I’ll make here is that vulnerability analysis doesn’t happen equally across all operating systems or even all versions of an operating system at the same time,” Beale says. “Vulnerability discovery is sometimes accidental, but it will often follow particular auditing group’s schedules. So, we’ll see a huge number of vulnerability releases on Internet Explorer one month, because perhaps ISS directed their Xforce guys to examine that one month.”

Still, Windows supporters might point out that if we look at just one Linux distribution, Mandrake, we find that it had 33 vulnerabilities reported in the first half of 2001, compared to 24 for Windows. Even if Mandrake and Red Hat have the same vulnerabilities, doesn’t that make Windows more secure than Linux in a one-on-one comparison?

Beale again points to SecurityFocus’ warning not to read too much into the stats. He says an exhaustive run through the database, looking for all of the vulnerabilities found in each distribution of Linux versus each respective version of Windows, might yield more accurate results, but that hasn’t happened.

“Let me clarify the real reason that the (SecurityFocus) page doesn’t support the conclusion that articles have made is this: Those statistics don’t answer the real question,” he adds. ” The real question is: ‘For comparable operating system installs, does a particular Windows version have more or fewer vulnerabilities than its matching Linux version?’ Remember, Linux distributions often ship two to three versions per year, sometimes substituting one program for another, always updating versions. In both cases, there are often vulnerabilities shipped each numerical version of a Linux distribution that weren’t in the previous one. If we count vulnerabilities per year, we’ve got some double-counting going on! “

The vnunet and WinInformant reports also overlook the fact that Linux distributions often give users multiple options for everything from browsers to IRC clients to mail clients, meaning multiple chances for security problems, and that’s only talking about desktop applications, not server applications. “Because so many programs are available on Linux for free, Linux distributions tend to package huge numbers of programs,” Beale says. “These will include many server programs that are ordinarily separate items in the Microsoft offerings … Many security people who have seen the vnunet or WinInformant stories have made the point that the numbers are skewed greatly by this — to do an accurate comparison based on numbers of programs, we’d have to bundle all of the primary server programs used on Microsoft platforms, like Oracle, and do a recount.”

Beale points out another issue with the SecurityFocus statistics that’s more difficult to judge: how serious is the vulnerability? “The numbers don’t consider severity, which is what I’m most interested in when I hear of another security vulnerability,” he says. “While you may be tempted to hold this against SecurityFocus, remember they weren’t making a comparison of operating system security. They weren’t trying to answer that question, in part because the comparison is so complicated.”