In part four of this four-part series, Xen Project Advisory Board Chairman Lars Kurth takes a closer look at the Xen Project’s Security Policy and its evolution from its inception in 2011 to today and what it means for IT teams. Read Part 3: Are Today’s Open Source Security Practices Robust Enough in the Cloud Era?

Before we dive into Xen Project’s security practices, it is worth noting that no users of large or small cloud providers have been put at risk from security issues related to the Xen Project since it established a security process in 2011.

A Brief History of the Xen Project’s Security Policy

The Xen Project’s policy looked pretty much like the standard open source security policy a few years ago and followed the Distro Model of Responsible Disclosure, which was introduced in part three of this series. However, the Xen Project user community forced the project to change in 2012 and into 2013. Guided by community input, the Xen Project established one of the most codified, fair, transparent and battle-hardened security processes in the industry.

Before 2011, Xen Project didn’t have an established security process, but began to adapt Debian’s approach to security issues with the exception that the Xen Project included “large service providers” on the pre-disclosure list. These were added, so service providers could plan and prepare to deploy security fixes once issues were published.

In June 2012, FreeBSD, NetBSD, Solaris, Microsoft Windows and Xen were exposed to the Intel SYSRET privilege escalation (or CVE-2012-0217). This vulnerability was discovered in the context of Xen and FreeBSD on April 9, 2012. The fix was originally scheduled to go out on May 1, but in the end was not publicly announced until June 12. The primary reason for the delay was there was severe pressure on the Xen Project to keep the bug and the fix under wraps for months.

The Xen Project resisted this pressure, but eventually had to cave in when the vendors in question successfully convinced the discoverer of CVE-2012-0217 to request a delay of its public release and transfer the handling of CVE to US-CERT. At this point, it is worthwhile reminding ourselves that vulnerability processes rely on trust between vulnerability discoverers and the open source project’s security team. If that trust is broken, fewer vulnerabilities may be reported in future.

This incident highlights one of the biggest fundamental tensions in operating vulnerability processes: balancing the needs of different stakeholders. It is also presumably the primary reason why Linus Torvalds dislikes Responsible Disclosure, which is shared by many others and has been the topic of controversy for many years.

Following CVE-2012-0217, it became clear that the Xen Project needed to adapt a new security process for virtual environments given the diverse nature of the players involved. It established a maximum SLA of 3 weeks from discovery to publication of a security issue to ensure that no undue pressure on the project could be exerted. Xen Project also embarked on a prolonged and difficult community consultation to create a more fair and balanced security process. During the consultation, it became apparent that a large section of Xen users — namely smaller cloud and service providers — felt that “a major security issue could potentially put them out of business, while larger vendors may merely suffer a damaged reputation.”

In the end, the Xen Project departed from the common approach of “minimizing the number of privileged stakeholders” at any cost. Instead, it allowed service providers of any size to join its pre-disclosure list trading off the risk of information leaks in the name of fairness.

The year of 2014 ushered in a number of major security blunders (Edward Snowden, Heartbleed, Shellshock) by big corporations and led to heightened concerns about cloud security. Although no cloud providers were put at risk from security issues related to Xen Project, it made the project aware that it needed a better way to scale the Xen Project’s security process.  

The Xen Project security process was designed before it was clear what operating an efficient security policy under heightened scrutiny would entail. For example, the Project underestimated the number of requests to join the pre-disclosure list and had difficulties verifying whether an application was valid. In addition, different vendors interpreted the text of the policy in different ways leading them to make varying assumptions on what could and could not be done with privileged information. Consequently, the project again consulted with its community, which led to further improvements to the Xen Project’s security processes.

Key Elements of the Xen Project’s Security Policy

In order to address the issues of the cloud computing environment and ensure that the Xen Project security process can scale, there are several ways that Xen Project’s security process differs from the majority of open source projects. These include:

• For each vulnerability, Xen Project clearly states whether the security fix for the vulnerability can be deployed on public facing services before the security embargo ends.

• There is a public application process for pre-disclosure membership based on simple criteria that are not biased toward distros or large vendors. In addition, the Xen Project publishes the companies that receive privileged security information.

• The Xen Project frequently handles security issues for projects that it depends on and that affect its users by working closely with security teams of interfacing open source projects.

• It pre-discloses information related to a vulnerability in cases where a fix is not available early on. This enables service providers to assess risk and plan changes to their systems.

• Note that these cases are extremely rare – there has only been one in the history of the project.

• It allows members of the pre-disclosure list to share issues and experiences on a dedicated secure mailing list.

• It allows for public post-mortems of security issues on request of members of the pre-disclosure list to improve the process and operations in the future.

• The Xen Project treats each reported vulnerability equally, regardless of severity, and keeps a central repository of historical information.

What IT Teams Can Learn From the Xen Project Journey

This security process has served the Xen Project well in many ways and core elements of it have been adopted by OpenStack and others (see the “Cloud Model of Responsible Disclosure” in part three of this series). It has led to a deeper understanding of the issues that service providers face, improved trust between Xen Project users and the community at large, and has ultimately led to better security for the Xen Project.

One critical component of building trust is transparency. The Xen Project believes that its approach to publishing vulnerability-related information and its willingness to modify security practices when issues occur, speak for themselves. However, the high level of transparency around this security process has also had costs. Understanding these trade-offs are important for IT teams when assessing a technology for security. The disadvantages of Xen Project’s approach include:

• CVE Databases: The approach to treat each reported vulnerability equally and to assign CVEs for each vulnerability does have some disadvantages. CVE numbers are stored in searchable databases, such as NIST and CVEDETAILS, and can be a useful resource to assess a technology for security. However, vulnerability databases have to be used with utmost care to avoid drawing wrong conclusions. Primarily, because they do not provide information that makes comparing technologies possible. For example, not all projects and vendors assign CVE numbers to all discovered vulnerabilities. This can lead to the wrong impression that one product is more secure than another. Similarly some projects/products cannot be easily identified. While Xen is easily identifiable, KVM and LXC are not due to tight integration with Linux. Issues affecting them are often classified as general Linux issues.

• Press Coverage: High levels of transparency make it easier to attack a project’s track record compared to projects that are less transparent. This has become apparent during the last few Xen Project point releases, where recent press coverage has been focused almost exclusively on security issues. In addition, almost every vulnerability we published in 2015 has become a security story, regardless of its severity and impact. Similar stories are simply not published for KVM, LXC and Linux, because it is much harder for journalists to find relevant information. However, lack of press coverage, does not mean that users of KVM, LXC and Linux are safer, it simply means that the Xen Project is more proactive, transparent and swift in resolving security issues.

Conclusion

To make it easier for IT Teams to evaluate the security of open source technologies and vulnerability management processes, the open source industry needs to establish better best practices. What the Xen Project is doing can be a great framework for other projects going forward, especially in the new cloud environment where there are more components and complexities than when we first began with Linux.

References

The following section contains some pointers to articles, which show some of the discussions leading to the latest version of the Xen Security Process in detail and may be of use to some readers.

Discussion leading to v2.0 of the Project Xen Security Process

• June, 2012: Security vulnerabilities – the coordinated disclosure sausage mill

• June, 2012: Xen.org Security Policy Update: Get Involved

• August, 2012: Disclosure process poll results

• December, 2012: Security disclosure process discussion update

• June, 2013: Security vulnerability disclosure process accepted

Discussion leading to v3.0 of the Xen Security Process 

• October, 2014: Xen Project Security Policy Improvements: Get Involved

• March, 2015: Updates to Xen Project Security Process 

Discussion leading to v3.1 of the Xen Security Process 

• May, 2015: XSA-133 Retrospective

• May, 2015: Earlier embargoed pre-disclosure without patches

Read more:

Part 3: Are Today’s Open Source Security Practices Robust Enough in the Cloud Era?

Part 2: Containers vs. Hypervisors – Protecting Your Attack Surface 

Part 1: A Cloud Security Introduction