Author: Mayank Sharma
To start with, all major distributions have security teams that collaborate with the main release team to ensure no vulnerable packages make their way into the final release. For example, Chris Gianelloni, release engineering strategic lead of Gentoo says that the release engineering team works with Gentoo’s security team and individual architecture teams to make sure Gentoo doesn’t have security vulnerabilities when Gentoo is released.
Tools of the trade
The various distributions use different means of capturing vulnerabilities in packages. Debian’s Moritz Muehlenhoff points to Debian’s Security Tracker which keeps track of bugs reported by their own team, as well as from other sources like the National Vulnerability Database (NVD).
“During preparation for a new release, the security response team audits the packages in the release to ensure that we ship with as few public vulnerabilities as possible,” says Mark Cox of the Red Hat security response team.
Red Hat makes use of several testing tools and does its own regression testing to try to catch issues. “We also want to try to catch new security vulnerabilities and this is where our security innovation comes in. All packages in our products are compiled with various security technologies including Exec Shield, FORTIFY_SOURCE and stack protector. Some of these are able to catch common security programming mistakes even at compile time,” explains Cox.
So what happens when a new vulnerability is discovered? Debian’s security FAQ spells out the Debian process: “Once the security team receives a notification of an incident, one or more members review it and consider its impact on the stable release of Debian (that is, if it’s vulnerable or not). If our system is vulnerable, we work on a fix for the problem. The package maintainer is contacted as well, if they didn’t contact the security team already. Finally, the fix is tested and new packages are prepared, which are then compiled on all stable architectures and uploaded afterwards. After all of that is done, an advisory is published.”
Roman Drahtmueller, Linux security architect at Novell, says identifying and fixing vulnerabilities is an on-going process that constantly improves the security properties of their products. “Novell products with known severe vulnerabilities will not be shipped; lesser severe vulnerabilities may, however, appear during the time that elapses between the product’s master date and its installation at the customer’s site. When just released, a product has only a few updates provided for installation.”
Drahtmueller adds that significant development effort is being put into improvements to the operating system so that the susceptibility to security weaknesses constantly decreases. “For everything else, four and six eyes check-in procedures for changes to the code base in the configuration management system guarantee that no unintended modifications can sneak into the product. All of these processes, including the patch delivery and the check-in policy, have been investigated as part of the successful Common Criteria certifications that SUSE has undergone during the past three years.”
But what about CentOS, which strives to be 100% binary compatible with Red Hat Enterprise Linux? “Our users do not want us to change packages in our base or updates repository unless they are changed upstream,” says Johnny Hughes, lead developer of CentOS.
They do, however, submit patch and update requests to the upstream provider if there are security issues or bugs, via their public bug tracking system. The same thing, submitting patch recommendations to the upstream packager, is done for rebuilt packages in the Extras and CentOSPlus repositories.
“For items that we wrote the SRPMS for, we have a developer who is in charge of that package and that developer is in contact (usually via a mailing list) with the provider of source files. Even in these cases there is usually a program provider, they just normally provide the packages in a form other than via SRPMS/RPMS. We would normally provide feedback and recommended patches to the maintainer of the files in question and roll in those changes when the maintainer has implemented them”, says Hughes.
Keeping the distribution secure after release
So it takes a lot of effort to pump out releases free of known vulnerabilities. But is that it? Shouldn’t the projects ensure their distribution stays secure after release? Apparently, they do.
Debian’s tracker can be used to keep track of issues that creep in after a release. Muehlenhoff also suggests that paranoid users install the debsecan tool, which reports live vulnerabilities for a certain system.
Hughes also says that the CentOS team monitors several security lists and quickly publish patches that are released by the official package maintainers in all cases.
Gentoo publishes security advisories along with instructions to eradicate the vulnerability in the affected package. Gianelloni points out that due to the nature of Gentoo’s portage tree being a live system, updates are done in the tree itself. Syncing the portage tree and upgrading every package should also keep one up-to-date security-wise. As per Gentoo’s security documentation, integration of security-only updates in portage tools is underway.
Moving on to the enterprise distributions, Cox says that Red Hat has been providing secure, signed, and automated update mechanisms for their products since 2000, ensuring that customers can easily obtain timely updates to protect themselves against vulnerabilities and threats. He adds, “our security innovations are also designed to help reduce the risk of vulnerabilities discovered after a product has shipped. Red Hat Enterprise Linux 4, for example, shipped with SELinux enabled by default with policies protecting the more risky services. We definitely try to make security a default where possible rather than requiring a system to be hardened to be protected.”
The Red Hat Enterprise Linux security manual provides details of best security practices. Cox also points out to the security reports they provide from time to time evaluating vulnerabilities and threats for users.
According to Drahtmueller Novell’s security team handles “the complete spectrum of possible security incidents in Novell products, starting with the quick response to a reporter, the cooperation with other vendors and security organizations, and the technical and organizational supervision of the internal work on the software until a user can benefit from it in the end.”
But, apart from installing updates, Drahtmueller points security sensitive users to Novell’s security page that contains valuable hints and further links “for additional measures and indispensable basics”.
Drahtmueller also touts the benefits of Novell’s AppArmor security product. “For those cases where a system becomes exposed to a yet unfixed security vulnerability, the containment technology AppArmor effectively protects the customer’s assets from intruders. AppArmor is activated by default in newer products, and it comes with a rich set of predefined profiles for all of the commonly used services of a Novell system. Services or applications that are not bundled with the original product and that could benefit from additional protection by AppArmor can easily be profiled and mitigated using the tools that come with AppArmor.”
Time consuming process
From what we’ve seen so far, it’s not child’s play to release and maintain a secure distribution. But how much time and manpower does it really take? Even after digging up the procedure involved in making and maintaining a secure distribution, I was a bit surprised by the number of people involved in the process and how time consuming the job was. All 13 of CentOS developers are on the security team. “I would estimate that 25-30% of the time we spend on CentOS is directly related to security and security updates of our products,” says Hughes.
Muehlenhoff says that six people make up Debian’s security team. Due to the nature of the work and the way it’s integrated into the development cycle, he couldn’t give an exact figure of how much time the team spent on security related issues. “I don’t really know, as there are more people involved in checking, for example the 1,000-plus Debian maintainers. It can be five minutes if information is coming from a trusted source, up to several hours for a review of a large package like XFree86.”
Gentoo has 11 people on the security team, though a few are inactive. Again, Gianelloni has no breakdown of time spent on security alone, but says that it takes them about two to three months to put together a release.
“We generally spend the first month to two months doing preliminary QA on the packages for the release. We try to locate and resolve any bugs in the packages, as well as the tools used to build the release. We then create a snapshot of the package repository and begin our release build. During the course of building, we adjust the packages to resolve build-time and security issues, as they arise.”
According to Drahtmueller Novell’s security response process involves many developers, from working on the actual patch, testing and quality assurance of the update package, its final publication and the announcement of its availability. “For an average package, up to 21 people are involved. The security team — made up of four people — initiates, supervises and, if necessary, corrects the process of working on the updates, and manages the public announcement of the results in a very timely manner.”
As for the amount of time it takes, he reasons that “work on security vulnerabilities consumes more resources than a comparable flaw that is being fixed in the course of the regular integration process for a new release of a product, because the incidents are automatically of the highest priority, and they do not keep to any project plan. Also, security relevant malfunctions in software can cause damages that cannot be calculated by the vendor on a generic basis because unknown assets are at risk. The only valid response to these circumstances is to deliver the best work possible in the least amount of time.”
Cox says that Red Hat has a number of security teams, each focused on a different aspect of security. “Our security response team deals with vulnerabilities after release and has employees situated in different countries to provide global coverage. For critical vulnerabilities, the security team can pull in staff from various other parts of the organization to rapidly respond to issues.
“We also have teams working on SELinux, engineering our other security technologies, security certifications, and so on. Most of the things we do are part of our process, but the security response team’s audit is something that takes a large block of time towards the end of the release cycle.”
Conclusion
Putting together a secure distribution might not seem like a lot of work, but popular vendors spend a lot of thought, effort, and time on this. Several developers collaborate to ensure their release bundles applications sans any vulnerabilities and bugs. Special tools, security bulletins and meticulous package scanning goes into ensuring a distribution stays secure even after being put into use.
Category:
- Security
