Patch testing

Critical to the concept of patch management is testing. When applied indiscriminately, patches can override working drivers, break existing software, and change the links to key files. They may introduce new features that override services on which you rely. Best practices suggest that you test every patch before installing it on production computers. Ideally, you’ll have one or more extra computers available for this purpose.

This article is excerpted from the newly published book Linux Patch Management: Keeping Linux Systems Up To Date, by Michael Jang. © 2006, Pearson Education, Inc. Used with the permission of Pearson Education, Inc. All rights reserved.

I define production and test computers as follows:

• A production computer is a desktop or server system, configured with the services and applications that are actually used to run your business.
• A test computer is a desktop or server system that should be used exclusively for testing new software. Ideally, test computers are also configured with the same services and applications that you have on production server and desktop computers. They should be segregated from production computers with a firewall.

Test computers may be a luxury unavailable to administrators of smaller networks. If you’re in that position, you should at least test each patch on your own administrative system. Every problem you see on your own system is one less problem that you hear about from each of your users.

There is no single method for patch testing. Testing methods are as diverse as available applications and services. I’ve divided the testing process into three categories: installation, applications, and services. In any case, always follow this maxim: Do not install a patch on a production computer before testing.

Testing patch installations

The first step in testing a patch is installation. When you use the techniques described in this book, there should be no unmet dependencies. However, dependencies often require you to install more patches. Therefore, when you test a patch installation, you need to know what you’re installing to know the scope of the patch.

So, in summary, when checking the installation of a patch on a test computer, check for the following:

• When checking a patch, make sure all package dependencies are satisfied. List the packages required to satisfy those dependencies.
• If there are dependent packages, include those in your testing.

These are general guidelines that vary with what you’re installing. For example, if you’re installing a new Linux kernel, test everything on a production-level system. If you’re upgrading a stand-alone application, make sure that the upgrade won’t break features on which your users depend. If you’re upgrading a service, make sure that your configuration file still works as intended. If you’re upgrading dependent libraries, make sure they don’t affect the application or services which depend on them.

If you have problems with a patch, check the troubleshooting steps described at the end of this section.

Testing a new kernel

The kernel is the most important patch to test. As the core of the Linux operating system, an upgraded kernel can affect everything. There are several principles that you should follow when testing a new Linux kernel:

• Make sure you have a reason to change kernels. Check the release notes for that kernel. You may not need the patch. For example, if a new kernel incorporates a driver for network cards that you don’t use, there is no reason for you to go through all this trouble.
• Never upgrade to a new kernel — in a way, that overwrites your existing kernel. Do not delete the existing kernel until you’re finished testing the new kernel.

Most distributions support side-by-side installations of new and existing kernels. In fact, when you install (and don’t upgrade) an existing kernel using a customized binary package, Red Hat/Fedora, SUSE, and Debian all “dual-boot” the new and existing kernels.

• Test how the new kernel manages your boot process. Watch as it detects your hardware and starts your services. If there are problems or even unusual delays, the new kernel may have a problem with the associated hardware or service.
• Check the log files associated with the boot process. Pay particular attention to services that won’t start or hardware that isn’t detected.
• Test the new kernel on your system, based on the release notes. Make sure the new kernel works as intended on your computers.
• Test your services and applications under the new kernel. Pay attention to any changes in behavior. Document these changes.
• When you’re ready to upgrade the kernel on production computers, tell your users what they can expect.

If you identify a problem with the kernel and absolutely need the upgrade, you may need to recompile the kernel yourself. For more information, see the Kernel Rebuild Guide, available from www.tldp.org. After you rebuild, you can install and start the testing process again.

Testing application patches

Many applications are patched frequently. Application patches may incorporate security improvements, additional features, bug fixes, or more. When you see an application patch, don’t just blindly upgrade. Consider the following:

• Read the release notes. Evaluate the reason for the patch. Does it add any features that you need? Does it address any bugs which annoy you about the application? Does it improve your security in a way that applies to your computers or your network?
• If there are configuration files associated with an application, back them up. For example, if a user customizes the look and feel of the GIMP, a patch might overwrite some of the settings in each user’s .gimp-* configuration directory.
• You may want to use configuration files from users who use the application frequently. Tell them that you’re considering an upgrade. They can help identify key features that you will want to preserve.
• If you decide that you need the application patch, test it carefully. Download and install it on your test system. Make a list of downloaded packages, especially if there are dependencies.
• Test the patch on your system. First, based on the release notes, make sure the patch works as intended.
• Identify any changes made to applicable configuration files, and assess their impact on your users. Make notes.
• If there are problems with the patch, re-evaluate the changes. Make sure you actually need the patch.
• When you’re ready to install the application patch in production, make sure your users know what to expect.

If you identify a problem with an application patch and absolutely need it, see the troubleshooting steps identified at the end of this section.

Testing service patches

Many services are patched frequently. Service patches may incorporate security improvements, additional features, bug fixes, or more. When you see a service patch, don’t just blindly upgrade. Consider the following:

• Read the release notes. Evaluate the reason for the patch. Does it add any features you need? Does it address any bugs that annoy you about the application? Does it improve your security in a way that applies to your computers or your network?
• If configuration files are associated with the service, back them up, at least to another directory.
• If you decide that you need the service patch, test it carefully. Download and install it on your test system. Make a list of downloaded packages, especially if there are dependencies.
• Test the patch on your system. First, based on the release notes, make sure the patch works as intended.
• Identify any differences from the previous configuration files, and assess their impact on your users. Make notes.
• If the service is shared on a network, test the upgrade from another computer. Measure the current network demands on this service. If practical, find some way to simulate these demands on the upgraded service.
• If there are problems with the patch, re-evaluate the changes. Make sure you actually need the patch.
• When you’re ready to install the service patch in production, make sure your users know what to expect.

If you identify a problem with an application patch and absolutely need it, see the troubleshooting steps identified at the end of this section.

Patch troubleshooting

If you have a problem with a patch, the problem may or may not be with the patch. It could be a symptom of a different problem. Because troubleshooting depends on what you’re patching, I can provide only general guidelines here. But as with any troubleshooting, it’s best to apply the scientific method:

1. Observe the problem. Identify its symptoms, using log files where applicable.
2. Formulate a hypothesis of the cause. Was it the patch? Is there a configuration setting which conflicts with the patch? Is there something else you might need to install?
3. Test the hypothesis. If correct, you should be able to test the patch in a different way to confirm your hypothesis.

If you’re having trouble formulating a hypothesis, get support. If you’ve paid for support, such as with a subscription to Red Hat Enterprise Linux (RHEL), use it. Otherwise, make use of the documentation. Many Linux administrators have documented their experiences in HOWTOs, FAQs, and more. They may have experienced the problems you’re seeing now.

If you can’t find the answer in existing documentation, you can try documenting your problem on an appropriate mailing list. They’re frequently available for distributions, applications, and services. Identify the problem, the symptoms you’ve seen, the documentation you’ve read, and what you’ve tried to do to address the problem. Many Linux gurus are motivated to “show off” by solving your problem.