A dozen tips for testing free software

One of the best ways you can participate in the free and open source software (FOSS) revolution is by helping to test software and reporting bugs and issues to project developers to help them improve their code. Even in the wild and woolly, sometimes undisciplined approach to development that we see in FOSS projects, there are ways to test more effectively. Here are more than a dozen tips suggested by testing gurus and developers that can help you become a successful tester.

When I first began programming in the 1960s, testing was done to prove that code worked. The first big change to that approach came with IBM’s Black Team, which took the opposite tack and tried to break the code — an approach that was radical at the time. In the next popular phase, the color of testing changed, and white box testing, in which you know what a program is supposed to do and check that it does it, was all the vogue.

Those approaches come from the world of closed source, proprietary software. Testing in the FOSS world is different. For one thing, bug reports can come from anywhere, not just from a team assigned to test the product. The transparency of the code invites everyone to explore.

Former Debian Project Leader Martin Michlmayr advises, “If you want to test something, run the latest version. This may sound obvious but apparently it’s not. If you run Debian, upgrade to unstable and maybe pull in some packages from experimental. For other software, try the latest version from the project’s version control system and see what breakage you can find.

“You have to be creative, and you have to be quick. You’d be amazed how many bug reports we get for issues that have already been fixed. It’s therefore important to run the latest version and to check for bugs in the bug tracker. If you cannot run the latest version all the time, at least try to verify whether the issue you see still applies to the latest release.

“Regarding the bit about being creative: we get a lot of bug reports in Debian and really appreciate them. But there are many issues that many people find, and that can be found easily. What really helps is if someone is creative about finding things, or maybe just very patient. You don’t necessarily need to read source code, although this may be a good exercise for someone trying to learn how to code. Someone could also take the manual or man page of a program and compare it with the actual behaviour — I bet there are plenty of examples where the docs are out of date.

“Try to do unusual stuff with the software or test it in unusual environments, such as especially slow hardware or uncommon platforms.”

In addition to Michlmayr, I got tips from GNOME bug master Luis Villa, Debian bug-finder extraordinaire Dan Jacobson, and
Dave Freese, author of several ham radio applications for Linux. They came up with these tips for better testing:

• Run the latest version of the software you’re testing
• Check for duplicates before filing a bug report
• Include enough information in your report so the issue can be reproduced
• Don’t apologize for your language
• Use a bug tracking system of some sort
• If possible, write automated unit tests
• If possible, use the code you’re testing under real-life conditions
• Use automated crash reporting tools
• Become familiar with the tools that will be used to compile, link, and test the code
• If possible, maintain a separate environment for testing
• Keep a few back revisions of the code to track when errors appear
• Describe as completely as possible the conditions leading up to a fault
• Your “newbie impressions” are critical — report everything you see
• Be patient with developers

Michlmayr also suggests several pages for additional reading:

How to Report Bugs Effectively
Bug Writing Guidelines
How to Write a Useful
Bug Report
Lessons from GNOME Project QA

Testing is a great way for non-programmers and even non-geeks to contribute to the greater good. Get involved with your favorite free software projects and send them a bug report or two.