Author: Lisa Hoover
The event will get under way at 12:00 UTC, with at least four moderators available on the #calendar-qa channel on irc.mozilla.org throughout the day to answer questions. Testing will fall into three main categories:
- Bugzilla testing to ensure that fixed bugs are indeed fixed
- Litmus testing to test basic functionality in Sunbird and Lightning
- Usability testing to be sure the final product meets Mozilla Calendar’s goals and standards
In a preliminary ramp-up to Tuesday’s test day, Calendar developers held a test case writing event earlier this month to gather data for Litmus testing, Mozilla’s Web-based test case tracking tool. Hoping to gather approximately 40 cases, developers were pleased to accumulate more than 90 by the end of the testing period. According to Mozilla developer Clint Talbert, the event sparked so much interest within the user community that it was extend an extra day. “A lot of people showed up toward the original end time, and they asked us to keep it going,” says Talbert.
As with last week’s testing, the top two participants in Calendar Community Test Day will win a gift certificate to Mozilla’s online store. Developers say at least 12 people have committed to participate, but that number could climb to 30 or more as the day progresses.
Mozilla developer Joey Minta says participation in the event is a great way for users of all levels to help perfect an application they enjoy using. “Anyone who’s familiar with Sunbird or Lightning is qualified to participate,” says Minta. “Power users and non-power users exercise different areas of the application, simply by their nature, and we’re hoping that a good mix of both will help to ensure the best test coverage.”
Marcia Knous, a member of Mozilla’s QA team, says past test days for Mozilla’s Thunderbird email application and Firefox Web browser have demonstrated the benefit of including the user community in the testing process. “What makes Mozilla successful as an open source project is the fact we get a lot of eyes on our builds. The benefit to participants is that they can help in making our software better by helping to test it, without having to write any code.”
Minta agrees that Community Test Days are a mutually beneficial arrangement. “Participants benefit first and foremost by having better software to use after the next release,” says Minta. “The test days [also] provide a way for the community members to get to know each other and really feel like they’re part of a larger effort to make software that makes people’s lives better.
“The data and bug reports that we gain from test days help make it much easier to identify areas where we need to improve quality. It also helps us understand scenarios that users are likely to get themselves into, as opposed to those that the developer assumes they’ll get into.”
Once the data is collected from the test day, Calendar 0.3 will enter the final stages of preparation for release. “We’re going to look carefully at any tests that users report as FAIL, as well as at any bugs that are filed during the test day,” says Minta. “These (and any) bugs can be nominated to block the 0.3 release. The calendar lead developers will then look through all of these nominations to identify which bugs are critical to the release. Then we can assign developer time to fix those bugs in time for the release.
“In the last week we’ve had a deluge of new code attached to bugs, fixing a lot of long-standing bugs and providing functionality that we didn’t expect to get until a later release. For example, emailing event invitations and accepting those invitations was originally scheduled for a later release, but it looks like we can now have it ready for this one. In an effort to include all those new patches, we’ve extended the release schedule about two weeks, into mid-September.”
Some members of the user community have suggested that testing is best left up to the “Mozilla pros” to ensure a stable product that has been tested by people trained to spot flaws and bugs — a notion with which Minta and Knous soundly disagree.
“‘Mozilla pros’ provide a very valuable service and help ensure that we release high quality software,” says Minta. “However, the circle of ‘pros’ is still fairly small, and asking them to test every inch of a large application isn’t possible. End-users find bugs in software every day. Every time an end user uses a program that crashes, they’ve found a bug that the ‘pros’ didn’t have enough time/resources to spot.”
Knous echoes the same sentiment. “I like to think that we have a lot of ‘Mozilla pros’ on the team, but I don’t think that we are the only ones who can spot significant bugs,” she says. “Some of the best bugs often come from members of our community, since everyone sees different bugs when we look at a build. I think of the entire QA process as being like a finely tuned orchestra. We have community members working in concert with our QA team, exposing bugs, verifying them, making sure they get visibility and get assigned to the right development person to be fixed. The end result is very pleasing to the ears — a [product] that we can all be proud of.”
