Ingimp’s tools may improve FOSS usability

Since May, ingimp, a modified version of the GIMP, has collected daily logs on what users do with the program in the hope of improving its usability. The richness of this data is unprecedented, yet improving the GIMP is only a sideshow for the project. What ingimp is really designed to do, according to the project’s leader, is develop the software and practices to put free and open source software (FOSS) usability testing on a professional footing “without placing an undue burden on either the developers or users.”

Michael Terry, the assistant professor at the David R. Cheriton School of Computer Science at the University of Waterloo in Canada who heads the project, explains that ingimp is working with the GIMP largely because he is familiar with the code base and has been a minor contributor to the GIMP in the past. Currently available for Debian, Ubuntu, and Windows, ingimp assigns users a random identification number, and then collects information on which commands they use, how they use the interface, the types of documents they open, and the operating system and hardware they are using. Participants can tag a session with information about what they are doing, and have the option of disabling logging for a session. The resulting data is posted to the project Web site for anyone to see.

Explaining the reasons for ingimp, Terry says that usability procedures were developed long ago for commercial companies or lab research. However, “they don’t always cleanly translate into the socio-cultural environment of free and open source software development. You want the sorts of practices in which anyone can contribute.”

Currently, usability software for FOSS is practically nonexistent. Terry points to the Debian popularity-contest package, which collects data on which software users install, and Fedora 7’s Smolt, which collects hardware data, as two examples that are presently in use, but, in general, the amount of FOSS usability software and project practices for usability compare poorly to those for contributing a software patch.

Without suitable software and practices, developers trying to decide how to prioritize features or modifications can only make decisions based on what Terry calls “gut feelings and anecdotes. What we’re trying to do is provide a more comprehensive view of how the software is being used in the wild, so that the data can be referred to when decisions are made.”

As an example of the kinds of thing ingimp is learning, Terry noted in exchanges following Slashdot’s coverage of ingimp that, so far, 1024×768 resolutions are more common than his research team had anticipated. One reader wrote back suggesting that the reason was that this was the default resolution for GNU/Linux. However, by looking at the data and noting that most of those using ingimp were Windows users, Terry was able to discount that explanation with concrete figures. Instead of relying on hunches or unsystematic observations, he was able to frame the discussion with a precision that has generally been lacking in FOSS usability decisions until now. Through his work, he hopes that such discussion in the future “can be grounded in the data, so we’re not just shooting from the hip. My hope would be that we’re going to provide the baseline, a reality check.” Although software cannot completely replace traditional live observation of users, Terry does suggest that it can provide a unique view that can enrich it.

Terry is quick to put to rest one set of fears about how the data will be used, and emphasizes the privacy and transparency features build into the study. Only then does he go on to say, “It’s turning out to be a very rich set of data that we’re collecting.” With the project intended to be open-ended, ingimp has the potential to collect data over an extended period of time, showing how individual users’ work habits change over time or with different types of documents, providing a chronological perspective that is almost unheard of in standard usability testing — largely because it would be fabulously expensive.

The research team plans to issue detailed analyses of the data, but, after only a few months of operation, Terry says, “We’re just starting to mine the data.” While he does not rule out eventually making specific recommendations, he is understandably reluctant to make any analysis after only a couple of months of the project. However, he does suggest that ingimp promises “to quantify the user base” — that is, to provide FOSS developers with a clear picture of the hardware used with the software, the types of users, and what commands are used together. For example, he suggests that if GIMP developers want to add more floating windows, ingimp could provide information on monitor resolution that could tell them whether enough users had high-resolution monitors — and, therefore, the screen space for added features. Such information might be useful not only to GIMP developers, but also to those working on other projects.

The project is also developing “personas” — stick figures that are wearing or carrying different objects that graphically represent user profiles — that can tell participants what category of users they fit into, and how their usage changes over time.

Meanwhile, people outside the ingimp team have started using the team’s data. Terry has noticed that people looking at the collected statistics frequently turn first to the list of most commonly used commands. This observation may eventually help to improve online help files and tutorials, but, for now, Terry is considering adding links to the list that would help users learn more about the GIMP.

Before making any deeper analysis, Terry would like to see the data enriched in other ways. One high priority is to find an RPM packager to make ingimp available on other distributions. Another question that the research team would like to explore is why anyone might not want to use ingimp, so that their needs might be met, or, at the very least, the limits of the data collected better understood. Understanding which categories of people will not use ingimp would help the researchers better understand the categories of those who do.

Despite ingimp’s promise, Terry says it is not a complete replacement for interviewing users and observing how they use the software. “When you have a real live person in the lab, and you have a trained person doing the observations, you’re going to uncover a lot of good data about how the software can be improved. Our data is never going to have that sort of rich insights. Instead, it is going to characterize in broader terms how the general community uses the software in its day to day lives.”

Ideally, Terry sees traditional usability testing and ingimp as complementary. For example, if the logs revealed that many GIMP users were pressing Ctrl-D to create a duplicate of an image, then immediately closing the duplicate, analysis of the logs alone might cause researchers to conclude that this was a common typo. However, by interviewing people, the researchers might find that this keyboard combination was being used because, in Photoshop, it is used to deselect objects. In turn, this insight could help to estimate the number of Photoshop users were being logged, and to learn more about how their behavior changed as they grew accustomed to the GIMP.

In the end, Terry says, “If we can create a system that can collect this type of data, then richly characterize the user base and tie that data into the design and development cycle, that would be great.” However, what he would most like to see is ingimp providing the standards for other FOSS usability software and practices. “There’s a lot of design efforts we’ve made to protect people’s privacy and to make the process completely transparent. What we would like to do is generalize these principles and make them available to the public, so that others who would like to do something similar will be able to learn from our experience.”