Usability Updates Newsletters

June 2005

Paper Prototypes Work as Well as Software Prototypes

by Dr. Bob Bailey

Introduction

Many, if not most, website developers like to wait until they have a software-based (usually HTML-based) prototype before they attempt to do usability testing. The question becomes, “Is a high-fidelity prototype necessary to have a meaningful usability test?” A high-fidelity prototype is computer-based, and usually allows realistic (mouse-keyboard) user interactions. A low-fidelity prototype is usually paper-based, could possibly be a series of sketches or printouts, and does not allow realistic user interactions.

High-fidelity prototypes are assumed to be much more effective in collecting true human performance data (e.g., time to complete a task), and in demonstrating actual products to clients, management, and others. Whereas, low-fidelity prototypes seem to be helpful in enabling early visualization of alternative design solutions, which helps provoke innovation and improvement.

The First Study

A few years ago, Virzi, Sokolov and Karis (1996) attempted to determine if in the later stages of user interface design the low-fidelity prototypes were as effective as high-fidelity prototypes in identifying usability problems. Most usability professionals agreed that low-fidelity prototypes were useful in the early stages of design, but not necessarily in the later part of the design process.

Why? Low-fidelity prototypes can be limited in one or more of the following dimensions:

• Amount of functionality - They usually do not include all of the system features.
• Interaction capabilities - They usually do not allow a mouse and keyboard interaction.
• Aesthetic refinement - They usually do not use screen colors, or high level graphics.

Virzi, et al. compared a computer-based prototype (high-fidelity) with a paper prototype of screens that performed the exact same activities (low-fidelity). Their results showed no reliable (statistically significant) difference in the number of usability issues detected using the two different prototypes. They concluded that the two approaches appeared to have the same sensitivity, and to be tapping into the same pool of potential problems. A second study using a different application resulted in the same findings.

Follow-on Studies

A couple of years later, Catani and Biers (1998) had participants use prototypes to perform four typical library search tasks. They used both high fidelity (computer-based) prototypes and low-fidelity (paper) prototypes. All interfaces were exactly the same except for the medium of presentation. Again, they found no reliable difference in the number of usability issues found with the different prototypes. In fact, there was considerable commonality in the specific problems uncovered.

The same year, Uceta, Dixon, and Resnick (1998) had participants use prototypes to order various food and beverage items. They compared user performance using either a computer-based prototype or a paper prototype. All interfaces were exactly the same except for the method of presentation, and contained the same number of usability problems. Uceta et al. reported no reliable difference in the number of usability issues found with the different prototypes. However, testing with the paper prototypes took about 30% longer than with the computer-based prototypes.

More recently, Walker, Takayama and Landay (2002) compared user observations using either high-fidelity or low-fidelity prototypes. Participants carried out online banking tasks on two different websites using the two different prototyping conditions. They measured differences in the number, severity and type of usability issues identified. They found no reliable difference in the number of usability issues detected between low- and high-fidelity prototypes.

Finally, Sefelin, et al. (2003) attempted to determine if different ways of presenting the prototypes affected the participants’ critique of the websites. They developed a computer-based prototype and a paper prototype for two different systems. Again, the functionality of the comparative prototypes was identical. In a usability test, each participant used one paper-based and one computer-based prototype (using different applications).

The two types of prototypes produced essentially the same quantity and quality of critical user observations (there was no reliable difference). However, 92% of the test participants preferred working with the computer-based prototypes. The researchers recommended that developers only use paper prototypes when computer-based prototyping tools do not support the ideas the designer wants to implement, or when all members of the design team need to be able to create prototypes.

Conclusions

Taken together, the results of these studies seem to support the contention that one can extract about as much information from a usability test using low-fidelity (paper) prototypes as from one using higher fidelity (computer-based) prototypes. In other words, low-fidelity prototypes appear to be as effective as high-fidelity prototypes at detecting many types of usability issues. Low-fidelity prototypes have an additional advantage in that they can be created quickly and easily, and they do not require advanced computer skills.

High-fidelity prototypes, however, tend to be preferred under many circumstances, such as when low-fidelity prototypes are much too limited in the features they represent, or the low-fidelity prototypes are not sophisticated enough to allow evaluation of important user interface characteristics. Also, high-fidelity prototypes sometimes are required when usability issues will be identified differently depending on the type of prototype, e.g., the time to complete a task or to make the first click on a page. Finally, high-fidelity prototypes are sometimes required for marketing purposes (to better show a product) or for communicating with developers to help them more clearly understand some aspect of a prototype.

________

Catani, M.B. and Biers, D.W. (1998), Usability evaluation and prototype fidelity: Users and usability professionals, Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, 1331-1336.

Sefelin, R., Tscheligi, M., and Giller, V., (2003) Paper prototyping – what is it good for? A comparison of paper-and computer-based prototyping, Proceedings of CHI 2003, 778-779.

Uceta, F.A., Dixon, M.A. and Resnick, M.L. (1998), Adding interactivity to paper prototypes, Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, 506-511.

Virzi, R.A., Karis, D. and Sokolov, J.L. (1996), Usability problem identification using both low- and high-fidelity prototypes, Proceedings of CHI’96, 236-243.

Walker, M., Takayama, L. and Landay, J.A. (2002), High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes, Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting, 661-665.