Learn About Usability Testing

• What is usability testing?
• What are you looking for in a usability test?
• How does usability testing fit into user-centered design?
• When should you do usability testing?
• What can you learn through usability testing?
• What should you keep in mind when usability testing?
• Do you need a lab to do usability Testing?
• How many participants are needed for a usability test?
• How much does it cost to do usability testing?

What is usability testing?

In a usability test, representative users try to do typical tasks with the product, while observers, including the development staff, watch, listen, and take notes. The product can be a Web site, Web application, or any other product. It does not have to be a finished product. You should be testing prototypes from early paper-based stages through fully functional later stages.

top of page

What are you looking for in a usability test?

In each round of usability testing, you should first identify specific concerns and goals for that round of testing and develop the test to focus on those concerns and goals. For example, at the beginning of a project, you may be testing to set quantitative baselines (such as time, error rates, and satisfaction) for comparison to later tests of your revised site. For another example, if you have Set Measurable Usability Goals, you may be trying to see how well the product is meeting those goals.

In a typical usability test, you want to:

• identify any usability problems that the product has
• collect quantitative data on participants' performance
• determine participants' satisfaction with the product

top of page

How does usability testing fit into user-centered design?

Usability testing is a major part of user-centered design. A user-centered design process should include a series of tests developed specifically to evaluate both performance and preference.

top of page

When should you do usability testing?

Test early; test often. Usability testing lets the design and development teams identify problems before they get "set in concrete." The earlier those problems are found and fixed, the less expensive the fixes are. As the project progresses, it becomes more and more difficult and expensive to make major design changes. The more you test and change based on what you learn, the more confident you can be that the site will meet your objectives and your users' needs when it is launched.

Iterating—developing a prototype, testing it with users, analyzing the test results, changing the prototype based on the findings, and then repeating the test, analyze, revise cycles—is the best way to produce a successful Web site or Web application.

top of page

What can you learn through usability testing?

In a typical usability test, you may want answers to these questions:

• Are the test participants able to complete the task scenarios successfully?
• Considering successfully completed tasks, how fast do participants do each task?
• Considering successfully completed tasks, how many pages (clicks) does it take to complete each task?
• Do participants perform well enough to meet the usability objectives?
• How satisfied are participants with the site?
• What changes are needed to make sure that the site will enable more users to perform more successfully?

You might also have more specific questions. For example, if one of your concerns for this round of testing is how well the search function works for users, you might focus on these questions:

• Do participants click to pages or do they use search?
• What words do they use most when searching?
• Is the search box in a good location and is it large enough for most of the words used?
• Do the search results provide leads to quick answers to users' questions?
• When search results do provide answers, are the answers usually on the first page of results?
• Does the search do a good job of detecting and helping to resolve typing errors?

top of page

What should you keep in mind when usability testing?

Four important points to keep in mind:

• You are testing the site not the users.
• Rely more on what you learn about performance than preference.
• Make use of what you learn.
• Try to find the best solution given the reality of your many users.

You are testing the site not the users

For some users the term "testing" has a negative connotation. We try hard to ensure that participants do not think that we are testing them. We help them understand that they are helping us test the prototype or Web site. In fact, don't use the term "testing" with participants at all. Instead, invite participants to help by "trying out the prototype."

When users have difficulty completing a task, we fix the Web site, not the users. Both out loud and in your thoughts, ask "How well does the site allow typical users to meet their goals?" rather than "How well do users do on the site?"

Rely more on what you learn about performance than preference

We can measure both performance and preference. Performance measures include success, time, errors, etc. Preference measures include users' self-report of their satisfaction and comfort.

Some designers believe that if they design to meet users' preferences, the site will enable users to perform well. The evidence does not support that.

In fact, people's performance and preference do not always match. One study found that about 70% of users had performance and preference measures that agreed with each other. That is, they either performed well and liked the Web site or performed poorly and disliked the site.

However, this leaves a relatively large percentage of people (30%) for whom performance and preference measures did not agree. They either performed well and disliked the site or performed poorly and liked the site.

Various reasons have been proposed for why people often rate a site more highly than their performance would lead us to expect. They may blame themselves for the problems that they have, rather than the site. They may think they would be hurting our feelings by giving the site a low score. They may not be aware of the problems they had and think they were successful when they were not. All of these reasons support the recommendation to rely more on what you learn through performance measures than through preference measures.

Make use of what you learn

Usability testing is not just a milestone to be checked off on the project schedule. A usability test is not finished when the last participant leaves. The team must consider the findings, set priorities, and change the prototype or site based on what happened in the usability test.

Try to find the best solution given the reality of your many users

Creating any product, including most Web sites and Web applications, requires considering many different users with different ways of working, different experiences, and different questions and needs. Most projects, including designing or revising Web sites, have to deal with constraints of time, budget, and resources. Balancing all those is one of the major challenges of most projects.

As you weigh constraints and trade-offs, push for creating the Web site or Web application that will enable the greatest percentage of your users to successfully answer their questions and complete their tasks. Research shows that the cost of supporting unsuccessful users after a product is launched is far greater than the cost of fixing the product to meet users' needs before it is launched.

As you consider your personas, their scenarios, and what you learn in usability testing, try to find the ideal solutions to your design problems - given the needs of the different users you have identified. Settling for less than the best means having users be less successful than they otherwise would be. Evidence shows that even after extended use and experience, when they have a less than optimal product to work with, users never achieve the success they would have if they had had the better interface.

top of page

Do you need a lab to do usability Testing?

No. You can do usability testing in either a formal or informal setting. In any type of setting, your methodology can also range from formal to informal.

You can do effective usability testing in any of these settings:

• a fixed laboratory having two or three connected rooms outfitted with audio-visual equipment
• a conference room, or the user's home or work space, with portable recording equipment
• a conference room, or the user's home or work space, with no recording equipment, as long as someone is observing the user and taking notes
• remotely, with the user in a different location

You should be doing usability testing even if you do not have a fixed laboratory or access to one. Do not let anyone say, "We can't do usability testing because we do not have a usability lab." Just do it! Do it in any space you can find.

top of page

How many participants are needed for a usability test?

It depends. A typical range is from 8 to 16 (per user group) each test. If each user works with you for an hour, that means one or two days of testing, peruser group.

You might need only four to six people to help you find serious problems, if you:

• are doing paper prototypes or are in early development
• plan several rounds of testing throughout development
• have a fairly homogenous user population

If you have different potential user groups (for example, physicians, patients, researchers), try to include representatives of all these groups. If you are likely to have users with a range of Web or computer experience, try to include both less experienced and more experienced users.

If you want to conduct formal quantitative testing on your products or systems, you'll need more people to derive statistical results. For diagnostic usability testing, six to eight users are usually enough to uncover the major problems in a product.

If you do iterative (repeated) usability testing over the course of developing the Web site, many users will participate in testing one or another version of the emerging site. Thus, while you may have fewer than 10 participants in each usability test, you may have 15 to 30 people who have tested some version of the site before it is launched.

top of page

How much does it cost to do usability testing?

Cost depends, of course, on the size of the site, how much you need to test, how many different types of users you anticipate having, and how formal you want the testing to be.

Having a standard process and reusable materials makes usability testing faster and less expensive. If you or your recruiting firm develops a database of users, recruiting becomes less time consuming and, therefore, cheaper.

Consider these elements in budgeting for usability testing:

• time to plan: identify issues to focus on in testing, identify types of users to involve in testing, write a screening questionnaire to recruit users, write scenarios for users to follow
• cost of recruiting: time of in-house person or payment to a recruiting firm (often a good option)
• time of usability specialist to become familiar with the site and of team to do a dry run to see how scenarios work with the site
• cost of renting laboratory space or a portable lab or other videotaping equipment, if you choose tovideotape sessions.
• time of team to observe users (conduct the test)
• cost of paying participants or gifts for participants
• time to consider what the team saw and heard, identify problems, recommend solutions to those problems
• time to discuss changes with developers, write up report of findings and recommendations

Remember to budget for more than one usability test. Building usability into a Web site (or any product) is an iterative process. You will find it more valuable to use your budget to do a few small tests throughout development than to do just one large test at the end.

top of page

Next steps

Now that you understand usability testing, you may want to review the steps in usability testing. The first step is Develop the Test Plan.

top of page