|
 |
 |
 |
Test & Refine |
Develop the Test Plan
The first step in each round of usability testing is to develop a plan for the test.
Don't make it a long narrative that is difficult to read. Use headings and bullets so it is easy to scan.
The purpose of the plan is to think through and write down what you are going to do and get approval for that so management and the entire team (design and usability) all agree.
If the test plan is well-prepared, much of it can be reused as the background information in the report after the test.
Who prepares the test plan?
Typically, the usability specialist meets with the rest of the team to decide on the major elements of the plan. Often, the usability specialist then drafts the plan, which circulates to management and the rest of the team. Once everyone has commented and a final plan is negotiated, the usability specialist revises the written plan to reflect the final decisions.
What goes into a plan?
This table gives you a quick overview of the parts of a typical plan. The rest of this article gives you more detail about each of these parts.
|
What are you testing? |
|
|
What concerns, questions, and goals is the test focusing on? |
|
|
When and where will the test take place? |
|
|
How many users of what types will you recruit? |
|
|
What will participants do with the product in this round of testing? |
|
|
What will you ask at the beginning and end of the session? |
|
|
What will you count? |
|
|
What system will you use for testing? Will you be videotaping and/or audiotaping? Will you be using a specific technology to capture data? |
|
|
Who will do what in the usability test? |
The reason for including this information in your plan is for management and other team members to understand and comment on what you plan to do. If they raise questions or issues about the plan, you must negotiate to resolve any questions or issues. In the end, your plan will drive the usability test and you do not want management or other team members to have expectations of the test that you will not be meeting.
Scope
Indicate what you are testing: Give the name of the Web site, Web application, or other product. Also specify how much of the product the test will cover. Examples might be:
- the prototype as of a specific date
- just the navigation
- navigation and content
Purpose
Identify the concerns, questions, and goals for this test. These can be quite broad; for example, "Can users navigate to important information from the prototype's home page?" They can be quite specific; for example, "Will users easily find the search box in its present location?"
In each round of testing, you will probably have several general and several specific concerns to focus on. Your concerns should drive the scenarios you choose for the usability test. (Of course, when conducting the test, also be alert to any usability issues that emerge that you had not included in your original list of concerns.)
Schedule and location
Indicate when you will do the test. You may want to be specific about the schedule. For example, you may want to indicate how many sessions you will hold in a day and exactly what times the sessions will be. (You will need that information to recruit participants, so you must decide the schedule early on.)
Also indicate where the test will be held. Again, you need that to recruit participants as well as to let observers know where to come to see and hear the test.
How long should each session be? Typically for an informational Web site, participants come for one hour. To test a Web application, you may want to schedule 90-minute sessions. Leave yourself a little time between sessions to reset the environment, to briefly review the session with observers, and to take care of personal needs.
How many sessions should you do in a day? Watching and listening to people working with a product can be exhausting, particularly if you are watching participants really struggle. Some teams schedule as many as 15-16 sessions in a single day. Others limit themselves to two to four sessions in a day.
Do we need a usability lab for testing? No. You do not need a lab for testing. If you have access to one, use it. Most usability labs have cameras, microphones, recording equipment, etc. so they have an environment all ready to go. Most also have separate observation rooms where people other than the participant and facilitator can watch the test. These days, rather than having one-way glass to watch through, many usability labs feed the audio and visual into the observation room. If you do not have access to a usability lab, you can still do usability testing. You can test in a conference room, hotel room, the participant's work or home space, or other venue where you can arrange to meet users.
Participants
One of the most important parts of the test plan is a clear indication of the number and types of participants to be tested. This can range from a general set of user profiles to specific personas that represent each of the major user segments. The more precise the user descriptions, the easier it will be to develop a screening questionnaires that will bring you the types of people you need as test participants.
How many participants? The number of participants is frequently dictated by the budget and time allotted for the test. If you can have as many participants and as much time as is needed, you can calculate how many that should be. We give you three useful methods here. Even if you will be constrained by budget and time, you may want to understand these three methods so you can decide which is appropriate for the test you are doing.
Method 1 relates the number of participants you need to the number of usability problems that you expect to uncover. According to the formula behind this method, if you are testing an early prototype or a particularly difficult site, you need only a few users (four to six) to uncover the most serious problems.
Method 2 is the one to use if you must make strong inferences from the participants to a larger population. For example, if it is important to feel confident that certain usability objectives are truly met, then using "95% confidence intervals" is a good choice. This could show that a minimum of 12 to 15 people will be needed for each unique group (type of user) and possibly as many as 20 to 30, or more.
Method 3 is the one to use if you must ensure that a usability test has sufficient statistical power to identify real differences when they exist. This method is most appropriate in comparison testing, when you are measuring against a previous product or against competitors. In this case, the number of people could be at least 35 (or more) in each unique group.
Scenarios
Scenarios are very short stories that motivate participants to attempt tasks with the product. At this stage (drafting the test plan), concentrate on listing the tasks that you will test through the scenarios rather than on the final wording of the scenarios. We cover how to Create Final Scenarios in another article.
How do you decide on tasks for a usability test? Consider these three questions:
- How does the task relate to one or more of the items that you listed in the purpose section of the plan?
- How frequent and/or how important is each task for users of the Web site?
- How susceptible a particular Web site element to a usability issue? For example, you may want to test items that you think are a usability issues to see if they really are.
Relate each task to the items in the purpose section. The purpose section tells you what you are trying to learn about the Web site. The tasks and scenarios are the vehicle for that learning. A task (scenario) is only valuable for the usability test if it is going to help you answer one or more of the concerns, questions, or goals you listed in the purpose section. (One scenario may address more than one of the items under purpose. You may have several scenarios that address one of the items. The relationship is not 1:1, but you must have at least one scenario for each item in your purpose section and every scenario must address at least one of your concerns, questions, or goals for the test.)
Select scenarios that rate high on frequency and/or importance. The Web site must allow users to do their most frequent and most important tasks, so you should concentrate on those in testing the site. You can find out which are the high frequency and most important tasks through your Task Analysis (for example, user surveys and contextual interviews and observations of users at work) as well as other usability activities, such as Card Sorting. You may also get useful information from Web logs, search logs, and other ways of tracking what users do at your site.
How many scenarios do you need? For a typical one-hour usability test of an informational Web site, you probably want to end up with about ten scenarios. (Of course, depending on your Web site, users may be able to accomplish more scenarios - or fewer - in the time of your testing. Consider how long a task will take typical users to complete.) Consider conducting a pilot test to see how many scenarios can be realistically in one hour.
How many scenarios should we include in the draft plan? For your first draft of the test plan, consider including several more than you will use (perhaps as many as twice what you need). This will allow you to discuss the relative importance and value of different tasks/scenarios with management and the rest of the team. Through negotiations (dropping, adding, changing tasks/scenarios), you should arrive at a set that everyone agrees are:
- realistic
- relevant to your purposes
- frequent and/or important to users
Lastly, the test plan should indicate whether the scenarios will be presented in random or sequential order.
Questions
The plan should include the types of questionnaires you are going to include, when and how you will ask the questions, and the specific questions you will ask. If you are going to use pre- or post-test questionnaires, you should include that in the test plan.
Most usability tests include three types of questionnaires:
- Participant profile questions
- Questions when the user first sees the site
- Questions at the end of the session
Participant profile questions. These allow you to validate that the person represented himself or herself correctly when recruited. You may also want to add a few other questions to learn more about the participant than you covered in the recruiting screener. Participant profile questions might be about age, gender, education, work experience, and computer or Web experience.
You can ask participant profile questions as:
- a questionnaire that the participant fills out on arriving for the session
- in a brief interview at the beginning of the session
Advantages of using a written questionnaire:
- can be done before the session starts (saving session time)
- no interviewer bias
Advantages of asking the questions as an interview:
- serves as an "ice-breaker" to make the participant comfortable
- can probe to be sure information is accurate and useful
Questions when the user first sees the site. You may want to get the participant's initial impressions of the site by asking a few questions before starting the scenarios. These might be:
- What site is this? How can you tell?
- Who do you think the site is for? How can you tell?
- What do you think you will find at this site? How can you tell?
- What are your initial impressions of the site?
Questions at the end of the session: You probably want to get participants' impressions of the site after they have worked with it. You can do this with:
- open-ended questions
- closed questions using radio buttons, check boxes, or a Likert scale (rating something on a numerical scale)
- a standardized satisfaction questionnaire
If you ask open-ended questions, it is critical to ask them neutrally, so you do not bias the participant to give you a particular answer. Here is a set of six open-ended questions for a usability test that focused on navigation and search:
- What is your overall impression of the site?
- How would you describe finding what you were looking for on this site today?
- What is your impression of the search capability?
- What did you like best about the Web site?
- What did you like least about the Web site?
- If you were the Web site developer, what would be the first thing you would do to improve the Web site?
Data to be collected
In addition to qualitative notes about what you see and hear in a usability test, you probably want to collect quantitative data.
Success measures. Indicate in the report how you will determine "success" for each scenario.
Time measures. Indicate if you are going to time each scenario and what time measures you are going to take. (Total time to complete the scenario? Separate time figures for navigating and for understanding [time spent on content pages]? Time to recover from an error?)
Error measures. Indicate if you are going to measure errors and what you are going to count as an error.
Number Pages. Indicate if you are going to count the number of clicks to information.
Pathway. Indicate if you are going to need users' paths through the Web site.
Satisfaction Rating. Indicate if you are planning to administer a stisfaction questionair.
Set up
The test plan should include information about the computer system you will be using. This should reflect the typical system that most users of your site work with in their daily lives. Include information about:
- monitor size and resolution
- operating system
- browser
You should also indicate if you are planning on videotaping or audiotaping the test sessions and if you will need any special usability testing tools.
Roles
Include a list of the staff who will participate in the usability testing and what role each will have. The usability specialist should be the facilitator of the sessions. The usability team may also provide the primary note-taker. Other team members should be expected to participate as observers and, perhaps, as note-takers.
Next steps
Once you have a test plan in place, you are ready to Create Final Scenarios and Recruit Participants.
|
Related Templates & Examples |
|
Related Methods |