Techniques for Testing and Evaluation

Many sponsors are inhibited from testing materials and evaluating their project because they are not familiar with the methods available for these purposes. This leads them to skip the testing step during development, or to take short-cuts to assess the impact of a project. However, while some effort is usually better than none, a poorly conceived approach to testing or evaluation can be worse than nothing if it leads to a wrong conclusion or misses important effects.

The purpose of this section is to provide an overview of common techniques that can be used for both testing and evaluation, depending on how you apply them. These methods can be grouped into two categories:

• Qualitative Methods.
  These include one-on-one interviews and focus groups.
• Quantitative Methods.
  These include surveys or other sources of data, such as enrollment or administrative records.

Each of these methods can be used for different purposes at different stages of your project. The best method for you will depend on the nature of the materials, the intended audience, the questions you are hoping to answer, and the amount of time and resources available. But there is no formula and no perfect method; they all have their pluses and minuses.

The bottom line: Getting input is more important than the method you use. Choose whatever method or combination of methods suits your project and is most practical. A combination of methods is especially useful because it allows you to overcome the limitations of a single technique. For example, if participants in focus groups raise some concerns about your materials, you can use one-on-one interviews to probe into those concerns in greater detail.

Select links below to learn more about the following:

• Qualitative Methods.
• One-on-One Interviews.
• Focus Groups.
• Common Issues:
  Recruiting Participants and Selecting Interviewers/Facilitators.
• Quantitative Methods.
• Surveys.
• Other Sources of Data.
• Choosing Which Methods to Use.

Qualitative Methods

In a qualitative assessment, the sponsor elicits reactions and opinions from potential and actual users of information on health care quality. You can use qualitative methods for different purposes at different stages of your project. For example, you can use focus groups in the initial stages of the project to learn about your audience's needs and awareness of quality issues. You can also use them during the evaluation stage to help interpret and elaborate on some of the data you are collecting (e.g., if a survey indicates that people didn't read your report, you can use focus groups to learn why). One-on-one testing can provide critical feedback early in the development phase about the way you portray data or explain technical language, and then again towards the end of that phase to reveal problems with navigation or other aspects of the design.

Select links below to learn more about:

• One-on-One Interviews.
• Focus Groups.
• Common Issues:
  Recruiting Participants and Selecting Interviewers/Facilitators.

One-on-One Interviews

An interview is an opportunity for a single individual to talk at length about specific elements of your materials. The people you interview may be drawn from your audience or from key informants, like physicians, family members, information intermediaries, or other people who may have some influence with your audience (such as leaders of community and faith-based organizations). You may also want to interview health plan representatives to learn about their concerns and opinions related to your materials.

Interviews with members of your audience are a useful method for gathering information about how people are likely to understand and interpret what you're telling them about quality. You can get feedback and suggestions that will tell you the following things about your audience:

• Whether they can read the information.
• Whether the information is relevant and salient to them.
• Whether they understand the contents.
• What they understand about its meaning—and whether that understanding is consistent with what the information was intended to communicate.
• Whether they trust the source of the information and the accuracy of the information.
• How they interpret what they're reading.
• Whether they like the way the information is organized and can find what they are looking for.
• Whether there is other information they would like to see included.
• Whether they would use the information and how.

Interviewing is also a way to explore responses to different approaches you may be considering. You can have people compare two versions and say which one they prefer and why. In addition, interviewing can suggest how consumers might use the materials to perform specific tasks.

For example, in a task-oriented interview, you could ask the respondent to use the information to identify which plan best meets certain criteria or to recommend a plan for a friend with a specific set of health care concerns.

For more information on testing the usability of your report, go to Testing Your Materials and Dissemination Strategy.

For details, select links below to read about:

• The Pros and Cons of Interviews.
• The Practice of Cognitive Interviewing.

To read about how interviewing is done, select An Overview of One-on-One Interviews. 

The Pros and Cons of Interviews

Interviews are a great way to get detailed feedback from either a few people or a large number of people who are representative of your intended audience. They are flexible: you can do informal interviews that allow you to gauge an initial reaction, or in-depth interviews that reveal a more detailed response to your materials. And they can elicit honest, personal responses; in a private encounter, people tend to be more willing to speak their minds than they are in a group setting.

However, interviewing can be time-consuming and labor-intensive. To learn more about the costs associated with interviewing, select An Overview of One-on-One Interviews. Moreover, you can't really generalize the conclusions you can draw from interviews. Be very careful about using what you learn in interviews to predict what the rest of your audience would think or do. Nor is it always obvious how to turn the reactions you get into changes in the materials. While interviews can offer valuable insights, you won't always know what to do with what you're hearing.

Finally, while individual interviews avoid the intimidation factor of a group interview (such as a focus group), people are often reluctant to be overtly critical in a one-on-one situation. The interviewers must lay the foundation for an honest assessment by clearly explaining that the purpose of the interview is to learn what the respondent really thinks, and that specific, critical feedback is exactly what they want.

The Practice of Cognitive Interviewing

A cognitive interview is a special kind of interview that focuses on how people absorb information, think about what they see, and draw implications or conclusions. Through cognitive interviews, you can learn whether readers can understand the information that is presented and interpret its implications appropriately. The questions are specifically designed to reveal differences between what you, the sponsor, think the materials mean and how the intended audience actually interprets them. Cognitive interviews can also give you an early indication of whether the intended audience will be able and willing to use the information in decision making.

Select to read more about How Cognitive Interviewing Works.

Focus Groups

A focus group is a special kind of group discussion that is designed to elicit information about the wants, needs, viewpoints, beliefs, and experiences of your intended audience. Focus groups can help you better understand the expressions and terminology commonly used by people in your audience, as well as their attitudes and beliefs about health care. They are especially good at helping to uncover the reason behind people's responses, e.g., why they prefer certain terms or why they would or would not do something. As a result, focus groups can also play a useful role in evaluations by shedding light on short-term outcomes and helping to explain findings from surveys. However, since the number of participants in a focus group is too small for them to be completely representative of your audience, this is not a good technique for evaluating long-term outcomes.

Select links below to read about:

• How a Focus Group Works.
• How a Focus Group Is Different from an Interview.

How a Focus Group Works

In a focus group session, an experienced moderator (also known as a facilitator) leads a carefully planned discussion on a specific topic, listening closely to the conversation and urging each person to share their thoughts and experiences. Although the moderator uses a discussion outline to guide the conversation, participants are encouraged to talk freely and spontaneously.

Sponsors should recognize that the group dynamics of a focus group can have an unpredictable effect on the participants. For example, one person may dominate the group, inhibiting others from speaking up. Or the candid comments of one participant may spur others to be more honest in their responses. A skilled moderator should be able to limit the impact of a domineering personality and capitalize on the strengths of the methodology.

After the session, the moderator will provide you with a written report that summarizes any insights, trends, patterns, and themes that arose during the discussion. Typically, snippets of individual comments are used to illustrate certain ideas; however, the moderator's job is not to relate individual opinions but to present the larger issues raised by the group.

For more details, select An Overview of Focus Group Methodology.

How a Focus Group Is Different from an Interview

There are two key differences between a focus group and a one-on-one interview:

• In a one-on-one interview, all of the interviewer's attention is on the individual respondent. In contrast, each participant in a focus group has a limited amount of time to talk.
• Focus groups are well-suited to learning about people's preferences for things like language, dissemination vehicles, key messages, etc. But a group discussion cannot tell you how an individual understands, perceives, or would use the materials. If you want to probe into individual responses (or if you need quantitative information), a focus group is not the right approach. One-on-one interviews are ideal for getting information about the understandability and usability of specific materials.

Common Issues

No matter what method you use to get qualitative feedback from your audience and other stakeholders, you will have to deal with the following two issues:

• Recruiting Participants.
• Selecting Interviewers/Facilitators.

Recruiting Participants

Taken as a whole, the participants in an interview or focus group should be a representative cross-section of your intended audience. If your quality report is intended for employees, recruit respondents from that pool. If it's aimed at the public at large, you'll need a random sample of consumers. If there are important subgroups or secondary audiences, be sure to include them as well.

To learn why this requirement doesn't apply to focus groups, select An Overview of Focus Group Methodology.

To the extent possible, your goal is to recruit a wide variety of people within the demographics you're trying to reach. When you select participants, try to include individuals from different segments of the intended audience, such as people of different ages or different employment status.

Select links below to learn about the following:

• How to recruit the public.
• Recruiting in a central location.

How to recruit the public

For sponsors preparing a community-wide report on quality, it is important to get a broad-based sample to participate in both the testing and the evaluation stages. To find potential respondents, look for the easiest way to get access to the people who are the most representative of your intended audience and who meet your criteria for inclusion and exclusion. When establishing those criteria, consider whether you want to use the same criteria for both testing and evaluation. For example, you may want to emphasize people with lower literacy during the testing phase to ensure that your materials will be widely understood.

Do it yourself. One approach is to focus on locations (or organizations) where you are likely to find people who are part of the community you're trying to reach. Some possibilities include:

• Religious organizations or churches.
• Civic groups.
• Small, locally owned businesses.
• Health organizations or providers that serve your audience.

That said, the people who are easiest to get may not give you the most complete or accurate feedback. You also want to hear from those who don't find it that easy to participate: those who are working, those with childcare responsibilities, and those with no transportation. To encourage their participation, you may need to offer help, such as childcare assistance or money for transportation.

You can save some money by recruiting people yourself, especially if you can work through your organization or your partners (such as participating employers or health plans) to identify potential respondents. A professional firm may do this more efficiently, but doing it yourself is better than not doing it at all.

Hire a professional. Another way to recruit participants for testing or evaluation of your materials is to hire professional recruiters such as market research firms or mall intercept firms. They have the skills and resources to identify and contact respondents who are representative of your audience. In addition, they offer access to trained interviewers and appropriate facilities. However, keep in mind that professional firms have their own "sampling frame"—you'll need to make sure that it is broad enough to meet your needs, and that they can apply your criteria for including and excluding potential respondents.

Recruiting in a central location

One way to quickly identify a large number of people for testing is to conduct central location intercept interviews. To learn more, select How Central Location Intercept Interviews Work. These are commonly known as "mall intercepts" because shopping malls are a convenient place to find people who meet the profile of a given audience. But the approach can be used in clinic waiting rooms, work sites, schools, government agency offices, Laundromats, or anywhere else you are likely to find representatives of your audience. The main advantage of this approach is that a high-traffic area allows you to recruit participants and conduct a number of on-the-spot interviews in a reasonably short time. Also, you may be able to get feedback from hard-to-reach segments of your audience very efficiently if you can identify a central location where they are likely to be.

The respondents recruited through "intercepts" may not be statistically representative. However, if you use this approach to do quick interviews or even impromptu focus groups, you can reach more people than you could hear from through scheduled focus groups or in-depth interviews.

Selecting Interviewers/Facilitators

Whether you choose to interview people individually or in groups, it is important to use interviewers (or facilitators) who are skilled in drawing responses from different kinds of people and keeping a conversation on track. Professional interviewers are trained to be neutral and nonjudgmental, and to lead discussions and ask questions in a way that is clear and succinct.

When you are selecting an interviewer or facilitator, look for someone with experience who will be sensitive to the cultural issues and concerns of different groups within your intended audience. If possible, use an interviewer with the same demographics as the intended audience; consider ethnicity, age, income level, and gender. Finally, while it is helpful to have some knowledge of health care and quality issues, interviewers only need to know enough about the purpose and topic of the individual or group interview to understand what they're hearing and ask good follow-up questions.

Here are a few other ways to learn about the person you may hire as a moderator:

• Meet with the person to get a sense of his or her behavior and interpersonal skills. This is important because the personality of the moderator can influence the quality of responses you get. Does the person seem tactful and kind? Does the person listen well and ask good questions?
• If possible, review any audio or video tapes of sessions conducted by that moderator.
• Look at copies of reports or interviewing guides that the moderator has written. Most moderators write part or all of these materials.

If you don't have the resources to hire a professional, look within your organization (or those of your partners) for someone who has experience in this area. The key is to find someone who can be objective and won't be defensive when talking with respondents about your materials.

The information above is adapted from: McGee J. Writing and Designing Print Materials for Beneficiaries: A Guide for State Medicaid Agencies. Baltimore, MD: Health Care Financing Administration, Center for Medicaid and State Operations. HCFA Publication Number 10145. October 1999.

Quantitative Methods

Although qualitative techniques can provide a great deal of valuable detail, you may want to use quantitative techniques to get a better sense of what is happening on a macro-level. This approach is not commonly used in the developmental stages of a quality reporting project, but it can be very useful in the evaluation stage when you want to assess changes or differences resulting from the information you've distributed. 

For more details on evaluation, go to Evaluating the Results of Your Project.

Select links below to read about:

• Surveys.
• Other Sources of Data.

Surveys

To evaluate the short-term outcomes of your project, you may want to conduct a formal survey of your audience. In this method, you ask questions (usually closed questions) of a statistically representative sample of your audience. Their answers can reveal how people think about the issues, what's important to them, what they thought about the report, and whether and how they used the materials you distributed. Depending on how you handle the survey, you can learn a great deal about the effect of your project:

• If you survey your audience both before and after you distribute the report, you can use the data to determine whether their knowledge or opinions changed. For example, survey data may suggest that the number of people aware of differences in quality rose by 15 percent (or not at all).
• If you collect demographic, behavioral, and attitudinal data, you may be able to detect differences in the impact of your project across subpopulations in your audience. For example, you may find that some people are more interested in your information than others. From surveys of consumers who received report cards in Denver and St. Louis, researchers identified several characteristics that were associated with people who were more likely to remember seeing the reports:
  • Women.
  • People already oriented towards using written comparisons of products.
  • People with a general interest in health and health care.
  • People who have a need to know because they are considering switching plans.

Select to learn more about How to Conduct a Survey. 

Other Sources of Data

One technique used for evaluation purposes is to analyze sources of data other than the reports of audience representatives. The idea is to look at the choices that people actually make after they receive information rather than what they say they do.

In the context of report cards, this approach usually entails measuring changes in health plan enrollment and relating those changes to the quality and cost performance of those plans. For example, the Buyers Health Care Action Group (BHCAG), a business coalition based in the Twin Cities, has been analyzing changes in enrollment in the care systems that provide health care services to employees and their families. For at least two years, BHCAG has been able to see a movement from care systems with high costs and poor quality to those with lower costs and higher quality.

While this kind of analysis can be very powerful, sponsors need to be cautious about three things:

1. It will take time for quality information to become pertinent to people's decisions. If you look too soon, you are unlikely to find evidence of any relationship between performance and enrollment decisions.
2. Few people change plans in a given year, so it can be statistically difficult to show an impact. For that reason, some experts suggest that sponsors focus on the decisions that people make when they have to make a choice (e.g., because they are new to a program) or change plans (e.g., because their previous plan is no longer an option).
3. Finally, it can be a serious challenge to separate out the impact of other factors that drive decisions, such as differences in benefit design. One of the reasons why BHCAG can conduct its analysis is that the care systems are required to offer a standardized benefits package, so enrollees are comparing apples to apples.

Choosing Which Methods to Use

You can use the techniques described above in combination, or you may want to use different ones at different points in the project. It all depends on what you are want to know and how you would use what you learn. Experts suggest that the choice of methods rides on the following seven issues:

• The characteristics of respondents (whether they're heterogeneous or homogeneous).
• The amount and complexity of material you're asking people to react to.
• The anticipated effects of being in a group.
• The sensitivity of the topics to be discussed.
• The depth of individual responses you want.
• The degree of continuity and connections among topics.
• Logistics.

Select to learn more about Focus Groups or In-Depth Interviews: Which Method to Use?.