blank

Testing Your Materials and Dissemination Strategy

Given all the time, money, and effort you're investing in developing and distributing a quality report, you want to be sure that what you're doing is as effective as it can be. In other words, you want every aspect of the report to work as well as it can for your audience. The best way to do this is to test each aspect of your report and your dissemination strategy with your audience during the developmental phase of the project. This kind of effort is especially important when you are introducing information that is new and complex.

Select links below to learn about:

Improving Your Materials.
Improving Your Dissemination Strategy.
Where to Seek Feedback.
When to Test.
What to Do with Results.

Improving Your Materials

This section answers the following questions:

Why Test Materials With Your Audience?
What Should You Test?

Why Test Materials With Your Audience?

The main reason to test materials is to get outside of your own head and see the information from your audience's perspective. While you may know a lot about the issues in your report, you cannot rely solely on your own judgment or that of your colleagues about the materials you produce—even if you have a great deal in common with your audience, but especially if you don't. The problem is that your understanding of the issues and terminology isn't representative. Also, details that at first glance seem inconsequential to you may be important to your audience. Moreover, you may not even be aware that your materials aren't consistent with the cultural norms of certain groups within your audience.

At its best, testing at the developmental stage (sometimes referred to as "pretesting") can offer valuable insights into how your audience thinks and the reasons for certain attitudes or misunderstandings. These insights can help you refine your messages and materials so that they communicate clearly and effectively. The information you glean from testing can also help you identify language that will be meaningful to your audience and issues that may be culturally sensitive.

What Should You Test?

Workbook Reminder
Question
26

You can test materials at any stage in the development of your report—from the initial brainstorming of concepts and strategies to the tweaking of a mock-up just prior to distribution. In general, testing reveals how the audience responds to different characteristics of your report: whether they think it's attractive, personally relevant, persuasive, understandable, and acceptable. To help them make these assessments, you can give them one version to respond to, or you can let them compare versions of the design, the visuals (such as photos or graphs), or the copy to see if they like one better than another.

Before you initiate this kind of research, form a clear statement of your objectives. What is it that you want to learn through testing? What are you prepared to do with what you learn? For example, you may want to know whether the language you used to provide a context for the quality data in your report is appropriate for your audience. Depending on the response you get, you may need to rewrite the text to address concerns that are more salient to your audience, or to use terminology and phrasing that they find more understandable.

Deciding What to Test

To help decide how to focus your testing efforts, consider the following questions:

How much do you know about the target audience?
How much do you know about them in relation to this issue (e.g., their concerns about health care quality)?
Is this issue new, controversial, sensitive or complex (for your audience, in your market)?
Have you conducted related research that you can apply to this topic?
Can you afford to make a mistake with a particular message or audience?

Adapted from: Making Health Communications Programs Work: A Planner’s Guide. U.S. Department of Health and Human Services. NIH Publication No. 92-1493. April 1992.

There are several things you may want to test:

Messages.
Text and Graphics.
Design and Navigation.
Usability.

Test Messages

Once you have learned something about your audience, you can start to develop and test different motivational and educational messages that will speak to their health care and information needs. Focus groups can help you determine:

Which messages resonate with the audience? Which ones do they find most appealing?
Which ones use the same language that your audience would use?
Which are most understandable to the target audience?
Which grab their attention?
Which are most credible?
Which are personally relevant? For your message to be relevant, the audience must understand the problem (that quality isn't consistently good across plans and providers), accept its importance in their lives (that this variation affects the quality of care they receive), and agree with the value of the solution for them (that they can use information to make better decisions).
Which are confusing, or have controversial or sensitive elements? Testing can reveal whether a message may alienate or offend your intended audience.

Test Text and Graphics

Show your audience pieces of your report, and eventually a final draft of the text with accompanying graphics, to confirm whether they find it to be clear, easy to read, and culturally appropriate. One-on-one interviews are usually the most effective way of obtaining this kind of direct, detailed feedback. Use these interviews to find out:

What people in your audience like and dislike.
What they think is helpful (or not).
Whether the level of detail is appropriate.
How they interpret what they see.
What they would change.
Whether they trust the accuracy of the information and its source.
How they see themselves using the materials.

The last version they review should be as much like the final product as you can get without investing in any expensive production.

Ask for feedback about any charts, tables, or other graphics you want to use, including photographs and drawings. Make sure that people understand and interpret them in the way in which they were intended, even if they're just meant to be decorative. For example, some people in your audience may not understand that a specific drawing has a purpose (e.g., to help the reader identify related information on different pages of the report). Others may presume that a photograph of young families means that the materials are not relevant for older adults. Or they may be offended by the use of cartoon characters or photos of people unlike themselves; some older people with disabilities, for instances, feel excluded when they see photos of healthy, active peers.

Although you may be using a template for graphs or tables that has already been tested with consumers (e.g., those recommended for CAHPS® survey results), don't assume that what worked for "John Q. Public" will work for your audience. A variety of factors will affect how your audience understands a graphic and whether they are attracted to different elements of a report. While such templates are an excellent starting point, you need to make sure that the contents and presentation of your report suit your audience. Testing will allow you to identify problem areas and tweak them as necessary.

Test Design and Navigation

The design of your quality report can play a major role in determining its understandability and effectiveness. Design involves more than colors and fonts. It encompasses decisions related to the sequencing of content, the layering (or "chunking") of text and data, as well as the aesthetics of the document. It's especially important to remember that a design that is aesthetically and artistically appealing (at least to some people) is not necessarily the one that will do the best job of communicating quality information. Let your audience tell you what works and what doesn't. Specifically, ask questions to learn the following:

Is the design confusing in any way? Is the reader able to find specific pieces of information quickly and easily?
Is the reader able to move through the document or Web site easily? Is it clear how to get from one topic to another?
Is the information organized and presented in an effective way? Is it easy for the reader to read and use the information?
Is the design attractive?
Is there anything about the design that makes the materials hard to read (e.g., small font size, overlapping colors)?
Is the design offensive in any way? Is there anything in the design that your audience could perceive as jarring or off-putting? For example, some people are offended by materials that seem too "slick"—i.e., like something that a health plan would produce to advertise its services. This can undermine their ability to trust the credibility and objectivity of the contents.
Are the colors appealing to your audience? Are there certain colors that would be more culturally appropriate (such as colors that your audience may associate with health care or with high quality)? Also, are you using colors or combinations of colors that could pose a problem for people with vision problems?

To learn more about the use of color in report design, go to Designing Your Report.

Test Usability

The purpose of usability testing is to learn what may prevent consumers from using your materials and what could help them use it. To test usability, you need to find out how consumers interpret and use the material. It's important that you get them to actually use the information, not just ask them how they would do so.

It is especially critical to not confuse preferences with usefulness. If consumers like what you show them, that doesn't tell you whether the information "works"—i.e., whether they can use the information to perform a task (typically to identify the health plan or provider that is best for them). For example, people often say that they prefer to get more information, but they don't actually make better decisions when they have more data.

For more information on this topic refer to the following article:

Hibbard JH, Slovic P, Jewett JJ. Informing Consumer Decisions in Health Care: Implications for Decision-Making Research. The Milbank Quarterly 1997;75(3).

You can conduct usability testing with just part of a report (e.g., a table that shows comparative data), but you may get more useful feedback if you can show a complete mock-up so that the experience is as real as possible. While real-world data is best, it's fine to use fictional information as long as it's realistic. It's best to conduct the testing during the developmental phase of your project so that you can apply what you learn, but some sponsors gather information about usability after the materials are disseminated as part of their project evaluation. For example, you could get information about usability by debriefing new employees to see whether and how they used the information on quality when choosing a plan.

Another way to test usability is to ask consumers to choose the health care organization that would be best for someone else with specific circumstances. For example, to see how consumers might apply the CAHPS® information in a report, researchers asked interviewees to use the information to pick a health plan for a fictional person with certain needs and concerns (e.g., a woman with small children or an elderly couple with chronic health problems). By observing and asking follow-up questions, the researchers were able to identify specific barriers to the use of CAHPS® information.

Ask People to Show—Not Describe—the Use of Information

To assess the usability of nutrition labels, the Food and Drug Administration used objective tasks where they could evaluate the accuracy of responses. For example, consumers were asked to:

Name all the differences between 2 products.
Explain how eating one product (e.g., with a high fat content) might affect what else they eat during the day.
Evaluate advertising claims given information on nutritional value.

Improving Your Dissemination Strategy

When designing a dissemination strategy for information on quality, sponsors often leap to conclusions about the best (or easiest or cheapest) way to get the information out to the audience. And they pay a high price when those conclusions aren't correct—usually because the intended audience never actually sees the information. To minimize the chances of this happening to you, think of your audience as a key source of information about which distribution channels and media are most likely to be effective. For example, you can use focus groups to find out:

Which channels and media are most accessible for your audience?
Which are most trustworthy?
Which specific media does your audience prefer? Do different segments of your audience prefer different kinds of media?

The developmental stage of your project is also the time to consult with the people who will be at the front line of the dissemination effort such as the human resources staff, the senior center staff, the mailroom staff, the people who run the server, and the brokers. Talk to them to learn about the feasibility of different approaches, the timeframe they need to get the work done, and the support they will need.

It is especially important to begin this information gathering process in the early stages of your project, rather than waiting until the end when the report is complete. This is because the dissemination strategy you choose can strongly influence the design of your report. For example, you may conclude from the feedback you receive that your audience would be most receptive to summarized quality information incorporated into other health benefits information. In that case, a lengthy, stand-alone report could be a poor fit with their needs as well as a waste of your project resources.

Where To Seek Feedback

Perhaps the best way to design an effective report card is to involve stakeholders in its development. Testing is a key strategy for folding audiences and intermediaries into the process. It is also a way to get critical feedback and buy-in from the health plans and providers whose quality is being reported.

Most Important Source: Your Primary Audience

The only way to find out what information would be valued and understood by your audience and where they are most likely to see it is to ask them. While expert opinions and standardized tests (such as readability tests) are helpful, they are not substitutes for getting a response directly from the audience. For example, you can run a test on a computer that will tell you that a document was written at a 6th grade reading level. But only by testing the document with your audience can you find out for sure whether they comprehend what they're reading—which goes beyond reading words to understanding and linking concepts, interpreting what they see, and grasping the connection to other issues.

When you design tests to get feedback from your audience, consider how the literacy level and acculturation of your audience might affect the questions you can ask and the responses you can expect. Also, if your audience is composed of different subgroups, be sure to recruit representatives of each group to respond to your materials and discuss dissemination options.

Another Source: Secondary Audiences and Information Intermediaries

A second important source of feedback are the people who deal with your primary audience on a regular basis, often developing a thorough understanding of their needs as well as some influence over their decisions. This group includes:

Secondary Audiences.
Information Intermediaries.
Health Plans and/or Providers.

If you can get information from these other sources, try to do it at the same time that you're testing materials with your primary audience. This will allow you to review all the responses at the same time.

Secondary Audiences

Depending on your intended audience, secondary audiences may include:

The employee benefits staff.
The children of elderly parents.
The staff of community groups and agencies (including senior centers).
The staff of help lines (i.e., those most likely to field questions from consumers).
Employers and union leaders.

Because these informants may be no more representative of your audience than you are, use some caution in how you apply the feedback you get from them. But do take their comments seriously, especially at the earlier stages of your project when they can offer broad insights into your audience's attitudes, behaviors, preferences, and likely responses to information.

Information Intermediaries

If you can identify people (or organizations) who are likely to serve as information intermediaries for your intended audience, be sure to give them an opportunity to review and comment on the materials you are developing and the dissemination strategy you are considering. This is important because these intermediaries may serve as gatekeepers for the information—that is, they may be the ones who control the distribution of materials. If they are part of the development process, they are more likely to feel like they have a stake in what you are doing. But if they don't get a chance to contribute, and they disagree with your approach, your reports may never get into the hands of your audience.

In addition, thanks to their regular interaction with the audience, intermediaries may be very familiar with the needs and behaviors of certain population groups, so they may be able to provide insights that would be hard to get by yourself.

Health Plans and/or Providers

Unless you are dealing with a large number of organizations, you can usually use an informal approach to get feedback from representatives of health plans or provider groups. For example, you may want to let them review elements of the report and voice any concerns in a brief series of meetings. Or you may want to interview them one at a time so that each organization has the opportunity to be heard. Since health plans and providers will worry about the public perceptions that your materials will create, you can expect them to focus on the tone (i.e., is it positive or negative?), the use of terminology (namely, the risk of misinterpretation), and who is getting the credit for providing the information.

When to Test

Testing can take place at any and every stage of the development of your quality report. During the early stages of your project, you may want to get reactions from your audience to a few key messages and graphics you are considering. This is also a good time to gather information about the media and channels that your audience would prefer. The findings from this testing process can point you in the right direction as you develop the information for the report.

Later, you would want to recruit people to review a mock-up of the materials you have developed to confirm that your audience will understand them, find them appealing, and regard them as culturally appropriate. Your final test prior to distribution might focus on the usability of your document.

Keep in mind that testing will help you identify problems, but not necessarily solutions. As a result, it is not a one-shot deal. You should plan on multiple rounds of testing to refine the different aspects of your report. For example, you may find in the first round of testing that respondents are confused by a table that shows a comparison of health plan performance. Based on the feedback you get, you may do another round of testing with a table that uses different symbols, then a third round that lets respondents compare the revised table with one that has a different design. Of course, both time and resources are usually limited, so do what you can. In nearly all cases, some testing is better than none at all. The exception would be a situation where an effort to cut corners leads to biased findings: for example, if you had a staff member conduct a focus group with colleagues rather than hiring a professional facilitator to lead a group of people more representative of your audience.

To make this kind of ongoing testing feasible, it is important to plan for it by incorporating multiple rounds of testing into the overall schedule for your project. Recognize that each step of the testing process takes time, whether or not you use professional research firms to help you. Experts in this kind of work estimate that one round of testing can take two to three months:

Contracting with research firms	(if necessary)
Arranging for the required facilities (in-house or external)	1-2 weeks
Developing and testing the questionnaire, interview guide, or protocol	2-3 weeks
Recruiting interviewers and respondents	2-4 weeks
Gathering feedback	1-2 weeks
Summarizing the results	1 week
Total time	7-12 weeks

Select for Text Version.

(Adapted from Making Health Communications Programs Work: A Planner’s Guide. U.S. Department of Health and Human Services. NIH Publication No. 92-1493. April 1992.)

After each round of testing, it will take more time to make any necessary changes and test again. This substantial time commitment is another reason why it's good to test materials simultaneously with multiple audiences (e.g., your primary audience as well as any important intermediaries).

What to Do with Results

Use Your Judgment

What you hear in testing are the opinions of respondents who represent your intended audience. They can offer hints and guidance, point out problems, and reveal how they are likely to use a document, but they cannot provide solutions or clear direction. This means that you have to use your own judgment to determine how best to use the feedback to make your materials more salient and attractive to your audience. For example, if you do not get consistent answers to some of your questions, you'll know you have a problem and may get some ideas about how to fix it. But it is up to you to decide how to fix it and what to test in the next round. You may want to test just the piece that was changed; it's not necessary to do full-blown testing every time.

Be Cautious

The most common error made in testing is over-generalizing the results. Keep in mind that the respondents in these small-scale tests are not fully representative of the public, and their responses cannot be projected to the population as a whole. For example, if 8 of the 10 participants in a focus group reacted negatively to a message, you can't conclude that 80 percent of the audience wouldn't like the message. Don't allow yourself (or your partners) to use the responses from testing to estimate large-scale results. The responses can only indicate problems or areas that need work.

Be Fair

As you make changes in response to testing, take care to ensure that the information you're providing in the report remains accurate and that you continue to portray the results in a way that is appropriate and fair to all of the health care organizations. If the changes you are making could affect how health plans or providers are perceived, give them an opportunity to review the revisions before the report is finalized.