Six Steps to Implementation

Many organizations have invested a great deal of effort in educating and informing consumers about health care quality. The most ambitious of these sponsors have embarked on multi-year projects to measure quality of care at various level of the health care system and report their findings to the public. Others with more limited resources and expertise may adapt publicly available materials to their own needs or recommend alternative sources of information for their audiences.

Although no two projects are exactly alike, successful sponsors generally follow the following six steps. Understanding what is involved in implementing a quality measurement and reporting project from start to finish is especially important for those sponsors that do it all themselves. But even sponsors that borrow as much as they can from others need to be aware of these steps so that they can make a conscious choice to skip those that do not apply.

• Step 1. Getting Started.
• Step 2. Collecting and Analyzing Data.
• Step 3. Presenting the Information.
• Step 4. Disseminating Information.
• Step 5. Supporting Consumers.
• Step 6. Evaluating the Project.

Step 1. Getting Started

The first step in a consumer information project is to lay the foundation for the project—financially, politically, and organizationally. While this may sound obvious, many projects fail because sponsors do not take the time to prepare. They dive right into the process of collecting and reporting data, only to be taken by surprise when someone objects to the project or the money runs out.

On the other hand, some projects never even get off the ground because the sponsors can't move beyond the planning process. The key to moving beyond the organizational stage is to develop and stick to a schedule that will force decisions and push the participants forward in the process.

For details on starting up a quality information project for consumers, go to the section on Getting Started.

Step 2. Collecting and Analyzing Data

During the planning stage (Step 1), sponsors choose what types of quality measures to share with the audience.

For guidance, go to Choose Quality Measures.

For many sponsors, the next step in a consumer information project is to gather that data and conduct the appropriate analyses so that these measures have meaning for consumers.

For more information on this step, choose one of the following topics:

• Where You Can Get Information on Quality.
• How to Know When You'll Need Help.
• How to Skip This Step.
• Making Sure Information Is Credible.

Where You Can Get Information on Quality

This section discusses:

• Sources of Quality Data for Health Plans.
• Sources of Quality Data for Provider Groups and Health Systems.

Sources of Quality Data for Health Plans

Sponsors that want to obtain measures of health plan quality have several options:

Measures of Health Plan Quality

Select for Text Version.

Sponsors should note that while a great deal of information about health maintenance organizations (HMO) performance is available, it is much more difficult to get information about other coverage options. Some large purchasers have been trying to develop and report comparable measures for the fee-for-service plans they offer employees. Sources of information include claims records as well as surveys. However, while there is plenty of raw data to work with, it is not yet clear how sponsors will be able to provide equivalent information about these different approaches to coverage and care.

Sources of Quality Data for Provider Groups and Health Systems

Unlike with health plans, measures of the performance of provider groups and health systems are still in the very early stages of development. Although researchers are making progress, there is not yet a standardized measurement set like HEDIS® or a tool like CAHPS® for provider groups.

What can sponsors do in the meantime?

• To get HEDIS®-like measures about provider organizations, sponsors will have to work with the groups to pull relevant data out of medical and administrative records and calculate valid measures where possible. This is expensive and labor-intensive.
• To get measures of experience and satisfaction, sponsors can conduct their own surveys. The Pacific Business Group on Health (PBGH), for example, has been surveying patients since 1998 to generate data on the quality of California medical groups.

In order to generate comparable data, sponsors may want to follow the progress of recent efforts to create a reliable source of survey-based data at the level of provider groups. To learn about an ongoing initiative to develop a standardized CAHPS®-like methodology for assessing the quality of medical groups, select CAHPS® for Medical Groups and Provider Organizations.

How to Know When You'll Need Help

Analyzed Data Available? No Help Needed

Analysis Needed? Consider Some Assistance

If the data exist but require additional analysis to be useful to your audience, you may need assistance from a vendor qualified to conduct the type of analysis you need. For example, a purchasing coalition that offers five health plans to its members' employees could use HEDIS® results to calculate benchmarks or averages for those plans only. While some sponsors can handle an analysis of quality data on their own, others would benefit from expert advice and technical know-how.

Doing Your Own Data Collection and Analysis? Hire Help

If you want to collect and analyze quality data yourself, you would be well-advised to contract with a knowledgeable, experienced vendor. Few sponsors have the technical know-how, the manpower, or the clinical knowledge to generate their own data and do their own analyses. If, for instance, you want to use the CAHPS® survey kit to gather data on your employees' experiences with care, you have to rely on an outside contractor.

The CAHPS® protocol recommends that the survey be conducted by an independent vendor that will take responsibility for collecting the data, analyzing it, and presenting the results in a manner consistent with the CAHPS® methodology.

Sponsors may also need outside help when they combine measures into categories, calculate summary scores, or show relative performance; consultants with statistical expertise can ensure that the sponsors don't misrepresent the performance of competing plans.

How to Skip this Step

Of course, not all sponsors want to or can gather quality information on their own. While you may start out with the intention of doing everything yourself, it is not always practical or affordable to obtain the data you need when you need it. It may also be redundant and thus wasteful to ask health care organizations for information they have already given to someone else.

Rather than follow this path, you may prefer to use information that another party has already prepared, or direct consumers to data provided by others. In many markets, organizations such as regulatory agencies, purchasers, accrediting organizations, and provider associations are collecting health care quality data on a regular basis. Examples of useful sources of quality information include Medicare, the U.S. Office of Personnel Management, and possibly agencies in your own State government.

This approach is efficient and offers sponsors the ability to present more data than they could have ever gathered on their own. The potential downside, however, is that you will probably need to make some compromises. That is, the information made available by someone else may not be exactly what you want, ready when you want it, or applicable to the precise information needs of your audience.

Making Sure Information Is Credible

To ensure the credibility of your data, make sure you understand the methodology that was used to collect and analyze it. Some tools, such as the CAHPS® Survey and Reporting Kit, include specific instructions for sponsors regarding what they need to understand to maintain the validity of data. You do not need to understand all of the technical and statistical details, but you should feel confident about the following:

• The information has been collected and reported in compliance with an established, tested methodology. If the task is being handled by an outside vendor, confirm that the vendor is following an accepted protocol (e.g., using an appropriate sample size).
• The analysis and interpretation of the data has been handled independently of the health care organizations. That is, while these organizations may offer helpful and necessary advice, they should not be making the final decisions.

To confirm that health care organizations have followed the standard procedures for calculating measures, consider having the measures audited or requiring audited measures. To improve the consistency of auditing criteria and processes, NCQA certifies organizations to audit HEDIS® results from health plans. Since 1999, all health plans accredited by NCQA have had to submit HEDIS® results that have been audited by an NCQA-certified vendor.

For measures other than HEDIS®, sponsors may want to consider vendors that can apply the auditing principles recommended by NCQA. To minimize inconsistencies, hire the same vendor to handle all audits and confirm that all auditors use the same criteria in their work.

Step 3. Presenting the Information

Presentation—or how you say what you have to say—plays a critical part in ensuring that your efforts to convey quality information are successful. One of the most challenging aspects of a performance measurement project is figuring out how to present the data in a way that helps consumers interpret the information and apply it to their health care-related decisions. Many sponsors have struggled with this question, only to find themselves making the same mistakes as others before them. One of the key objectives of this site is to support project sponsors in learning from the experiences of their peers as well as from the findings of researchers regarding the best ways to discuss, format, and display information on quality.

Five Key Points About Presentation

Here are the five most important things to remember about presenting information on health care quality:

1. There is no one way to do this—but there are better ways and worse ways.

At this time, there is no universally accepted approach to providing information on quality to consumers. The marketplace is full of experiments, most of which have not been evaluated. But we do know that some approaches to presenting quality information work better than others in the sense that consumers find it easier to understand the data and evaluate their options.

2. The answer that's best for you depends on who your audience is and how they'll use the information.

The way you present information—and even the information itself—should be driven by the needs of whoever is supposed to be reading it. Consumers are the focus of this Web site, but they are not the only audience for health care quality information. Sponsors can develop information for:

• Public or private purchasers.
• Non-purchasing intermediary organizations, such as consumer advocacy groups.
• Policymakers.
• Provider organizations.
• Individual providers.
• Health plans.

Each of these audiences has different needs for quality information. As a result, an approach that is appropriate for individual consumers may not be at all useful for a different audience.

For example:

• Plans and providers may want more detail in order to identify specific opportunities to improve quality.
• Policymakers, on the other hand, may want a higher level of aggregation that would reveal larger trends in the marketplace.
• Intermediary organizations and purchasers may want enough data to draw their own conclusions rather than having the information interpreted for them.

3. Data cannot be presented in a vacuum.

The context you provide (or fail to provide) for the information affects what your audience pays attention to and how they interpret it. In particular, consumers are more likely to care about the information if you can connect it to their concerns about health care and the health care system. For instance, data on ease of referrals has greater relevance to an audience that understands how a health plan may limit their access to specialists. This implies that it is not enough to provide data; a quality report must include some explanation of how the health care system works and what the data reveal about a health care organization.

4. For the typical consumer, a quality measure has no meaning on its own.

It is the sponsor's job to turn quality measures into information that consumers can easily comprehend, evaluate, and use. You can do this by doing one or more of the following:

• Grouping measures into consumer-friendly categories.
• Offering a basis for comparison, such as an average or benchmark for the market.
• Interpreting the information for your audience by making it clear which results are truly better than others.

For explanations and examples of each of these tasks, go to How To Say It.

5. The medium shapes the message.

Whether you rely on printed reports, Web sites, or live presentations, the medium you choose to deliver the information can determine how much you present and how you display it. For instance, you can offer many more layers of detail on the Web than you can in print without overwhelming your audience. A printed report, on the other hand, offers the ability to create large displays of information across multiple pages. This suggests that you will need to have some sense of what the final product will be before you can decide on a design and format for quality information.

Making Sure Your Materials Work for Your Audience

No matter how limited your resources may be, do what you can to test the materials as you develop them to make sure they are suitable for your audience. This approach allows you to identify and remedy trouble spots along the way rather than waiting until the information has been disseminated to discover any problems. Specifically, sponsors can conduct ongoing testing to assess:

• Whether consumers can read the information easily.
• Whether they can understand it.
• Whether the content is appropriate for your audience.
• Whether people are interested in your content (i.e., its salience).
• Whether consumers can use your materials for the purpose for which they are intended.
• Whether they can navigate through the materials to find the information they want.

Techniques for this kind of iterative testing include one-on-one interviews as well as focus groups. 

For specifics on testing your materials, conducting interviews, and focus groups, go to Refining What You Do.

Step 4. Disseminating Information

Long before you have the information to distribute, you need to be thinking about how and when to get it into the hands of consumers. Sponsors should ask themselves:

• What can we do to ensure that our audience is aware of our information and motivated to use it?
• When is the best time to make the information available? Is this timing realistic?
• What channels can we use to distribute the information to our audience? How can we make sure they see it?
• How should we package the information? Should it stand alone or be incorporated into other information?

The answers to these questions will affect how much time you have to produce the report as well as the kinds of measures and depth of content that you can include. For example:

• If the goal is to release a performance report in early fall so that consumers have it in time for the open enrollment season, you may not have enough time to conduct your own survey of enrollees but you could ask your plans to share the HEDIS® scores—which include results of the CAHPS® survey—that they have to report to NCQA by early summer.
• If the most sensible way to package the performance information is to integrate it into open enrollment materials, you will have to deal with space constraints that would not be necessary in a stand-alone document. This may affect the way you present the data or how much data you can include.

For specifics on these dissemination issues, go to Into the Hands of Consumers.

Step 5. Supporting Consumers 

For many sponsors of consumer information projects, the job ends once the reports have been distributed. But both experienced sponsors and researchers testing consumer behavior have demonstrated that it is not enough to give consumers data. They need help in interpreting the information on quality, integrating it with other relevant data (such as costs), and using it to make decisions. Without this help, many simply ignore the information they have, or worse, use it inappropriately.

Strategies for providing "decision support" include the following:

• Refer consumers to consumer advocates and other "information intermediaries" who can help your audience understand and use the information you provide. You may need to provide those intermediaries with training, scripts, and documents they can share with consumers. For example, the Centers for Medicare & Medicaid Services (CMS), formerly the Health Care Financing Administration, funds a Nationwide program called SHIP (State Health Insurance Assistance Programs) to help seniors work through insurance issues and answer their questions.
• Offer consumers a worksheet that guides them through the process of evaluating their options. Worksheets help the reader keep track of the information needed to make decisions. While they can be too complicated for some consumers, they are a valuable tool for the information intermediaries who may assist consumers in reaching decisions.
• Design computer-based systems that facilitate decisionmaking by allowing the user to weight different factors or by ranking the available choices based on the user's response to a set of questions.

To determine which strategy will work best for your audience, you may want to consult with representative consumers and/or appropriate intermediaries. Also, be sure to evaluate your support strategy once it is implemented to find out how well it is serving the needs of your audience and how you can improve it.

For details on these decision support strategies, go to Supporting Consumers in Using the Information.

Step 6. Evaluating the Project 

Finally, the last step for sponsors is to evaluate the extent to which the project achieved its objectives. This could be as simple as asking whether consumers are aware of the information you produced or as complicated as finding out whether and how consumers used the information to help make decisions. 

Evaluations are important for internal purposes because they enable you to determine how effective your project is and how it can be improved. But they are equally critical for external reasons. First, being able to demonstrate that the project has had a positive impact will help you secure political support and continued funding. In addition, an evaluation allows other sponsors to learn from your experience.

The following methods are commonly used to evaluate projects:

• Focus groups.
• Surveys.
• Usability testing.
• Analysis of changes in enrollment patterns.

For specifics on testing and evaluating your materials, go to Refining What You Do.