Getting Started

Experienced sponsors of consumer information projects have learned that proper planning can determine whether an initiative succeeds or fails. Critical elements of the planning process include the following tasks.

• Assess Your Local Environment.
• Profile Your Target Audience.
• Clarify Your Objectives.
• Identify the Focus of Your Report.
• Choose Quality Measures.
• Identify Potential Partners for the Project.
• Agree on a Message.
• Find Sources of Financing and Establish a Budget.
• Agree on a Schedule (and Stick to It).
• Establish a Management Structure.

Assess Your Local Environment

Select below for a list of questions on the following topics:

• Availability of Choice.
• Players.
• Current Level of Activity.
• Costs and Benefits.

While you may not be able to find answer to all of these questions, the search for information will help you identify the factors that are most likely to affect your project and its effectiveness.

Availability of Choice

What kinds of choices do consumers have in your market?

• Can they choose among different kinds of coverage options (e.g., fee-for-service carriers, PPO plans, HMOs)?
• Do they have multiple choices within each of those options?

Assuming you are trying to reach a subset of all consumers in the local market, what choices does your particular audience have?

• Do they have different coverage options?
• Do they have multiple choices within those options?
• Will the comparative information you can provide be relevant to the choices they can make? If not, what purpose would the information serve?

Players

• Do managed care plans dominate the market?
• Are the plans primarily National or local?
• Who owns and manages these plans?
• Do providers—either hospital systems or medical groups—play a large role in shaping the market?
• Among purchasers, who are the leaders in your market?
• Is a regulatory body involved in quality measurement in your marketplace?

Current Level of Activity

• Who else is providing information about quality in your market?
• What information do they provide?
• How long has this information been available?

Costs and Benefits

Why do you want to do this? Are you doing this voluntarily (e.g., as a private purchaser) or because you are mandated to report on quality (e.g., as a State regulator)?

What are the potential political implications of providing information on health care quality?

• Who might be affected by this information?
• What reaction can you expect from them?
• Who is likely to become an ally in this effort?
• Who may be opposed to your project?

What are the implications for staffing?

• Do you currently have the manpower to undertake this kind of project?
• Do you have the expertise in-house?
• If not, can you afford to hire or contract for it?

Profile Your Target Audience

Who is the audience you want to reach with information on health care quality?

Typical primary and secondary audiences for quality information include:

• Active and retired employees, usually referred to as commercial enrollees.
• Medicare beneficiaries and those who advise them.
• Medicaid recipients.
• Parents of children eligible for State-sponsored health insurance programs.

What are their demographic and socioeconomic characteristics? 

Gather information about the following:

• Gender.
• Geography: Where do they live and work?
• Income: What percent are poor, middle-class, or wealthy?
• Age: To what extent are different age groups represented?
• Family status: Are they buying coverage for families or individuals?
• Cultural/ethnic background: What cultural or ethnic groups are included? What languages do they prefer to read? What languages do they use to communicate?

• How much education do they have?
• How comfortable will they be with the kind of information typically offered in report cards?

What are their health care needs?

• Where does your audience tend to go for health care?
• What kinds of services do they require?
• What kinds of information would help them make health care-related decisions that match their values?
• How likely are they to be interested in information on quality?

Expect to find variety even within populations that have many characteristics in common. Some will be more interested in the information than others, and some will be better equipped to understand and apply it. The challenge is to meet the needs of different segments in a way that allows them to decide for themselves how much information they want to see.

For more on how a comprehensive profile of your audience can help you decide what to say and how to present information, go to Know Your Audience.

Clarify Your Objectives

While all of these objectives are important, sponsors of quality measurement projects need to be clear about what exactly they are trying to do. First, this kind of clarity will allow the organization to stay focused on those activities that will move it closer to its goals. Second, the audience is less likely to be confused if the sponsor isn’t trying to do too much at once with the information.

Keep in mind that the objectives you establish for your project will become the focus of your evaluations once the project is completed. This suggests that you need to be realistic about what you can accomplish and to think ahead about what you’ll need to do to assess your progress appropriately. For instance, you may want to conduct a baseline survey now so that you’ll have a point of comparison when you talk to consumers after they receive your information.

Once you have established your goals, you can begin to:

• Identify the Focus of Your Report.
• Choose Quality Measures.

Identify the Focus of Your Report

Select below to learn more about:

• Deciding Where to Focus.
• How to Handle Multiple Answers.

Deciding Where to Focus

To decide which of these levels to focus on, you may want to consider the following five factors:

• Contractual relationships.
• The level at which consumers exercise choice.
• Feasibility.
• Available funds.
• The message you want to convey.

Keep in mind that you will probably need to make trade-offs, in the sense that you may not be able to satisfy all of these criteria at the same time. For example, while a focus on small physician groups may speak to your audience’s interests and support your message, it may be neither feasible nor affordable.

Contractual relationships, particularly for those sponsors that purchase health care services

Purchasers usually want to measure and report on the quality of whoever it is they contract with, whether plans, care systems, or medical groups. Those that contract with only one plan may pay more attention to local health systems or large medical groups in order to provide consumers with information at a level that is useful to them.

The level at which consumers exercise choice

For consumers, comparative quality information is not relevant unless it shows real differences and reflects real choices available to them. If employers offer a choice of four plans that all contract with the same provider groups, consumers will have little use for data that merely shows how similar they are. On the other hand, they may be very interested in seeing information about the provider groups they can select within each plan.

Feasibility

Wanting to measure and being able to measure are two different things. The feasibility of measuring and reporting quality at each level depends on multiple factors, including political obstacles, technical challenges, administrative issues, and costs.  For instance, to determine the quality of individual practitioners, you would have to overcome the problem of "small numbers." This refers to the fact that few if any physicians have enough patients with specific diseases for any data based on them to be statistically meaningful. Or you would have to juggle the logistics of contacting and surveying thousands of respondents in order to have a sufficient amount of data for each practitioner. The impracticality of this kind of project is one reason for the emergence of a more feasible middle ground: the measurement of quality at the level of medical groups or clinics where they exist.

Available funds

The choice of level is often dictated by the amount of money that is available for the quality measurement and reporting effort. The number of organizations has huge implications for the cost of the project. For instance, even if it were technically feasible to evaluate the quality of thousands of practitioners, focusing on the five to ten health plans or the thirty or so provider organizations in a given marketplace results in a much more affordable project.

The message you want to convey

The level you focus on should support the message you are trying to communicate to consumers. For instance, if your goal is to educate consumers about the wide variations in quality, measures of health plan performance may not be appropriate. The problem is that some of those measures represent the average of the quality offered by all provider groups within a plan. Let’s say that Plan A reported a childhood immunization rate of 70 percent based on the performance of its three provider groups, where one immunized 90 percent, the second immunized 80 percent, but the third immunized only 40 percent. As this example illustrates, plan-wide results can hide serious differences in quality of care. For that message to come across, you would want to focus on a level of the health care system where variations in quality are clear.

Similarly, if the message emphasizes the consumer’s ability to make choices, the information should reflect the level of the system at which consumers can actually exercise that choice.

For more detail, go to Agree on a Message.

How to Handle Multiple Answers

When you weigh the five factors explained above, you may conclude that your audience would benefit from information about more than one level of the health care system. Or they may want overall information about a level as well as details about the components of that level, such as the specialty departments within hospitals.

Some sponsors handle this situation by reporting on the quality of two levels at the same time. The Pacific Business Group on Health, for instance, provides performance information on health plans as well as medical groups.  This approach helps consumers work through the multiple levels of decisions they have to make (i.e., which plan to choose, then which group to choose), but it requires a large amount of resources in terms of money, staff, and technical expertise.

A less resource-intensive approach is to rotate what you measure each year: providing one set of measures in odd years and a second set in even years.

The benefits of this alternating strategy are that consumers still get both sets of information and that the sponsor’s annual costs are more manageable. The downside is that the two sets of information are not based on data from the same year. However, some sponsors have noted that the performance of health plan organizations does not change that quickly, so consumers are unlikely to miss anything important by getting data every other year rather than every year.

Choose Quality Measures

In the health care context, a quality measure is essentially any piece of data that tells you something about how health care is being delivered or how people respond to that care. For example, a measure may say what percentage of people are receiving preventive services (such as mammograms and immunizations), whether providers are qualified to deliver clinical services, whether patients feel better as a result of care, or what percentage of patients are receiving the information they need to maintain their health or take better care of themselves.

Select to learn more about:

• Types of Measures You Can Report.
• Where You Can Get Measures.  
• Some Criteria for Choosing Measures.
• Research on This Topic.

Types of Measures You Can Report

The following types of measures are common in consumer-oriented quality reports:

• Measures of process indicate what the health care organization does to maintain or improve health, such as screening members for diseases, providing preventive care, counseling patients to give up unhealthy behaviors, or delivering care that minimizes any deterioration in health. Process measures are a key component of HEDIS®, the Health Plan Employer Data and Information Set, developed by the National Committee for Quality Assurance (NCQA).
• Measures of accessibility help consumers understand how easily and quickly they will be able to receive the care they want. While some experts do not consider access to be a reflection of technical quality, consumers tend to regard it as a critical indicator of how responsive the organization will be to their needs. Typical access measures include the time it takes to get an appointment, the time spent in waiting rooms, and the time it takes for someone to answer the telephone. To get these measures, sponsors may either ask health care organizations to report them or survey consumers about their experiences gaining access to care.
• Measures of experience tell consumers what actually happened to other people in the health care system and how satisfied they were with the care they received. Satisfaction measures have been around for a long time, but they have undergone some recent changes. First, they evolved from an emphasis on satisfaction only to a broader portrayal of consumers’ perceptions of their interactions with the health care system. The measures have also become standardized so that they can be compared across plans, sponsors, and markets. These changes were driven by the Federally funded development of the CAHPS® (Consumer Assessment of Health Plans) Survey and Reporting Kit, which enables sponsors and health care organizations to customize a standardized set of questions to their audiences and analyze and present these items in a comparable format.
• Measures of outcomes give consumers a sense of the effectiveness of care that health care organizations deliver. Examples include surgical mortality rates (the percentage of patients who die), the health status of breast cancer patients, and the rate of readmission after receiving hospital care. Outcomes measures are intuitively appealing but hard to deliver. One problem is that health care organizations bear only partial responsibility for the outcomes of care; genetics and personal behaviors play a large role as well. Another is that most organizations, even huge health plans, do not have enough cases of different outcomes for a given condition to make measures statistically meaningful. Finally, to the extent that organizations have enough cases to report, results must be adjusted to account for differences in patient characteristics that can affect the outcomes of care. This is possible, but methodologically difficult.

  The Foundation for Accountability has spearheaded efforts to make outcomes measures more user-friendly. NCQA also includes a few outcomes measures in HEDIS®.

Where You Can Get Measures

In the past decade, sponsors, health care organizations, and researchers have put a great deal of effort into standardizing and evaluating the reliability of performance measures. As a result, it is now possible to turn to a handful of well-tested tools and measurement sets that you can use as the basis for collecting data and preparing comparative information for your audience.

However, the fact that many measures are available does not mean that the existing pool of quality measures will meet the needs of your particular audience. The truth is that many measures were not originally developed for consumers. Consequently, a number of sponsors and technical experts stress the importance of developing new measures or tweaking existing measures in order to provide the information that consumers believe they need to make good decisions about their health care. Depending on the needs of your audience, you may want to consider participating in efforts to test new, innovative measures designed for consumers.

At the same time, many consumers simply don’t understand how existing quality measures may be relevant to them. Sponsors need to include material in their quality reports that explains what measures mean and what they tell people about the quality of care they can expect from health care organizations.

Reliable sources of measures include the following:

• HEDIS®

  HEDIS® refers to the Health Plan Employer Data and Information Set, developed by the Washington, DC-based National Committee for Quality Assurance (NCQA). HEDIS® is a set of approximately 60 measures of health plan performance, including the results of the CAHPS® survey. This set has been evolving since 1991 as NCQA introduces new measures and "retires" less useful ones. The HEDIS® measurement set is explained in a set of documents available from NCQA.

  Ninety percent of health plans report their HEDIS® results to NCQA each year. NCQA analyzes those results and presents the scores for health plans that agree to release their results to the public on its database, which is called Quality Compass®.

  Select for a list of HEDIS® Measures Suitable for Reporting to Consumers.

• CAHPS®

  CAHPS®, or the Consumer Assessment of Health Plans, refers to a survey-based toolkit that enables sponsors to learn about their audiences’ experiences with the health care system. It was developed by a consortium of researchers with funding from the Agency for Healthcare Research and Quality (AHRQ), formerly Agency for Health Care Policy and Research (AHCPR). The CAHPS® survey consists of over 40 items that ask the respondent to rate their experiences with different aspects of care, including access, timeliness, communication, courtesy, and administrative ease. In addition to the customizable survey, the toolkit includes instructions for analyzing the responses and presenting the results. 

  Traditionally, sponsors generate this kind of information by surveying their own audiences (e.g., employees). However, it is now possible to get this information without doing your own survey because an expanded version of CAHPS® has been incorporated into HEDIS®. To get data on specific plans, sponsors may contact NCQA or request the data from those plans that are reporting HEDIS® results.

• FACCT

  The Oregon-based Foundation for Accountability (FACCT) has endorsed a set of quality measures that apply to a variety of care settings and approaches for financing and organizing care. These comprehensive measurement sets address adult asthma, alcohol misuse, breast cancer, diabetes, major depressive disorder, health risks for smoking, health status over age 65, health status under age 65, and consumer satisfaction. FACCT also has a set of measures that focus on children, including measures that address chronic conditions, early childhood development, and adolescent preventive care.

  FACCT has also created a survey tool called FACCT|ONE, which enables sponsors to collect data on patients’ experiences with care as well as the outcomes and processes of care. Unlike other more general surveys, FACCT|ONE focuses on quality of care for people living with illness, specifically patients who have any of three chronic conditions: asthma, diabetes, and coronary artery disease.

• CONQUEST

  In addition, the Agency for Healthcare Research and Quality (AHRQ) maintains a database called CONQUEST that allows users to identify, evaluate, and compare over 1,200 measures of clinical quality for health plans and provider organizations.  Although this compendium of measures and measurement sets (like HEDIS®) is primarily intended to facilitate efforts to assess and improve care, it is also available to sponsors that are seeking specific clinical measures for public reports.

Some Criteria for Choosing Measures

When you are choosing measures to include in a quality report for consumers, consider the following three questions:

• Is the measure designed to meet the information needs of consumers?
  Although there are thousands of different measures of quality, most of them are designed to meet the needs of health care organizations, which use detailed indicators to pinpoint and fix specific problems with the care they deliver. Such measures are usually too specific and clinical to be helpful to consumers. To be useful for consumers, quality measures must capture aspects of health care in a way that people without technical knowledge can understand and connect to their own experiences.
• Is the measure relevant to your intended audience and its needs for health care services and information?
  For instance, an audience composed of the parents of young children will have very different interests than an audience composed of Medicare beneficiaries. The younger cohort may be very concerned about prenatal care, obstetric outcomes, and childhood immunization rates and indifferent to measures of care for chronic diseases, while the elderly have the opposite reaction.
• Does the measure support the message you wish to convey to your audience?
  It is critical to avoid measures that could actually undermine the message. For example, let’s say you are hoping to communicate the ideas that quality matters and that quality varies. If you pick a quality indicator that has little relevance to your audience (like providing the elderly with data on childhood immunization rates) or does not actually vary much across plans or providers, your message loses credibility. On the other hand, a measure that reinforces your message can be very powerful; for instance, a display of health plan scores that all surpass a National benchmark lends strong support to the message that a purchaser only offers plans that meet high standards. To learn more about choosing a message, go to Agree on a Message.

More specifically, you can assess how well the measures you intend to use meet the following list, which is based primarily on criteria developed by RAND, a California-based nonprofit that helps improve policy and decisionmaking through research and analysis. 

• Standardization. The measures are standardized at the National level, which means that all organizations will be reporting the same kind of data in the same way.
• Comparability. If appropriate, the results are adjusted for external factors that could make an organization’s performance appear better or worse than it really is; such factors include age, education, gender, income, and health status.
• Availability. The measures will be available for the majority of health care organizations that you are profiling.
• Timeliness. The results will be available in time for you to produce and distribute a report when it is most needed by consumers.
• Relevance. The measures address the concerns of your audience.
• Validity. The measures have been adequately tested to ensure that they consistently and accurately reflect the performance of health care organizations.
• Experience. Health care organizations have experience with these measures, so that you can be confident that the measure reflects actual performance and not shortcomings in information systems.
• Stability. The measures are not scheduled to be "retired," e.g., removed from the HEDIS® data set to make room for better measures.
• Evaluability. The results can be evaluated as either better or worse than other results, in contrast to descriptive information that merely shows how organizations may be different from each other. For example, a complication rate is an evaluable measure because we know that a lower rate is always better; in contrast, a Caesarian-section rate is not evaluable because we don’t necessarily know whether a higher rate or a lower rate is desirable.
• Distinguishable. The measures reveal significant differences among plans.
• Credibility. The measures are either audited or do not require an audit.

To make your report card more palatable to consumers, it is helpful to include measures that are familiar to your audience as a result of repeated exposure. While this doesn’t really make the measures any more meaningful, experienced sponsors report that consumers find it reassuring to get information they have seen before, such as measures of satisfaction. Sponsors should also consider what others in the market have done or are currently doing. Since consumers may see multiple materials, it is preferable to provide information that is as consistent as possible (or at least to avoid being contradictory).

Research on This Topic

While there has been a lot of progress in understanding what measures are most helpful to consumers, the research is not yet conclusive. Some researchers have found that consumers prefer to hear assessments of quality from people like themselves, which would suggest that survey-based measures of experience with care are best. Other experts argue that consumers are most interested in the likely outcomes of the care, which is an area where we do not yet have many measures.

Identify Potential Partners for the Project

Select links below to learn more about:

• Where to Look for Possible Partners.
• Some Questions to Ask About Potential Partners.
• Examples of Successful Collaborations.

Where to Look for Possible Partners

Some partnerships seem obvious because the participants clearly share common concerns—the three US auto manufacturers, for example, contract with many of the same plans, have employees in many of the same States, already participate in similar quality measurement and reporting activities, and negotiate with the same union, which plays a large role in shaping health care benefits.

As this example suggests, you can approach the process of forming partnerships in two steps:

1. Focus on those organizations with which you have something in common.

Consider factors such as:

• Purchasing strategy—In particular, compare your level of commitment to the consumer choice model. For example, a purchaser that offers employees a choice of several different kinds of health plans would not want to partner with one that selects a single health plan for all of its employees.
• Location.
• Size—in terms of numbers of employees and retirees.
• Philosophical compatibility—For example, if you have been issuing detailed RFPs to health plans asking about quality improvement programs and auditing their quality measures, you would probably not want to partner with a purchaser that simply asks for rates and self-reported member satisfaction rates.

2. Look beyond the usual crowd.

Many private sector companies never even think to talk with purchasers in the public sector and vice versa. But those who do have been surprised by how much they have in common and how much they can learn from each other.

Public sponsors include more than the agencies responsible for State employees and Medicaid: various organizations purchase health care benefits on behalf of school, university, and local government employees as well as for the beneficiaries of State insurance programs for children and others. A public purchaser can lend clout to a project due to its size and the strength of its contractual relationships. For instance, some States designate a single agency to contract for many, if not all, health care services for governmental organizations, including counties and schools; as a result, they can be a major purchaser in the State or in specific counties, often surpassing the biggest private employers. In addition, public entities—especially regulatory agencies—imply the promise (or threat) of a political mandate, which can compel the cooperation of health plans or provider organizations.

For their part, employers and other private-sector sponsors of quality measurement projects often offer several years of experience with quality measurement and reporting as well as the flexibility to move quickly to make things happen. Many large employers and business coalitions have been actively engaged in such projects in various markets around the country since the early 1990s; while they may not have all the answers, they have a good sense of the right questions and the best paths to pursue.

Examples of Successful Collaborations

A coordinated approach to quality measurement and reporting has become a key strategy for several innovative health care purchasers around the country. By their very nature, business and other community coalitions offer a vehicle for multiple organizations to come together to develop and distribute materials on quality. Dozens of coalitions have taken on this role for their members.

Outstanding examples of the cooperative approach include the following two:

• In California, two major statewide sponsors work together to spearhead quality measurement projects at both the health plan and provider levels. These organizations—the Pacific Business Group on Health (a coalition of large West Coast purchasers) and CalPERS (a California State agency responsible for providing health care coverage and other benefits to State and local government employees and retirees)—have coordinated their efforts for several years so that consumers and health care organizations receive a consistent message about quality. 
• In Michigan, the CARS (Coordinated Autos/UAW Reporting System) project brings together the three major auto manufacturers, the United Auto Workers (UAW), and the Greater Detroit Area Health Council (a broad-based coalition of business, labor, hospitals, health plans, and others organizations in Southeast Michigan). The State of Michigan is also a participant in this project. While each organization puts out its own information about health care benefits, this project allows them to streamline the process of data collection and analysis and to minimize confusion in the marketplace.

Agree on a Message

The messages you choose form the foundation for the contents of your report. When you know what messages you want to convey, you can also start to think about what information your audience needs in order to understand and apply the message to their health care-related decisions. What kinds of background or explanation will help your audience digest the information you are providing? Taking the time to get to know your audience and its information needs will help you figure out what to say in your performance report.

To learn more about the messages you’ll want to include in your report, go to What You Have to Explain.

To learn about FACCT’s Consumer Information Framework, which discusses some of the messages you could share with consumers, go to Supporting Quality-Based Decisions at http://www.facct.org. 

Find Sources of Financing and Establish a Budget

A quality information project is not cheap, although there are ways to cut corners and spread the costs across multiple parties (including other sponsors and participating health plans or providers). Costs can range from as little as a few thousand dollars for a brief, simple report based on easily available information to several hundred thousand dollars for a comprehensive report that requires a large number of observations and labor-intensive data collection and analysis.

This section briefly discusses:

• Where to Look for Money.
• What to Include in the Budget.

Where to Look for Money

Sharing Costs with Health Plans and Providers

The most common strategy is to ask the health plans or providers to pay for parts of the project on the theory that they benefit from seeing comparisons of their own performance to that of their competitors. However, competitive forces and multiple demands in the health care industry are making plans and especially providers less willing to absorb the costs involved in providing detailed data. Some are even refusing to participate or at least limiting their involvement.

One reason for their lack of enthusiasm is that many health care organizations think no one is using the data. To win back their support, you may need to conduct an evaluation of the project that demonstrates progress with the information’s intended audience. Should you decide to use your evaluation for this purpose, be sure to manage the expectations of your health care partners. Both you and they must be realistic about the impact any project of this kind can have in the short term. For instance, while you may aim for shifts in market share in the long-term, reasonable short-term goals might include an awareness that the information is available and a basic understanding of the message that quality varies.

Sharing Costs with Other Sponsors

Perhaps the most effective way to limit your financial exposure is to undertake the project with one or more partners. By tackling the project together, partnering sponsors can reduce their costs significantly. For example, they can conduct one survey of consumers rather than two (or more). Or they can produce just one performance report that meets the needs of all their audiences. Coordination also allows sponsors with limited resources to benefit from the deeper pockets of their partners. In Detroit, for example, eight organizations are sponsoring a project to measure and report on the quality of care delivered in area hospitals. The partners evenly split the costs of gathering and analyzing data and each bears its own cost for distributing the report. However, the three large auto manufacturers contribute significant in-kind services, such as legal review, project management and administration, purchasing, and media relations.

What to Include in the Budget

Agree on a Schedule (and Stick to It)

Sponsors of quality measurement projects commonly experience one of the two following problems:

• They never get beyond the planning stage—especially if they are collaborating with multiple partners that insist on questioning every decision.
• They do move into the implementation phase, but by the time they have something to report, the information is old and therefore useless.

Establish a Management Structure