Making Quality Measures Manageable

When you look at a typical health care quality report card, the first thing you notice is how much information it contains. It is hard to know how to absorb all the data, let alone interpret it. Which measures should the reader focus on? How do these measures relate to each other, if at all? In essence, how does the reader manage all the information on the page?

To help sponsors think about concise ways to present an abundance of data, this section discusses:

• What Research Tells Us about giving people too much information.
• A Framework for Organizing and Condensing Information.
• Using Categories to Group Measures.
• Combining Measures Into Summary Scores.
• Offering Information in Layers.

What Research Tells Us

For most people, the task of taking in and processing a large number of variables is very difficult. Researchers have found that most of us can only hold five to nine items of information in short-term memory, which limits how much we can incorporate into our decisions. Considering that most health care consumers are already including price and physician availability in the equation, that doesn't leave much room for more than a handful of measures representing quality. 

Researchers have also discovered that while people usually say they want more information than they already have, they do not actually make better decisions when they have more factors to consider.  In one study, consumers' decisions did not improve when they were given more than five variables.

Too much data can lead to poor decisionmaking.

When people have too many items in front of them, they tend to handle the information in a way that may lead them to the wrong conclusions. For instance, researchers noted that consumers will focus on one measure they understand to the exclusion of everything else. This process is similar to the voter who picks a candidate based on his or her position on only one issue.

People value and use information they can easily understand and interpret.

While this may not be a conscious process, consumers place a higher weight on concrete information—i.e., things that are familiar or easy to understand—than on information that seems vague or complicated, even if they say they would do otherwise. For example, people may say that they want to use information on the outcomes of care, but if that information is too hard to interpret, they will base their choices on information they can understand, such as data on overall satisfaction.

Similarly, researchers have found that consumers are more likely to act on information that is presented in a way that helps them draw conclusions about what they are seeing. This suggests that sponsors must provide a point of comparison that puts health care quality data in context for consumers, letting them judge what's good and what's bad. If you don't do this, readers tend to focus on whatever information they can evaluate, regardless of how (relatively) unimportant it may be. 

People are better able to handle information that has been organized in a framework that places general information at the top and more specific information at the bottom. Researchers have found that this kind of hierarchical structure make it easier for consumers to understand and store information in their memories and retrieve it when they need it. In addition, hierarchies help consumers make decisions about the importance of the information, since people tend to assign more importance to the information at the top of the hierarchy. 

For a list of articles on these topics, select What Research Tells Us: Relevant Studies

A Framework for Organizing and Condensing Information

There are a number of ways to organize and condense information on quality. You can implement just one strategy or combine two or more by providing information in layers.

• Strategy 1: Report Scores for Individual Measures Only.
• Strategy 2: Group Measures Into Categories.
• 2a: Report scores for each measure in a category.
• 2b: Combine measurement scores into summary scores for each category.
• Strategy 3: Report a Single Summary Score that represents overall performance.
• Which Strategy is Right For You?

This decision will depend on the nature of the information you are reporting, the information needs of your audience, and the medium you use to deliver the information. For specifics, go to Choosing Media.

For a diagram that shows different ways to report the same eight individual measures, select Levels of Aggregation (51 KB) or an alternative text description.

Strategy 1: Report Scores for Individual Measures Only

One common way to report quality measures is to show each one with its score, in no particular order. This approach is simple and straightforward, which is a plus. However, since it is not practical to collect or report everything, you have to choose a limited set of measures—ideally ones that are both relevant to your audience and meaningful indicators of quality. This makes it harder to present a comprehensive picture of quality.

For more details on this topic, go to Choose Quality Measures.

To determine how many and which measures to report, you can:

• Look at the research on different measures; some are thought to be more reliable indicators of overall quality than others.
• Learn from the experiences of other sponsors, especially those in your own market.
• Test measures with your audience to identify which measures are most salient to them.
• Consider the medium you'll be using. You can include many more measures in a Web-based report than in a paper report.

Strategy 2: Group Measures Into Categories

Because categories place measures into a manageable number of groups, they offer a way to present a great deal of information, and therefore a broad picture of quality, without overloading your audience. You can use categories to organize information as well as to condense it.

Report scores for measures in each category

You can impose some order on quality measures by grouping them into related categories, but still providing the details of individual measures and scores within that category. This helps consumers focus on the areas that are most relevant to them.

Combine measurement scores into summary scores for each category

To further condense the information you are providing, you can report summary scores for each category, with or without showing the individual measures that make up the category. Experienced sponsors have found that you can maximize the flexibility and usefulness of your report by layering the information, which means that you provide both summary scores and detailed results for individual measures. Readers can then focus on whatever is of greatest interest to them.

Strategy 3: Report a Single Summary Score

Finally, for the ultimate in aggregation, you could provide one score to reflect all elements of performance. A single score is easier for consumers to use to compare their options. However, the process of combining multiple scores into one score often obscures differences across health care organizations, making them all seem fairly similar in performance. It also makes it harder for consumers to see any specific differences that may have been important to them. Finally, a summary score tends to be less concrete than scores for individual measures.

One good example of this approach is the NCQA's accreditation scoring system, which collapses a comprehensive evaluation of a health plan's structure and capabilities into a final decision about accreditation, with layering used to show performance in various categories.

Which Strategy Is Right for You?

The primary reason to group measures into categories is to provide a lot of information in a way that doesn't overload your audience. But while you may agree with the goal, this approach may not be appropriate for you. Consider your answers to the following questions:

• How homogeneous is your audience? If your consumers are fairly similar in age, education, income, and other characteristics, it may not be too difficult to select a handful of individual measures that most of them would find relevant. In this (unusual) case, there is really no need to group or combine measures.
• How prescriptive do you want to be? The more you aggregate measures, the more you have to make decisions about how to weight the results and which aspects of quality to value over others. For employers with a long history of making decisions on behalf of their employees, this may be fine; for others, it could be completely unacceptable.
• How do you feel about blurring differences in quality? Higher levels of aggregation make it harder to detect variations in results at the level of individual quality measures. When you consolidate measures into categories, you are obscuring differences that your audience would otherwise see. For example, a major difference in immunization scores will not be as noticeable when those scores are rolled up with scores on five other measures. That said, some experts would argue that consumers should not be basing decisions on single measures of performance, so aggregation helps to encourage them to consider multiple dimensions of quality.

To learn more about making quality measures more manageable for consumers, you can read about:

• Using Categories to Group Measures.
• Combining Measures to Create Summary Scores.
• Offering Information in Layers.

Using Categories to Group Measures

The purpose of categories (or composites, in the context of CAHPS®) is to group multiple measures of quality into a small number of topics, which makes it easier for consumers to digest the information that is presented and focus on the areas they find most relevant. Ideally, these topics should reflect how consumers think about health care and the ways in which they naturally identify themselves (e.g., as healthy versus sick, young versus old).

Categories introduce an element of standardization to the presentation of quality information. Over the past five to ten years, the standardization of measures has been a major focus for both researchers and sponsors. This was important because people were trying to compare quality measures that did not include the same inputs or were not calculated in the same way. Now that standardized measurement tools are available, sponsors and researchers are turning their attention to the huge variation in reporting practices. You only have to look at a few report cards to see how differently they can organize and present information.

This section discusses:

• Benefits of Categories.
• Drawbacks of Categories.
• Kinds of Categories.
• Constructing Categories.
• How to Pick Categories.

Benefits of Categories

Categories represent a movement towards a common and consumer-friendly framework for organizing information. At their best, they offer a way to communicate a large amount of data through just a few concise pieces of useful information. Other benefits of categories include the following:

• Creating an overall picture. Categories create a picture for consumers of the relative strengths and weaknesses of competing organizations without (necessarily) emphasizing one specific piece of information over another.
• Getting everybody on the same page. As more sponsors use categories, it will become easier for people to talk about quality using the same language, which will facilitate dialogue among consumers, purchasers, health plans, and providers.
• Making new information manageable. Categories provide the "buckets" that can hold many different pieces of quality information as they becomes available; without categories, it may become impossible to manage all the measures that may someday be reported.
• Driving the development of new measures. The development of categories reveals areas where measures are missing, such as indicators of quality of care at the end of life. By pointing to gaps in knowledge, categories encourage the development and testing of worthwhile measures.

Drawbacks of Categories

Although they have great potential to simplify quality reporting, categories also have several weaknesses:

They obscure information that individual consumers may want to know.

The strength of categories—that they can reflect what consumers want to know about health care quality—may also be their biggest downside. Because categories are designed to represent the preferences and concerns of the community, individuals may not be able to focus on their own particular needs. For instance, if you group quality measures into categories without providing the scores for each individual measure, consumers will not be able to organize and value measures in whatever way makes the most sense to them.

They can create an inaccurate perception of comprehensiveness.

The name of a category may imply a more general picture of quality than is justified by the measures currently included in that category. For instance, a category called "Staying Healthy" is usually composed of a handful of screening and preventive care measures, primarily centered around the health care needs of women and children. But the name suggests aspects of disease prevention and health maintenance that apply to men and women of every age.

The decision to use categories means that you are imposing a specific system for grouping and possibly combining measures on your audience. If you are not comfortable with taking on that responsibility, it may make more sense for you to provide individual measures. Also, because no categories are well established at this time, you may need to justify your framework to various stakeholders, including health plans and provider groups.

While these are important considerations, keep in mind that the use of quality measurement categories is very new and there is plenty of room for sponsors to experiment. By testing categories with your audience, you can help determine whether what we see today resembles the categories in use five or ten years from now.

Kinds of Categories

CAHPS® reporting template (PDF file, 46 KB; HTML)

The second kind of category pulls together measures that are related but capture different aspects of quality and come from multiple sources. These categories may focus on specific concerns (e.g., living with illness), conditions (e.g., breast cancer), or populations (e.g., children, elderly). For instance, a category called "living with illness" could include health status indicators, a few disease-specific clinical measures, hospital readmission rates, items that reflect satisfaction with physician care, and global ratings of health care for those with chronic illnesses. However, while you can easily list different kinds of measures under the same heading, it is fairly complicated to equilibrate these measures so they can be combined into one score.  

Constructing Categories: Top-Down Versus Bottom-Up

Once you have chosen an organizing principle, there are two ways to go about constructing the categories:

• Option 1: Start with the categories you think you want.
• Option 2: Start with the measures you know you have.

Option 1: Start with the categories you think you want.

This first choice is the "top-down" approach, which means that you start by figuring out what categories would be most relevant to your audience and then try to fill in the categories with available measures. To decide which measures belong in which categories, you could call upon experts to give their opinions, ask consumers to judge what belongs where, or rely on an analytic assignment, which would look at statistical relationships between measures and categories.

The advantage of the top-down strategy is that you are likely to end up with categories that are meaningful to people (assuming you can find appropriate measures for each category). It also offers consistency over time and consistency in the market because sponsors use the same categories year after year, filling them in with new measures as they become available.

The downside is that adequate measures are not available for every category. For instance, while consumers may be interested in quality of care at the end of life, very few measures address that concern. Also, both consumers and experts may disagree on which measures belong in which categories. For instance, the HEDIS® measures related to diabetic retinal exams could reasonably be assigned to a category called "Staying Healthy" because the exams are meant to screen for eye problems; at the same time, these problems are complications of a chronic disease, so the measure could also be part of a category called "Getting Better." Finally, since categories can incorporate new measures each year, summary scores are not necessarily comparable to those of previous years.

For more information, go to Combining Measures Into Summary Scores

Option 2: Start with the measures you know you have.

The "bottom-up" approach means that you are creating categories based on the ways in which measures naturally fit together. While this approach may be more practical, it is not necessarily consumer-friendly. For instance, in the past, some report cards grouped measures according to their data source (but with no regard to whether they addressed the same topics): measures from survey data were displayed together, as were measures based on health plan records. While convenient for the sponsor, this organizing principle had little meaning for consumers.

More recently, sponsors have put some effort into combining the measures they have into categories that make sense to consumers. This approach allows sponsors to use the data that are available to give consumers useful information. However, it doesn't necessarily result in categories that tell consumers everything they want to know. One sponsor, for example, noted that it didn't have enough information (specifically data on quality of care for chronic illnesses) to create a category that it knew would be important to consumers.

For specifics on a framework that exemplifies composite reporting, select CAHPS®: Consumer Assessment of Health Plans.

How to Pick Categories

Select links below to learn more about your options:

• Adopt an Existing Model.
• Modify Existing Categoriess.
• Do Your Own Thing.

Adopt an Existing Model

The easiest alternative is to use a model that has already been evaluated with consumers. Several organizations have created and tested frameworks for presenting quality measures:

• CAHPS® (Consumer Assessment of Health Plans).
• Foundation for Accountability.
• National Committee for Quality Assurance.
• CARS (Coordinated Autos/UAW Reporting System).
• U.S. News and World Report.
• Consumer Reports.
• Pacific Business Group on Health.
• Medicare Quality Information Project (funded by The Commonwealth Fund).

Modify Existing Categories

Reporting models are a recent development, which means that no frameworks for grouping measures are set in stone. Even if you start with an existing model, you may want or need to modify the categories depending on which measures are available to you and what you want to emphasize.

For instance, the NCQA incorporated accreditation information into FACCT's categories. Similarly, the sponsors of the CARS project discarded the NCQA's term "Qualified Providers" in their first year because they feared that consumers might interpret a low score to mean that a plan's providers did not meet minimum standards. Instead, in that first report in 1998, they stayed with a category called Consumer Satisfaction simply because their audiences were accustomed to seeing that term in previous report cards. In 1999, they changed that category to one that captured consumers' experiences with physicians: "Doctor Communication and Service." These adaptations are important because the experiences of sponsors with different categories will influence the direction in which the frameworks evolve.

Do Your Own Thing

A sponsor may be better off devising its own categories if either of the follow circumstances apply:

• The sponsor's audience is truly different from the general population (in terms of age, education, income, or other factors that affect the need for health care services and information).
• The measures available to the sponsor are different from the measures typically used for the categories (for example, if the sponsor is testing a new measurement set or a new survey).

If you must create your own model, be sure to design the categories so that they are all equally important. That is, don't mix relatively narrow or trivial matters with big, serious issues. The problem is that people tend to weight categories equally when they use the information to make decisions – even when they say that they value one dimension of quality more than another. If your categories are not equivalent in importance, consumers may be overly influenced by information that has limited value, or may discount the information that's most pertinent for them.

The Downside: While this approach lets you customize a report to suit a specific audience, it places a big burden on sponsors to develop appropriate categories and figure out how to fill them with measures in a way that is analytically and statistically justifiable. Most existing categories are already supported by research with consumers. It also thwarts one of the purposes of having categories, which is to impose some standardization on reporting practices.

Combining Measures Into Summary Scores

Reasons to Offer Summary Scores

There are two good reasons why you might want to include a summary score in a quality report.

• It is difficult for consumers to process all the information contained in report cards; there are simply too many variables for them to think about—and that's ignoring the many other factors that may enter into their decisions, like premium costs or provider networks. A summary score helps to minimize how many pieces of data the consumer sees.
• A summary score also allows the sponsor to influence what consumers pay attention to when they consider the performance results. You can exert this influence by deciding whether and how to weight the measures in each category when calculating the score.

Reasons to Avoid Summary Scores

Summary scores have their drawbacks as well. Problems with summary scores include the following:

• When scores are presented without supporting detail, consumers lose the ability to identify the measures that are most important to them. As a result, they are not able to make decisions based on personal preferences. For example, a summary score for the category "Living with Illness" would not allow an individual with a specific illness (such as diabetes) to distinguish which health plan offers the best care for that condition. Similarly, a score that combines different measures of access would be of little use to someone concerned about a specific measure, such as access to specialists.
• Since categories are not as specific as individual measures are, there's a risk that consumers could misunderstand or misinterpret the meaning of the summary score. They may also confer a level of certainty and "concreteness" to the summary score that is not appropriate given the uncertainty inherent in quality measures.
• It is not clear how sponsors should combine data to come up with summary scores. How do you calibrate different kinds of measures (such as satisfaction measures and preventive care rates) so that they can be rolled up into one score? Although it can be done, the methodological issues pose a significant challenge and are controversial.
• Even when summary scores are based on similar measures, it is hard to know whether and how to weight each measure. That is, should some measures in the category have a greater impact on the summary score than others do? If so, how much more?
• Summary scores can obscure differences in results at the level of individual measures, which means that consumers will not be aware of the variability across organizations. For example, let's say that the childhood immunization rate is one of eight measures in the category called "Staying Healthy," and both Plan A and Plan B received three stars based on their summary score for that category. Plan A's immunization rate could be much higher than that of its competitor, but if Plan B excels in other areas, their overall scores may be the same despite notable differences in specific areas of performance.
• Finally, because summary scores can hide variations in individual measures, it is hard for plans to demonstrate improvement at the level of a summary score. Plan B may improve its immunization rate from 60 percent to 85 percent, but even such a sizeable change in this single measure may not register in a summary score that is based on seven other measures as well.

What You Need to Know About Combining Measures

As a result, the development of summary scores is not a job for the typical sponsor, unless you happen to have staff with the requisite statistical skills. Most sponsors that want summary scores hire a consultant to develop an appropriate methodology and conduct the analysis. However, sponsors should note that this first step is not necessary for CAHPS® items because the CAHPS® 2.0 Kit spells out the methodology for calculating valid composite scores and provides software programs to do it.

But while you do not need to know the nuts and bolts of calculating a summary score, you do need to understand how methodological decisions—i.e., the approach used to combine measures—can affect the results you present. Assuming you do not possess the expertise needed to generate these scores, your chief task at this stage is to make sure that the vendor or consultant you hire makes choices that are defensible and consistent with the messages you want to send to your audience and to the health care market. 

• How will they handle missing data? For a number of reasons, most of which are related to the inadequacy of information systems, some health care organizations will not be able to provide the data you need to calculate a summary score. There are several ways to handle this problem, each of which sends different messages to the organization and to consumers.
• Will they standardize scales? Quality measures may be based on any number of different scales; many HEDIS® measures, for example, are on a 100-point scale, while survey items may use a three or four-point scale. To combine these measures so that they each have an equal influence on the summary score, it is necessary to make the scales equivalent in some way. Standardization of scales refers to the process of compensating for differences in the ways in which scales are constructed.
• Will they weight measures? If so, how? Weighting is a way of assigning different levels of importance or value to measures in a given category. Unlike standardization of scales, which is primarily a technique for making different kinds of measures equivalent, weighting is purely a policy decision that should be driven by the goals of the sponsors.

Offering Information in Layers

Select links below to read about:

• Reasons to Layer Quality Information.
• Reasons to Avoid Layering.
• Two Ways to Make Layering Work.

Reasons to Layer Quality Information

With layering, the consumer can get an overview of quality, then zero in on the issues that are most personally important. This allows consumers to choose the product (or health care organization) that is best for them, which may not necessarily be the product (or organization) that is best overall.

From the sponsor's perspective, layering also improves the chances of reaching a broader audience with the information. People have varying levels of interest in quality information: some want a quick, easily digestible assessment of their options, while others want to consider every detail. By layering the information, sponsors can meet the needs of each kind of consumer without losing the attention of the others.

New Jersey Managed Health Care Plans: Compare Your Choices (PDF file, 176 KB; HTML).
© Copyright 1998. New Jersey Department of Health and Senior Services. All Rights Reserved. Used with Permission.

The CAHPS® 2.0 reporting template suggests a way to present composite scores as well as information on individual items (PDF file, 46 KB; HTML).

Reasons to Avoid Layering

While the idea of providing information in layers has a lot of appeal, it can be hard to implement.

Layering is not appropriate for everyone.

Before offering layers of information in a report, sponsors need to consider whether this approach is best for their audience. For some consumers, the details of quality measures may be too complex or intimidating. Just seeing so much information in one place may be enough to stop them from looking at it.

Layering is difficult to do in a printed product.

You can display many layers of information on paper, but it is hard to show the reader how those different layers are connected. It also takes many pages to provide various layers of detail; few consumers are willing to wade through so much information to find what they are looking for. Electronic media like the Internet are better suited for displaying information in layers, although even here the information may be overwhelming unless the linkages are clean and well-constructed.

Two Ways to Make Layering Work

To make layers of information easy for your audience to understand, consider the following strategies:

• Use Electronic Media.
• Offer Multiple Versions of the Same Information.

Use Electronic Media

Using electronic media like CD-ROMs and the Web, the reader can "drill down" to the detail they want to see more easily than they can on paper. On a computer, the sequence of information can be obvious because the user is choosing which detail to see and when. For example, users can choose to see the rates that were the basis for the score. They could even select those rates to see trend data. Unlike on paper, the reader does not have to sort through a great deal of information in one view.

The downside of this approach is that it doesn't help consumers without access to computers or those who are not familiar or comfortable with them.

	Buyers Health Care Action Group's ChoicePlus site (www.choiceplus.com)
	The Maryland Health Care Commission's Web Site for Comparing the Quality Of Maryland HMOs: A Guide for Consumers.
	The NCQA's Health Plan Report Card.
	The Pacific Business Group on Health's Healthscope.org. © Copyright 2001. The Pacific Business Group on Health. All Rights Reserved. Used with Permission.

Offer Multiple Versions of the Same Information

Even in a population that has many common characteristics and information needs (such as Medicaid recipients), people will differ in their level of interest in information on quality and in their ability to use that information. Rather than create one document for everyone, sponsors could create a coordinated set of materials so that people could select the level of information they want. For example, one document would provide summary-level data only, while another reveals the detail that supports the summary-level results. This is similar to the approach taken by sponsors that have used the same data to develop summary information for consumers as well as detailed reports for employers and health care organizations.

Of course, the production of two or more versions compounds the expense of reporting quality information; it also creates a number of logistical challenges related to distribution.