Future Directions for the National Healthcare Quality and Disparities Reports

You Are Here: AHRQ Home > Quality & Patient Safety > Measuring Healthcare Quality > Future Directions for the National Healthcare Quality and Disparities Reports > Appendix F

Appendix F. The Expected Population Value of Quality Indicator Reporting (EPV-QIR): A Framework for Prioritizing Healthcare Performance Measurement

David O. Meltzer, MD, PhD, and Jeanette W. Chung, PhD
The University of Chicago

I. Introduction

In The Opportunity Costs of Haphazard Social Investments in Life-Saving, Tengs and Graham (1996) studied the costs and benefits of 185 interventions that reduce the risk of premature mortality to evaluate the allocative efficiency of investment in life-saving opportunities in the United States. According to their estimates, the United States spent approximately $21 billion on life-saving interventions that prevented roughly 56,700 premature deaths. However, reallocating those dollars using cost-effectiveness criteria to maximize the number of lives saved could have avoided an additional 60, 200 premature deaths.

Tengs and Graham's analysis provides a cautionary tale for stakeholders in healthcare quality improvement, patient safety, and disparities. There are currently more than 1,400 measures in the U.S. Department of Health and Human Services (HHS) National Quality Measures Clearinghouse (NQMC) and more than 250 measures in the Agency for Healthcare Research and Quality (AHRQ) National Healthcare Quality and Disparities Reports (NHQR and NHDR). Given limited resources and an ever-proliferating set of healthcare measures, Tengs and Graham's analysis reminds us of the importance of asking whether we are maximizing the returns on our investments that seek to improve healthcare quality and safety.

This paper proposes a conceptual and methodological approach to quantifying the population value of efforts to improve quality and reduce disparities, specifically through the selection of quality and disparities indicators such as the AHRQ National Healthcare Quality and Disparities Reports that are the subject of this IOM Committee. To do so, the paper draws upon the literature using measurement approaches from medical cost-effectiveness analysis to prospectively assess the value of research (Claxton and Posnett, 1996; Fenwick et al., 2008; Meltzer, 2001). The result is an approach to estimate the expected population value of quality indicator reporting (EPV-QIR). Although analytic tools of cost-effectiveness analysis are used, our approach recognizes that "identifying and issuing guidance regarding the use of cost-effective health technologies does not, in itself, lead to cost-effective services provision" (Fenwick et al., 2008). This gap between evidence on the potential cost-effectiveness of an intervention and the cost-effectiveness of its implementation in practice can arise for many reasons. One reason is uncertainty about the costs and benefits of an intervention. In such cases, modeling the expected value of research has led to useful applications in prioritizing research agendas in domains including Alzheimer's disease treatments (Claxton et al., 2001), antipsychotic drugs in schizophrenia (Meltzer et al., 2009), bronchodilators in chronic obstructive pulmonary disease (Oostenbrink et al., 2008), and anti-platelet medications in cardiac care (Rogowski et al., 2009). However, while uncertainty in the effectiveness of interventions is relevant in addressing quality and disparities, quality and disparities reporting is more often targeted at variability in the implementation of available information. Recently, value of research approaches have been adapted to address issues of imperfect implementation (Fenwick et al., 2008; Hoomans et al., 2009).

The expected population value of quality indicator reporting (EPV-QIR) we propose is intended to be a useful tool in selecting quality indicators that can produce the largest improvements in population health. Quality indicators can be ranked in terms of their EPV-QIR and a set of indicators can be identified that offer the highest expected returns to investing in quality improvement. The EPV-QIR depends on several factors:

The net health benefit of the appropriate implementation of the intervention, which is the magnitude of the potential health benefit of the intervention (measured in quality adjusted life years (QALYs)) net of the opportunity costs in health when the intervention is fully implemented to maximize its benefit net of costs.
The size of the population of persons who should receive the intervention given the standard of care, e.g., those with a positive net health benefit from the intervention.
The current state of implementation, which potentially includes both the rate of utilization among parts of the population with positive net health benefits and the rate of use among those parts of the population with negative net health benefits (for whom there are potential gains in net health benefits that can be obtained by eliminating inappropriate use in that population).
The potential for quality improvement, especially as produced by reporting quality indicators. This depends on the probability that providers (or patients) will make choices likely to improve quality when given information on provider performance is provided, and the effectiveness of existing quality improvement interventions to improve outcomes. Because data on these effects may be especially incomplete, our approach also specifically highlights uncertainty in the extent to which quality reporting will stimulate quality improvement action, and quality improvement action will change implementation. This includes both estimating the expected (average) effects of reporting on quality, and bounding estimates of these effects when data on the effectiveness of reporting on quality is especially incomplete. For example, if an intervention is not currently used or at least not used in persons in whom it produces net harms, one such bound would the value of perfect implementation, which is the total benefit that can be achieved in a population if everyone who should receive an intervention receives it and everyone who should not receive an intervention does not receive it.

We explicate our framework in detail in the remainder of this paper, and demonstrate its application in calculating the expected value of quality improvement for selected NHQR measures. We develop our framework in Section II, progressively developing concepts that are critical to the EPV-QIR framework. In Section III, we demonstrate the EPV-QIR calculations for selected measures in the NHQR, while also paying close attention to opportunities to bound estimates of EPV-QIR with more limited data. In Section IV, we discuss the scope of potential application for the EPV-QIR method and its limitations and implementation issues. Section V concludes with a discussion of areas for future development.

II. The EPV-QIR Framework

Our framework begins with the assumption that all measures are based explicitly or implicitly on some standard of care, which we denote by S. We use O to denote all other alternatives, which could include some other standard of care, or "usual care" or "doing nothing." Our model could easily be generalized to include multiple alternative standards of care (O_i) by indexing groups additionally according to the care they receive currently. For simplicity, however, we develop our theoretical framework in the case in which there is only a single alternative current pattern of care.

Given this single current pattern of care, the incremental benefit of S is the difference between the effectiveness of the standard of care (e_S) and the effectiveness of the alternative (e_O) current pattern. The incremental benefit of S can be written as Δe = e_S – e_O. The incremental cost of S is the difference between the cost of the standard of care (c_S) and the cost of the alternative (c_o). The incremental cost of S can be written as Δc = c_S – c_o.

Net Health Benefit (NHB). The net health benefit of the standard of care (NHB_S) relative to O, is the incremental health benefits of the standard of care net of its incremental costs, where costs are denominated in units of foregone health benefits due to the financial costs of the standard of care (Stinnett and Mullahy, 1998):

Equation 1

NHB_S = Δe - Δe - Δc / λ

In Equation 1, λ is a society's threshold willingness-to-pay for an additional unit of health benefit, which might be measured in life years or quality-adjusted life years (QALYs).¹ In these cases, λ would be the amount of money that society is willing to pay to save an additional life year or quality-adjusted life year. The term Δclλ is in units of health benefits and represents the foregone health benefits that could have been obtained by allocating money to some marginally cost-effective standard of care. In other words, Δclλ represents the opportunity costs in terms of health of accomplishing the standard of care. When an intervention is cost-effective, so its incremental cost-effectiveness ratio (ICER) <λ, the NHB will be positive. Conversely, the NHB will be negative when an intervention is not cost-effective, because the opportunity cost of the intervention will exceed its health benefits.

Because the NHB depends on how opportunity costs are valued in terms of health, NHB depends on the level used for λ. Thresholds of $50,000 and $100,000 per QALY have been commonly used in cost-effectiveness studies, but no universally accepted reference value for λ exists (Hirth et al., 2000). More recent literature has scrutinized the validity of these traditional threshold values and general failure to adjust the threshold for inflation (Ubel et al., 2003). Studies have suggested threshold values of: $109,000-$297,000 USD₂₀₀₃ per QALY (Braithwaite et al., 2008); $12,500-$32, 200 USD₂₀₀₃ per QALY (King et al., 2005); $24,777-$428,286 USD₁₉₉₇ per QALY (Hirth et al., 2000). Because the net health benefit framework is sensitive to the value used for λ, the NHB is traditionally reported over a broad range of values of λ.

Population Value of Perfect Implementation (PVPI). A standard of care should generally be implemented when its expected benefits exceed its expected risks. We define the number of individuals, N_S, in a population who should receive the standard of care as the measure population. Assuming that individuals outside the measure population do not receive the care, perfect implementation occurs when all individuals in the measure population receive the standard of care. The population value of perfect implementation (PVPI) is the total NHB achieved in the measure population when the standard of care is applied to every patient in the measure population. PVPI is calculated by multiplying the total number of individuals in the measure population (N_S) by the net health benefit of S(NHB_S):

Equation 2

PVPI_S = N_S X NHB_S

Population Value of Current Implementation (PVCI). Under perfect implementation, all individuals in a measure population receive the standard of care. When a standard is "underused," the rate of current implementation, r_SC, is less than 100%. The population value of current implementation (PVCI) is the total net health benefits achieved from the health intervention given current implementation rates:

Equation 3

PVCI_S = N_S × r_SC X NHB_S

When performance is perfect, every eligible individual in the population receives the standard of care, so PVPI = PVCI, and no further net health benefits can be gained from improving performance.

Maximum Population Value of Quality Improvement (MaxPVQI). Quality effort improvements can be thought of as interventions to perfect implementation. The maximum population value of quality improvement (MaxPVQI) is the total net health benefits that can be gained by improving implementation from current rates to 100%. Max-PVQI is simply the difference between PVPI and PVCI, or

Equation 4

MaxPVQI_S = PVPI_S – PVCI_S = N_S X (1 – r_SC) X NHB_S

This MaxPVQI defines the maximum population net gain in health from adopting some standard of care relative to the absence of that standard, in essence providing the net health benefits of the intervention to the fraction (1 – r_SC) of the population who should receive the intervention who are not currently receiving it.

Inappropriate Use and Overuse. As noted above, these same general equations can be used to estimate the value of quality improvement when there are multiple other patterns of care, as in the case in which an intervention is overused or inappropriately used, for example. The adjustments that are needed in such cases are to define the relevant population in terms of their current (inappropriate) treatment and then to measure the net health benefit of the change to the current standard of care relative to that inappropriate care. The net health benefit of S implemented within the measure population to which S is meant to apply will not be the same as the net health benefit of implanting S in another population. Hence, calculating the EPV-QIR of measures of overuse or inappropriate use will require estimates of the costs and health effects of implementing the standard in patients outside the measure population. Because the focus of the AHRQ quality indictors is on increasing appropriate use, we do not focus on overuse in our primary exposition, but we do discuss in Appendix A how our analysis can be extended to incorporate overuse and illustrate one calculation incorporating overuse.

Expected Population Value of Quality Improvement (EPV-QI). The MaxPVQI assumes that both the current rate of implementation is known and that quality improvement results in 100% implementation. The expected population value of quality improvement (EPV-QI) reflects the fact that there may be uncertainty about several aspects of the process by which quality initiatives will improve population outcomes. In particular, both the current levels of implementation and the extent to which quality improvement efforts will improve implementation. Indeed, it is well recognized that quality improvement approaches are generally not 100% effective in raising performance to levels of perfection (Oxman et al., 1995). To characterize the uncertainty in this imperfect implementation both before and after QI efforts, let r_SC^PreQI and r_SC^PostQI be the rates of implementation before and after some QI initiative so that ΔrSC = r_SC^PostQI – r_SC^PreQI is the change in implementation before and after the intervention. Because these elements and their change can be uncertain, we reflect this uncertainty by assuming the change in implementation with a quality improvement effort (Δr_SC^QI) is distributed f(Δr_SC^QI) so that the expected extent of quality improvement would be Δr_SC^QI and the expected population value of quality improvement would be:

Equation 5

EPV - QI = N_S X Δr_SC^QI X NHB_Sdc = N_S X NHB_S X Δr_SC^QIdc

Expected Population Value of Quality Indicator Reporting (EPV-QIR). A crucial element in the consideration of quality reporting and the reporting of other indicators is that they do not themselves change quality but instead depend on some sort of action model by which reporting leads to changes in the behavior of providers or others that can improve quality. Fully specifying such an action model is beyond the scope of this paper, but Figure F-1 provides some potentially salient elements of such a model, including that quality reporting would need to produce changes in behavior by either providers or patients in order to produce improvements in quality. Because such changes in behavior are unlikely to completely realize potential quality gains (Schneider and Epstein, 1996), it is important to account for the likelihood that the gain in implementation with quality reporting will generally be less than Δr_SC^QI. We denote this gain in implementation with quality reporting as Δr_SC^QR, and for simplicity assume that the uncertainty in how reporting will effect quality can be represented by a probability of undertaking quality improvement action, π_QI, so that Δr_SC^QR = Δr_SC^QIdc X π_Q is the expected change in implementation with quality reporting and the expected population value of quality reporting is:

Equation 6

EPV - QIR = N_S X NHB_S X Δr_SC^QIdc X π_QI

This equation provides our fundamental framework for developing estimates of the value of quality reporting efforts.

Summary of EPV-QIR Framework

The EPV-QIR framework provides a method for estimating the expected value of improving quality for existing quality measures, measured in units of net health benefits that can be gained within a specified population. This method can be used to estimate the potential value of improving performance on existing quality measures, which can then be used to prioritize measures for reporting or for other investment in quality improvement. Figure F-2 provides a summary of the EPV-QIR approach. First, we assume that reporting on a quality measure leads to quality improvement action with probability π_Q. The effectiveness of a quality improvement action is the effect size of that action, or Δr_SC. The population value of perfect implementation (PVPI) is equal to the net health benefit that can be achieved by improving quality on a measure to perfect or 100% levels of performance. The expected value of quality improvement (EPV-QIR) is the product of the likelihood quality reporting leads to quality improvement efforts, the improvement in implementation that comes from these quality improvement efforts, and the PVPI. Thus, the expected value of quality improvement for a specific quality indicator depends on the probability that quality improvement efforts will be undertaken, the effectiveness of those efforts, and the maximum potential gain in population net health benefits that can be achieved by closing the quality gap for that measure.

The EPV-QIR will equal the PVPI only when reporting a quality measure will result in quality improvement action with certainty and that quality improvement action is 100% effective in perfecting performance. Thus the PVPI and, if current implementation is known, the MaxPVQI, provide bounds on the EPV-QIR.

III. Using the EPV-QIR Framework to Prioritize Measures

Using the EPV-QIR framework to prioritize measures ideally requires data on all the elements included in Equation 6. Because all the other elements depend on defining an intervention in terms of its net health benefits, we begin by exploring the data requirements for NHB and then proceed to defining the other elements. Along the way, we also elaborate on these opportunities identified above in which it may be possible to bound the EPV-QIR using more limited data.

Net Health Benefits. Calculating an estimate of NHB_S requires information on: (1) the total cost of implementing the standard of care per person (or per unit, e.g., per infection avoided); (2) the effectiveness of implementing the standard of care per person (or per unit, e.g., per infection avoided); (3) the total cost of implementing the comparator per person (or per unit, e.g., per infection avoided); (4) the effectiveness of implementing the comparator per person (or per unit, e.g., per infection avoided); and (5) the societal cost-effectiveness threshold. As noted, the societal cost-effectiveness threshold is generally varied across a range of values reflecting the uncertainty in this value from the literature. Items 1-4 may be obtained from published cost-effectiveness studies evaluating the standard of care against the comparator, if such studies exist. Preference should be given to cost-effectiveness studies conducted in a population that is similar, if not the same, as the population defined by the denominator of the measure in question. For example, for the NHQR measure, "Percent of individuals age 65+ who ever received a pneumococcal vaccination," a cost-effectiveness study evaluating the pneumococcal vaccination among adults age 45-55 would be less ideal than a cost-effectiveness study evaluating vaccination among adults age 65-85. Preference might also be given to cost-effectiveness studies conducted in U.S. populations, because difference in healthcare systems might influence total costs of implementing a particular treatment or standard of care. This will affect the validity of net health benefit estimates. It is essential that cost-effectiveness studies publish sufficient data to assess effects on both costs and effectiveness in QALYs for the standard of care/comparator in question. Cost-effectiveness studies that only publish cost-effectiveness ratios (dollars per QALY) are not sufficient to calculate NHB because neither costs nor effectiveness is known.

Number of Individuals Eligible for the Standard of Care. In order to calculate these population-based measures, it is necessary to have an estimate of the number of individuals eligible for the standard of care. In other words, it is necessary to have an estimate of the size of the denominator population. If maximizing population health remains the goal, the eligible population is best selected when the population is defined as that within which the intervention is cost-effective, but if another population is chosen for any reason then the size of that population should be used. To use the same example above, calculating the VPI for the measure "Percent of individuals age 65+ who ever received a pneumococcal vaccination" requires an estimate of the total number of individuals in the U.S. age 65+. For some public-health population-based measures, estimates of the eligible population may be as simple as obtaining age-group and perhaps sex-specific population estimates from the U.S. Census Bureau. For measures denominated on the basis of healthcare utilization such as hospitalizations, weighted population estimates of services and utilization from national healthcare surveys such as the National Hospital Discharge Survey (NHDS) may be useful. For measures defined on the basis of a specific clinical process of care, estimating the size of the denominator population may require estimates of the prevalence of certain conditions.

Rate of Current Implementation. The rates at which individuals in a population receive indicated standards of care are reflected by quality indicators. The denominator of the measure is equal to the measure population (N_S, as defined above), and the numerator of the measure is equal to the number of individuals in the measure population who received the standard of care within some reporting period—i.e., for whom "the standard was met," N_SM X M_S = r_SC = N_SM / N_S.

This data would typically be available for existing quality measures that had previously been collected, allowing for efforts to characterize the maximum potential improvements from existing levels of quality (MaxPVQI). Sources of data for implementation rates include: the National Healthcare Quality Report (NHQR) itself, the Behavioral Risk Factor Surveillance Survey (BRFSS), and other quality reports. For new measures being considered about which nothing is known, less informative bounds based on the population value of perfect implementation (PVPI) might be the most informative bound possible.

Expected Quality Improvements. To develop more precise estimates of the EPV-QIR, it is necessary to know the probability of quality improvement (π_Q) and the effect size of quality improvement interventions (Δr_SC).²

Probability of Quality Improvement (π_Q). One key factor is how providers might approach quality improvement when faced with new quality indicators. Understanding the distribution of quality improvement modalities in the provider population is necessary to derive an aggregate estimate of the effect size—i.e., the amount of change in provider behavior and performance rates that can be expected, conditional on a decision to undertake quality improvement. Indeed, studies have pointed to the heterogeneity of quality improvement efforts undertaken at the provider level (Bradley et al., 2005). As a result, information about both the range and potential effectiveness of these quality improvements efforts will very often be lacking.

Indeed, there are several reasons to believe that π_Q is much less than 1, as there is relatively little evidence supporting a strong direct link between public reporting and quality improvement activities (Epstein, 2006; Fung et al., 2008; Matthews et al., 2007; Robinowitz and Dudley, 2006). Part of the weak link may be attributable to the finding that hospitals and physicians often discount report cards on the basis of methodology, suggesting that in some cases performance reporting may have little direct effect on provider propensity to engage in targeted quality improvement efforts (Rainwater et al., 1998; Romano et al., 1999; Schneider and Epstein, 1996). A second issue complicating the link between public reporting of quality indicators and quality improvement action is that public reporting has often been studied in the context of pay for performance, making it difficult to parse out the independent effect of public reporting on provider quality improvement activities and/or outcomes (Lindenauer et al., 2007; Rodriguez et al., 2009). Finally, insofar as the existing literature has primarily focused on state-level or payer-specific reporting programs, it seems unlikely that responses to quality measures reported aggregated to the national level would elicit a stronger response to initiate focused quality improvement initiatives.

Effectiveness of Quality Improvement (Δr_SC). There are numerous studies of the effectiveness of quality improvement programs (e.g., systems-based interventions to improve cancer screening [Carney et al., 1992; Carpiano et al., 2003]), general approaches to practice/provider behavior change (e.g., continuing medical education [Davis et al., 1995], educational outreach [O'Brien et al., 2007]), and/or specific tools (e.g., printed educational materials [Farmer et al., 2008]) in the context of specific standards of care or clinical conditions (Arnold and Straus, 2005; Renders et al., 2001). However, even when there is some evidence on the efficacy of these approaches, it is unlikely that they will be equally effective in improving performance across different standards of care.

Summary. The relative paucity of evidence on the likely effectiveness of quality reporting on quality improvement activities and of quality improvement activities on implementation of standards of care suggest that efforts to quantify the EPV-QIR will have to rely heavily on bounds implied by estimates of the EPV-QI or MaxPVQI.

¹Quality-adjusted life years (QALYs) are a unit of measurement that is used in quantifying the health benefits or effectiveness of healthcare interventions. QALYs reflect the notion that years of life lived in less-than-perfect health may not be valued as much as years of life lived in perfect health.
² Quality-adjusted life years (QALYs) are a unit of measurement that is used in quantifying the health benefits or effectiveness of healthcare interventions. QALYs reflect the notion that years of life lived in less-than-perfect health may not be valued as much as years of life lived in perfect health.

Return to Contents
Proceed to Next Section