Adopting a More Quantitative and Transparent Measure Selection Process

The IOM Future Directions committee recommends changes to AHRQ's measure selection process in order to focus the outcome of the process on the central aspirations of quality improvement—improving health, value, and equity—by closing performance gaps in health care areas likely to have the greatest population impact, be most cost effective, and have a meaningful impact on eliminating disparities. In order to enhance the transparency of AHRQ's process for measure selection, a Technical Advisory Subcommittee for Measure Selection is recommended under the existing AHRQ National Advisory Council for Healthcare Research and Quality (NAC) to advise on ranking measures for selection, inclusion in the national healthcare reports, and retirement. As part of this process, this subcommittee should recommend strategies for the development and acquisition of new measures and data sources.

Conceptual models of improving health care quality and eliminating disparities include measurement and reporting as integral to achieving performance goals; performance improvement systems, in turn, depend on the quality of data to support measures (Berwick et al., 2003; Kilbourne et al., 2006; Langley et al., 1996). Over the past decade, growing attention to health care quality measurement has led to the generation of a large number of quality measures now being available for use. Illustrating the magnitude of the universe of possible quality measures, the National Quality Measures Clearinghouse inventory now contains 1,475 potential quality measures (National Quality Measures Clearinghouse, 2009a,b). Likewise as of October 2009, the National Quality Forum (NQF) maintained a list of 537 measures meeting its standards for endorsement (NQF, 2009b). The growth in the number of possibilities necessitates a critical assessment of how to prioritize among existing and future measures for use in the NHQR and NHDR. There have been calls to develop a parsimonious common set of measures to "serve policy and frontline information needs" (McGlynn, 2003, p. I-39).

Since 2003, AHRQ has refined its measure set for the national healthcare reports and related products, and the measure set now includes approximately 260 individual measures, including a set of 46 core measures that are more prominently featured in the body of the 2008 NHQR and NHDR. The larger set of 260 measures is featured in online products such as the Web-based State Snapshots, NHQRDRnet, and appendixes to the NHQR and NHDR. The selection of measures for the national healthcare reports by AHRQ has been influenced by the availability of national data sources internal to HHS.

ARHQ has been urged to add more performance measures to the NHQR, NHDR, and related products, and has asked the IOM Future Directions committee for guidance on prioritization among measures so that new measures could be added and highlighted in the reports while other measures could receive less emphasis or be removed entirely from AHRQ's tracking. AHRQ regards the production of the NHQR and NHDR as having reached capacity given the agency's current resources for measurement reporting, analysis, and presentation. AHRQ staff has deliberated about retiring some measures to allow for the incorporation of new measurement domains or measures, but the agency has found it difficult to retire measures because of advocacy, both internal and external to HHS, for each of the current measures.

The Future Directions committee reviewed AHQR's existing measure selection processes and criteria to shed light on how these processes might be improved, particularly in support of the committee's overall aim to have the national healthcare reports focus on areas that matter most and to encourage various stakeholders to take action on the highest impact areas for quality improvement and disparities elimination.

In this chapter, the committee describes how AHRQ's measure selection process might be enhanced by selecting measures that support national priority areas for health care quality improvement (go to Chapter 2), by incorporating concepts of value and equity (go to Chapter 3), and by applying more explicit quantitative techniques in the selection process. Taking these steps would help direct attention to those performance areas with the greatest potential impact to transform health care quality for the country and for specific populations, and identify key areas for measure and data source development.

AHRQ'S Approach To Selecting Measures

The measure selection process for the national healthcare reports has been undertaken primarily by AHRQ staff in consultation with an HHS Interagency Workgroup consisting of program and data experts, as well as with some limited external feedback from AHRQ's NAC.

AHRQ's Initial Measure Selection Process and Criteria

AHRQ's initial selection approach for measures in the NHQR and NHDR began with a call for measures involving all HHS agencies, as well as substantial input from private-sector entities that were solicited by the IOM during the research for its 2001 Envisioning the National Healthcare Quality Report (IOM, 2001). More than 600 candidate measures were generated through the call (AHRQ, 2003a).

Subsequently, the HHS Interagency Workgroup for the NHQR/NHDR reduced the 600 candidate measures for tracking to about 140: (1) by applying three basic criteria recommended by the IOM in 2001—importance, scientific soundness, and feasibility (go to discussion in Box 4-1)—to each individual measure; (2) by mapping potential measures to the elements of the earlier quality framework (effectiveness, safety, timeliness, and patient-centeredness); and (3) by selecting clinically important conditions within effectiveness measures (AHRQ, 2003a).

During the summer of 2002, public comments were solicited from hospitals, providers, researchers, and others via a public hearing conducted by the National Committee on Vital and Health Statistics (NCVHS) and through a Federal Register notice (AHRQ, 2002; NCVHS, 2002). As the HHS Interagency Workgroup refined the final package of measures for the NHQR and NHDR, input was sought from the HHS Data Council, technical and policy experts within AHRQ, and the Quality Interagency Coordination Task Force, which spanned several federal agencies (Veterans Affairs, Department of Defense, Federal Bureau of Prisons, and others).¹ A separate review process was held for home health measures, which were not included in the initial public review cycle (AHRQ, 2003b).

As a result of this effort, the first edition of the NHQR published by AHRQ reported on 147 measures; of these, effectiveness measures (97 measures; 65 percent of the total measures) focused on the clinical conditions chosen for Healthy People 2010 (cancer, diabetes, end-stage renal disease, heart disease, HIV/AIDS, maternal and child health, mental health, respiratory disease, and nursing home and home health care) (AHRQ, 2003a; HHS, 2009b).

Box 4-1. The IOM 2001 Recommendations for Measure Selection Criteria for the NHQR and NHDR In the IOM's 2001 report Envisioning the National Healthcare Quality Report, three major criteria were proposed for measure selection: Importance of what is being measured Impact on health. What is the impact on health associated with this problem? Meaningfulness. Are policy makers and consumers concerned about this area? Susceptibility to being influenced by the health care system. Can the health care system meaningfully address this aspect or problem? Scientific soundness of the measure Validity. Does the measure actually measure what it is intended to measure? Reliability. Does the measure provide stable results across various populations and circumstances? Explicitness of the evidence base. Is there scientific evidence available to support the measure? Feasibility of using the measure Existence of prototypes. Is the measure in use? Availability of required data across the system. Can information needed for the measure be collected in the scale and time frame required? Cost or burden of measurement. How much will it cost to collect the data needed for the measure? Capacity of data and measure to support subgroup analyses. Can the measure be used to compare different groups of the population? The 2001 IOM report stipulated that it is desirable for a measure to meet all 10 elements within the three overall criteria, but noted that it is not required that all 10 apply in order for a given measure to be considered for inclusion in the NHQR and NHDR. The 2001 IOM committee indicated that the three criteria, as listed above, provide a hierarchy by which measures should be considered, with priority to be given to measures evaluated for importance and scientific soundness and then by feasibility. For example, the committee stated: Measures that address important areas and are scientifically sound, but are not feasible in the immediate future, deserve potential inclusion in the data set and further consideration. However, measures that are scientifically sound and feasible, but do not address an important problem area, would not qualify for the report regardless of the degree of feasibility or scientific soundness. Source: IOM, 2001, pp. 83 and 87.

AHRQ's Current Measure Selection Process and Criteria

AHRQ reduced the number of measures presented in subsequent editions in response to criticisms that the first edition was unwieldy (Gold and Nyman, 2004). The intent was to be able to "highlight measures with indepth analysis, rather than broad, but sparse, coverage of all 179 measures" (AHRQ, 2004).² That basic format is maintained by AHRQ today, with a set of approximately 46 core measures presented in the body of the reports and more detailed tables available online for a larger set of measures. To select the 46 core measures for the NHQR and NHDR, AHRQ staff and the HHS Interagency Workgroup prioritized measures by the three original IOM criteria and several additional ones. Usability was added as a new primary criterion—one that is also articulated by NQF in considering the suitability of any measure as a voluntary consensus standard.³ AHRQ's current criteria and principles for prioritizing measures in the NHQR and NHDR are summarized in Box 4-2. AHRQ gives greater weight to "primary criteria" than to "secondary criteria," and the "balancing principles" were also added to ensure that the final set of core measures covered a variety of conditions and sites of care.

AHRQ also emphasizes health care process measures over health outcome measures due to the fact that the focus of the reports is health care delivery and that outcome measures are often too distal or rare (e.g., mortality) to be linked to the delivery of a particular service. Whenever a close relationship is deemed to exist (e.g., use of colorectal cancer screening to presentation with more advanced colorectal cancer), then AHRQ has tried to present paired process and outcome measures. The Future Directions committee recognizes the limitations of process measures, as does AHRQ, and encourages AHRQ to continue to report paired measures whenever possible. Additionally, the committee encourages AHRQ to develop or adopt outcome measures as they hold great interest for policy makers, particularly outcomes associated with the implementation of specific programs. For example, AHRQ already reports on receipt of care for heart attack and inpatient mortality, but could also report related information on outcomes such as: "Since the beginning of public reporting on readmission rates for AMI by the Centers for Medicare and Medicaid Services [CMS], the readmission rates have been reduced X percent, yielding a potential savings to the federal Medicare budget of $Y."

Assessing Importance of Topic Areas for Inclusion

Over time, AHRQ has taken stock of which health conditions or intervention topic areas warranted consideration within the NHQR and NHDR to determine if there should be measurement additions or deletions. AHRQ provided the Future Directions committee with a side-by-side comparison of the specific factors considered in identifying important topics for national reporting (Appendix E). These factors include: the leading causes of death, disability, or activity limitation; principal hospital diagnoses; costly conditions in general and among hospitalizations specifically; areas with notable Black-White racial and educational-level disparities measured in life years lost; other significant racial and ethnic disparities; and priority areas named in several advisory reports from the IOM and HHS (e.g., HHS strategic plans; the 2003 IOM report Priority Areas for National Action: Transforming Health Care Quality⁴ (IOM, 2003). From the sources, AHRQ has identified nationally relevant topics not yet reported in the NHQR, NHDR, or related products. For example, AHRQ added measures, such as obesity and substance abuse measures⁵ to the 2008 reports.

AHRQ's NAC provides advice on content. The NAC and an existing subcommittee consisting of a few NAC members with an interest in the NHQR and NHDR serve as a sounding board for AHRQ staff and provide input to the AHRQ report development process (e.g., recommendations to improve dissemination and to pay increased attention to child health measures; the need to close measurement gaps and set priorities; the need to address cost, waste, and value issues). Thus, the selection of new measures appears to be driven primarily by the need to address new topic areas based on expert opinion (e.g., IOM, NAC, HHS Interagency Workgroup), some general quantitative information about the overall burden of a condition on society and individuals, and the availability of data to report on a topic. In 2008, the NAC observed that the caliber of the NHQR and NHDR has improved with each updating (AHRQ, 2008a), and the Future Directions committee agrees.

Box 4-2. AHRQ's Current Criteria and Principles for Prioritizing Measures Primary Criteria Importance Impact on health (e.g., clinical significance, prevalence). Meaningfulness. Susceptibility to being influenced by the health system (e.g., high utility for directing public policy, and sensitive to change). Scientific Soundness (assumed because AHRQ only uses consensus-based endorsed measures). Feasibility Capacity of data and measure for subgroup analysis (e.g., the ability to track multiple groups and at multiple levels so a number of comparisons are possible). Cost or burden of measurement. Availability of required data for national and subgroup analysis. Measure prototype in use. Usability: easy to interpret and understand (methodological simplicity). Type of Measure: evidence-based health care process measures favored over health outcome measures because most outcome measures were too distal to an identified intervention. Secondary Criteria Applicable to general population rather than unique to select population. Data available regularly/data available recently. Linkable to established indicator sets (i.e., Healthy People 2010 targets). Data source supports multivariate modeling (e.g., socioeconomic status, race, and ethnicity). Balancing Principles Balance across health conditions. Balance across sites of care. At least some state data. At least some multivariate models. Source: AHRQ, 2005.

Improving Measure Selectiom

The Future Directions committee concludes that for the NHQR and NHDR to be more strategic and address the most important opportunities for concerted national action, AHRQ's approach to measure selection needs to be modified. The Future Directions committee recommends broadening the range of input that AHRQ currently receives, making the process transparent, and incorporating a more systematic and quantitative process for ranking measures. The proposed selection process more closely looks at the gap between current and desired performance levels and the relative value of bridging that gap while also taking equity into account. This is a somewhat different approach for AHRQ, one that focuses on closing the quality gap rather than simply selecting conditions and measures based on the highest prevalence and costs.

Focusing on High-Impact Areas

The committee's definition of high-impact areas for quality improvement builds on previous IOM and NQF guidance on determining what constitutes the criteria of importance in measure selection and endorsement (IOM, 2001, p. 83; NQF, 2009a). Specifically, the committee's definition refocuses how AHRQ evaluates "impact on health" for the purposes of selecting measures for the NHQR and NHDR.

High impact areas for health care quality improvement: Ideally, "high impact" quality improvement and disparity reduction areas would be assessed by quantitatively ranking the population health impact of closing the gap between current performance and desired levels of performance (such as 100 percent of persons in need achieving guideline recommended care). These could be assessed for the entire population of the nation and/or for specific priority populations when data allow.

The committee's advice should not be construed to mean that an area would be considered a high impact area solely based on how large the gap is between current performance and desired performance levels (e.g., a spread of 25 percentage points is not automatically more befitting of attention than one that has a spread of 10 percentage points); closure of a smaller gap could be ranked higher than a larger gap if its closure would yield a greater health outcome for the nation's population. While the committee members' emphasis is on quantitative assessment, they are cognizant that data limitations will at times require expert opinion to qualitatively rank measures, particularly in the absence of detailed data to allow assessment of equity considerations for different population groups. In these cases, a qualitative assessment of the impact of the intervention targeted by the measure would be combined with a quantitative assessment of the size of the gap or the disparity in order to rank the relative importance of the measure.

The NAC has observed that health care quality measurement in the United States has been "incremental and evolutionary," unfolding in the absence of a unified performance measurement strategy backed by a plan to obtain data to support key measures. The Future Directions committee hopes that an additional outcome of its proposed measure selection process would be the identification of measure and data needs and the formulation of a strategy for their development.

For the reasons just cited and discussed further below, the committee recommends that AHRQ establish a new Technical Advisory Subcommittee on Measure Selection that can advise the NAC and AHRQ on performance measure selection:

Recommendation 3: AHRQ should appoint a Technical Advisory Subcommittee for Measure Selection to the National Advisory Council for Healthcare Research and Quality (NAC). The technical advisory subcommittee should conduct its evaluation of measure selection, prioritization, inclusion, and retirement through a transparent process that incorporates stakeholder input and provides public documentation of decision-making. This subcommittee should:

• Identify health care quality measures for the NHQR and NHDR that reflect and will help measure progress in the national priority areas for improving the quality of health care and eliminating disparities while providing balance across the IOM Future Directions committee's revised health care quality framework.
• Prioritize existing and future health care quality measures based on their potential to improve value and equity.
• Recommend the retirement of health care quality measures from the NHQR and NHDR for reasons including but not limited to the evolution of national priorities, new evidence on the quality of the measure, or the attainment of national goals.
• Recommend a health care quality measure and data source development strategy for national reporting based on potential high-impact areas for inclusion in AHRQ's national quality research agenda.

The committee's rationale for the establishment of the proposed NAC Technical Advisory Subcommittee for Measure Selection is discussed below. Subsequent sections of this chapter discuss desirable attributes of transparency in AHRQ's process for selecting performance measures, a stepwise process to applying qualitative and quantitative criteria in prioritizing measures, and quantitative methods that have potential applicability to the process for assessing value and equity.

Establishing an Entity for Measure Selection

The Future Directions committee considered several organizational alternatives to take on the responsibility of measure selection, but ultimately recommended the formation of the NAC Technical Advisory Subcommittee for Measure Selection.

Retaining the Status Quo

Retaining the status quo, with responsibility resting with AHRQ staff and HHS Interagency Workgroup members, is considered less desirable, even after possibly supplementing the current process with opportunities for public input and comment, because the process would likely retain its current limitations. The status quo did not appear tenable because AHRQ and HHS Interagency Workgroup members have already acknowledged the difficulty of being able to prioritize and eliminate health care quality measures through the current process (other than plans to semi-retire from the 2009 reports process measures that have a greater than 95 percent achievement rate [AHRQ, 2008a]).⁶ Furthermore, a critical parallel can be drawn to the lessons learned from Healthy People 2010. While Healthy People 2010 contains too many "primarily disease-oriented" objectives, it is nonetheless a "challenge to move away from a biomedical model because it is easier to create specific and measurable health targets that are disease specific," "funding for many of the possible interventions is disease-specific," and there are "strong constituencies," both internal and external, for featuring those diseases (Fielding, 2009). Currently, the NHQR and NHDR are heavily weighted to the clinical conditions in Healthy People 2010, and a Future Directions committee concern is that some of the NHQR and NHDR content may be a product of this same history.

Ultimately, the committee felt strongly that the decision-making process about measures needed to be a public one rather than internal to the HHS Interagency Workgroup and AHRQ staff so that decisions are more transparent and justified to those who advocate for the inclusion or exclusion of specific measures. AHRQ could improve the transparency of its existing practices by (1) publicizing on its Web site the documentation supporting decisions behind the agency's selection of measures and (2) establishing a public comment period on those decisions. However, the Future Directions committee also believes that AHRQ needs more focused external support to make difficult decisions when ranking among measures, particularly as the selection process may result in a substantial change in the portfolio of measures over time. Furthermore, they need the technical, quantitative expertise to evaluate candidate measures.

---

¹ Go to http://www.quic.gov (accessed November 28, 2009) for a full list of member agencies. The HHS Data Council coordinates all health and non-health data collection and analysis activities of HHS, including an integrated health data collection strategy, coordination of health data standards and health information and privacy activities. The HHS Data Council consists of senior level officials designated by their agency or staff office heads, the HHS Privacy Advocate, and the Secretary's senior advisor on health statistics. It is co-chaired by the Assistant Secretary for Planning and Evaluation and a rotating Operating Division (OpDiv) head; AHRQ is the current OpDiv co-chair. For more information, go to http://aspe.hhs.gov/datacncl/ (accessed May 14, 2010).
² Additional measures were added to the initial full measure set.
³ The measure evaluation criteria used by NQF for measure endorsement are available at http://www.qualityforum.org/uploadedFiles/Quality_Forum/Measuring_Performance/Consensus_Development_Process%E2%80%99s_Principle/EvalCriteria2008-08-28Final.pdf?n=4701 [Plugin Software Help] (accessed March 26, 2009). There is substantial overlap in the criteria for measure endorsement and selection to date, whether past IOM recommendations or current AHRQ processes for selection.
⁴ Similarly, NQF uses factors such as "affects large numbers, leading cause of morbidity/mortality, high resource use (current and/or future), severity of illness, and patient/societal consequences of poor quality" in determining the importance of a measure for endorsement (NQF, 2009a).
⁵ Obesity-related measures include ones addressing whether adults with obesity ever received advice from a health provider to exercise more, or whether children received advice from a health provider about healthful eating or being physically active. Substance abuse measurement relates to the number of persons age 12 years and over who needed treatment for illicit drug use and received such treatment at a specialty facility in the past 12 months.
⁶ Personal communication, Ernest Moy, Agency for Healthcare Research and Quality, October 9, 2009.

---

Return to Contents
Proceed to Next Section