Back
Methods Used in the National Healthcare Reports
Last Updated: March 2014Contents
- Overview
- Organization of the NHQR/NHDR Measures Within the Integrated NHQR/NHDR Web Site
- Available Comparisons
- National and State-Level Comparisons to Achievable Benchmarks
- National Trends Over Time
- State Snapshot Comparisons
- Reports
- National View
- State View
- Data Query
- Resources
- Acknowledgments
- Endnotes
- Internet Citation
Overview
The newly integrated National Healthcare Quality and Disparities Reports (NHQR/NHDR) Web site provides comprehensive information about health care developments and overviews for policymakers, legislators, and reporters. It also contains detailed data tables for researchers. Comparative information, such as trends over time and current distance from achievable benchmarks, is easily accessible. The user can also “drill down” to more detailed information to demonstrate connections between the broader and more elemental levels.
The design of this Web site was based on recommendations from the Institute of Medicine (IOM) on how to improve upon the NHQR/NHDR reports and related products. The aim was to facilitate greater impact regarding quality improvement and disparities elimination1. A few key goals served as the foundation of the integrated NHQR/NHDR Web site design:
- Bring together the NHQR/NHDR and related products in a way that produces a cohesive story about health care quality
- Present information in forward-looking and action-oriented formats
- Highlight equity as an integral part of overall quality, rather than as a separate story line
- Present data as a solution to health care disparities, with prospects for evidence-based practices
- Improve overall navigation and usability of the site for various audiences
The new NHQR/NHDR Web site integrates information that was previously accessible from three different Web sites:
The following additional features were added to address the IOM recommendations:
- Benchmarks based on performance of the top 10 percent of States, to encourage achievable goals
- Displays that provide consistent definitions and comparisons across national and State levels
- Displays of racial and ethnic comparisons, as a part of the larger quality evaluation
- Provision of subject areas and topics that simplify access to the data and information
This Methods Report includes the following sections:
- An overview of the organization and presentation of materials
- An introduction to each of the five sections of the integrated NHQR/NHDR Web site—Reports, National View, State View, Data Query, and Resources
- A description of the methods behind the various graphics and data presentations within the five sections
Organization of the NHQR/NHDR Measures Within the Integrated NHQR/NHDR Web Site
The NHQR/NHDR reports include approximately 300 different measures that are collected from more than three dozen organizations, including the Agency for Healthcare Research and Quality (AHRQ), the Centers for Disease Control and Prevention (CDC), the National Center for Health Statistics (NCHS), and the Centers for Medicare & Medicaid Services (CMS). The term measure is used to define a specific metric, such as “Adults who received a blood cholesterol measurement in the last 5 years” or “Deaths per 1,000 adult hospital admissions with congestive heart failure.”
To facilitate the evaluation of comprehensive overviews and trends, the 300 individual measures are categorized into six subject areas-
- Diseases and conditions
- Priority populations
- Health insurance
- Access to care
- Types of care
- Settings of care
Each of the six subject areas is further divided into three or more topics, which are provided in Table 1.
| Diseases and Conditions | |
|---|---|
| Cancer | Cardiovascular disease |
| Chronic kidney disease | Diabetes |
| Children | Older adults |
| Priority Populations | |
| Individuals who are Black | Individuals who are Asian or from the Pacific Islands |
| Individuals who are Hispanic | Individuals who are Native Hawaiian or from the Pacific Islands (NHOPI) |
| Health Insurance | |
| Private | Public |
| Uninsured | |
| Access to Care | |
| Structural access | Utilization |
| Patient centeredness | |
| Types of Care | |
| Acute care | Chronic care |
| Prevention | Patient Safety |
| Setting of Care | |
| Ambulatory | Home health and hospice |
| Nursing home | Hospital |
A measure is often reported under more than one subject. For example, the measure “Adults who received a blood cholesterol measurement in the last 5 years” is included under the Cardiovascular Disease within Diseases and Conditions, under Prevention within Types of Care, and under Ambulatory within Setting of Care.
For all of the subject areas except Priority Populations and Health Insurance, the collection of measures is based on chapters in the NHQR/NHDR Report. The measures that are reported under Priority Population and Health Insurance are limited to those that have sufficient data to include them as a reporting category. For example, for “Adults who received a blood cholesterol measurement in the last 5 years,” an estimate for females is included under the Priority Population of Women; however, this measure is not included under the Priority Population of NHOPI, because the data source does not provide an estimate for this racial group. In other words, only a subset of NHQR/NHDR measures has sufficient data to support reporting under Priority Populations and Health Insurance.
Available Comparisons
Within the different subject areas and topics, the integrated NHQR/NHDR Web site presents three types of comparative information-
- National and State-level comparisons to achievable benchmarks
- National trends over time
- State-level comparisons to the all-State average (i.e., State Snapshots)
National trends are always highlighted in the NHQR/NHDR reports. State-level comparisons to all-State averages had been highlighted in the State Snapshot. The addition of the comparison to achievable benchmarks was based on the IOM recommendations to promote best-in-class achievement at both the national and State levels.
National and State-Level Comparisons to Achievable Benchmarks
The incorporation of comparisons to achievable benchmarks sets standards based on the performance of the top 10 percent of States. These standards are considered achievable because they have already been attained by the best performing States.
Achievable benchmarks are only determined for NHQR/NHDR measures that have State-level data for at least 30 States. The benchmark for a measure is calculated as a straight average of the top 10 percent of reporting States. For example, if data from 50 States are available for a measure, then the benchmark is the average of the measure estimate across the best five States. Benchmarks are calculated using the most recent data. For a small number of measures, the data year for the benchmark lags behind the data year for national estimates, because the State-level data are not updated as regularly.
Measure estimates for the nation and individual States are then compared to the achievable benchmark for that measure. Based on this comparison, national and State-level measure estimates are assigned to one of three categories: achieved the benchmark or better, close to the benchmark, or far away from the benchmark. These categories are defined below.
- Achieved the benchmark or better. The value for a measure is no worse than 90 percent of the benchmark value. This includes the case in which the measure’s value is equal to or better than the benchmark.
- Close to the benchmark. The value for a measure is between 50 percent and 90 percent of the benchmark (i.e., the value is worse than the benchmark; it has achieved at least half but not as much as 90 percent of the benchmark).
- Far away from the benchmark. The value for a measure has not achieved 50 percent of the benchmark.
When comparing the estimates to the benchmarks, one critical consideration is the direction of the outcome. Specifically, a low value is the desired outcome for some measures, such as mortality, unmet needs, and communication problems. In contrast, a high value is the desired outcome for other measures, such as receiving recommended tests or reporting good communication. Consider a measure in which a low number is desired, such as “Adults who needed to see a specialist in the last 12 months who found it difficult to see a specialist.” If the benchmark is 18.1 and the national estimate is 14.6, then the measure achieved the benchmark—its value is 19 percent lower than the benchmark (the preferred direction). If the national estimate is 22.0, then it would be considered close to the benchmark—its value is 20 percent higher than the benchmark. If the national estimate is greater than 27.2, the measure would be considered far away from the benchmark—its value is more than 50 percent larger.
National Trends Over Time
National trends are determined for NHQR/NHDR measures with at least 4 years of data. The time span can vary across measures. The average annual percent change (AAPC) is defined as:
Trends are only reported as improving or worsening if the AAPC is at least one percent and the regression slope of the trend is statistically significant at p < 0.1.
State Snapshot Comparisons
Under State Snapshot, each State’s performance is measured relative to a weighted all-State average. This is in contrast to the comparison to an achievable benchmark in which the comparison is to the average of the top 10 percent of States.
A State is excluded from the all-State average if (1) the State estimate is unavailable, (2) the standard error of the State estimate is unavailable, or (3) the relative standard error (RSE) of the estimate was greater than or equal to 30 percent (RSE ≥ 30 percent), which indicates that the rate is unstable. The RSE is calculated by dividing the standard error by the estimate. The all-State average is weighted by the inverse of the State variances, which approximates the count of observations. This approach is used because the NHQR data tables do not include data on the number of observations for many measures.
For each measure, State performance is categorized as better-than-average, average, or worse-than-average relative to the all-State average, if the State-level estimate had a RSE of less than 30 percent. These comparison categories are defined based on a statistical test for differences:
- Better-than-average. The State rate on an NHQR measure is better than the all-State average, and that difference is statistically significant.
- Average. The State rate on an NHQR measure is not statistically different from the all-State average.
- Worse-than-average. The State rate on an NHQR measure is worse than the all-State average, and that difference is statistically significant.
Across a group of measures within a subject area and topic, each State receives a performance meter score. First, points are assigned to each measure as follows:
- 1 point for each State-level measure that was better than the all-State average
- 0.5 points for each State-level measure that was average, relative to the all-State average
- 0 points for each State-level measure that was worse than the all-State average
Next, the points are combined into a meter score:
Where
A = number of better-than-average NHQR measures in the summary
B = number of average NHQR measures in the summary
C = number of worse-than-average NHQR measures in the summary
Meter scores range from 0 (all measures are worse than average) to 100 (all measures are better than average). Scores between 0 and 100 represent the mix of measures that are worse than average, average, and better than average. Higher scores represent better performance, because the score increases with the number of measures that are average and increases more rapidly with the number of measures that are better than average.
A 180-degree colored semicircle that is divided into five categories is used for visual presentation of the data. The five categories are:
- Very Weak: 0 ≤ score < 20
- Weak: 20 ≤ score < 40
- Average: 40 ≤ score < 60
- Strong: 60 ≤ score < 80
- Very Strong: 80 ≤ score ≤ 100
The meter score for a given measure is depicted on the semicircle as a performance meter arrow as shown in Figure 1. A solid arrow is used for the most recent year of available data, if there are at least five measures available. A dashed arrow is used to show performance for the baseline year, when the baseline has more than two-thirds of the measures available in the most recent year. This criterion is applied to ensure similar comparisons between the baseline and the most recent year.
Reports
The Reports section of the integrated NHQR/NHDR Web site allows access to the following:
- NHQR Report
- NHDR Report
- Related Reports
- Fact Sheets
There are Web-enabled tables of contents for the NHQR and NHDR reports, which link directly to the most current report that is viewable on the AHRQ Web site. The reports provide the foundation on which all other sections of the integrated NHQR/NHDR Web site stand.
The section on Related Reports provides links to supplemental reports from the IOM, AHRQ, CDC, and National Academy for State Health Policy (NASHP).
The Fact Sheet section includes additional information on AHRQ research topics and programs: aging, children’s health, computers and medical informatics, coronary artery disease, diabetes, health care costs, health information technology, health literacy and cultural competency, health services research, hypertension, managed care, Medicare and Medicaid, medical errors and patient safety, mental health, minority health, outcomes research, patient-centered care, pharmaceutical research, pneumonia, primary care, health care quality, translating research into practice, women’s health, AHRQ tools, and a purchaser-provider synergies overview.
National View
The National View section of the NHQR/NHDR integrated Web site provides an overview of performance across NHQR/NHDR measures. This perspective shows areas of strengths and weaknesses at a glance. There are two types of comparative information provided in this section—
- Comparison to “achievable benchmarks”for current data
- Trends over time
This information is available across all NHQR/NHDR measures and by subject area and topic.
Comparisons to Achievable Benchmarks
On the National View page, there are three comparisons to achievable benchmarks: across all measures, all measures by race and ethnicity, and all measures by community income. The methods for the benchmark comparison are detailed under the section of this report titled National and State-Level Comparisons to Achievable Benchmarks.
The first summary graphic displays the total frequencies of each category of achievement (Figure 2). If a measure does not have an available benchmark (i.e., there are no State data from which to calculate a benchmark), it is not represented in the summary graphic. In the example in Figure 2, benchmarks are available for 124 measures: 29 measures that are far away from the benchmark, 58 measures that are close to the benchmark, and 37 measures that have achieved or performed better than the benchmark.
The second summary graphic displays the total frequencies of each category of achievement by race and ethnicity (Figure 3). For measures that report data, the comparison to benchmarks is provided for individuals who are Hispanic and non-Hispanic White, Black, and Asian and Pacific Islander (API).
The third (final) summary graphic displays the total frequencies of each category of achievement by community income (Figure 4). For measures that report data, the comparison to benchmarks is provided for low-income and high-income communities.
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the value of the achievable benchmarks, by clicking on any of the bar graphics or on the link labeled “Review underlying data.” An example is provided in Figure 5.
Summarizing Trends Over Time
Within the National View, the user may also view a summary of measures trending over time. The methods for the trend comparison are detailed under the section of this report titled National.
The trends summary graphic displays the total frequencies of each category of achievement. Measures that do not have at least four years of data to document the trend are shown in the “data not available” column. In Figure 6, there are 84 measures showing improvement over time, 58 measures with no change, and 22 measures worsening over time. There is no trending information for 130 measures, which represent 44 percent of all of the measures in this figure— 130/(130 + 84+58+22).
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the annual average percent change, by clicking on any of the bar graphics or on the link labeled “Review underlying data.” An example is provided in Figure 7.
State View
The State View section of the NHQR/NHDR integrated Web site is similar to the National View in that it provides an overview of performance across NHQR/NHDR measures, but the overview is provided for each State. This perspective shows each State’s areas of strengths and weaknesses at a glance. Two types of comparative information are provided in this section—
- Comparison to “achievable benchmarks” for current data
- State Snapshot comparisons to all-State averages
This information is available across all State-level measures and by subject area and topic.
Users begin by selecting a State from the selection map or by using the drop-down box (Figure 8).
State Comparisons to Achievable Benchmarks
Similar to the National View page, the first comparative graphic under State View is a summary of quality measures compared to achievable benchmarks. Three summaries are available: across all State-specific measures, by race and ethnicity, and by community income. The methods for the benchmark comparison are detailed under the section of this report titled National and State-Level Comparisons to Achievable Benchmarks.
The first summary graphic displays the total frequencies of each category of achievement, where State-specific data and benchmarks are available (Figure 9). If a measure does not have an available benchmark, it is not represented in the summary graphic. In the example in Figure 8, benchmarks are available for 155 measures reported for the State: 37 measures that are far away from the benchmark, 70 measures that are close to the benchmark, and 48 measures that have achieved or performed better than the benchmark.
The second summary graphic displays the total frequencies of each category of achievement by race and ethnicity (Figure 10). For measures that report data for the selected State, the comparison to benchmarks is provided for individuals who are White, Black, Hispanic, and Asian and Pacific Islander (API).
The third (final) summary graphic displays the total frequencies of each category of achievement by community income (Figure 11). For measures that report data for the selected State, the comparison to benchmarks is provided for low-income and high-income communities.
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the value of the achievable benchmarks, by clicking on any of the bar graphics or on the link labeled “Review underlying data.” An example is provided in Figure 12.
State Snapshot
The State Snapshot provides insight into a selected State’s performance by comparing it to an all-State average for the available measures. The methods for the State Snapshot are detailed under the section of this report titled State Snapshot Comparisons.
Figure 13 shows the State Snapshot performance meter for a sample State. By comparing the solid blue arrow to the dashed arrow on the figure, the user can see that the State performed better in the most recent data year than it had performed in the past (baseline year). The most recent score was 43.75 across all measures, which is better than its baseline score of 41.9. In addition, the State’s performance is in the “average” range relative to the comparisons to the all-State averages. Beneath the performance meter, there is a table that lists the meter score for all States. The list can be sorted by State or by descending meter score.
The user can obtain information about (1) which measures are included in the performance meter score and (2) the average annual percent change, by clicking on any of the bar graphics or on the link labeled “Review underlying data.” An example is provided in Figure 14.
State Dashboard
The State Dashboard provides “snapshot” information across all subject areas and topics on one Web page. The first graphic is the State Snapshot performance meter across all available State-level measures. Next, rectangular versions of the performance meter present information on each subject area and topic. The methods for the State Snapshot meters are detailed under the section of this report titled State Snapshot Comparisons.
Figure 15 shows a portion of a sample State Dashboard. The solid blue triangles indicate performance in the most recent data year; the white triangles indicate performance in the baseline year. The State’s performance has improved in the areas of cancer, cardiovascular disease, chronic kidney disease, and diabetes. For chronic kidney disease, the State’s performance improved from the average range to the strong range from the baseline to the most recent year. Quality performance in the area of HIV/AIDS has not changed from the baseline year to the most recent year, as indicated by two arrows in the same location. The meters for mental health and substance abuse and for musculoskeletal disease are portrayed in gray because there are no data on these measures for the selected State.
Data Query
The data query section of the NHQR/NHDR integrated Web site provides detailed analytic capabilities that allow the user to view individual measures rather than summaries of performance. Figure 16 contains a copy of the data query selection form. The user must complete the following hierarchy to narrow the selection:
- Geographic area—national or a State
- Subject area—diseases and conditions, priority populations, health insurance, access to care, types of care, or settings of care
- Topic within the subject area.
These selections define the list of available measures. Only one measure can be selected at a time. The user can choose to examine the measure over time or by one or two categories. The available categories are determined by the measure and depend on the available data reported to the NHQR/NHDR from the original data source.
Based on the user’s selections, the data query will return a data table of the information. The information is a subset of the rows in the full NHQR/NHDR data table for the chosen measure. The results of a query on the national level for the cardiovascular measure “Deaths per 1,000 hospital admissions with congestive heart failure (CHF)” are shown in Figure 17, by ethnicity and gender. The full NHQR/NHDR table is available by clicking on the link for the table number. The section of the table selected by the data query may be saved to Microsoft Excel using the link “Excel — Open/Save File.”
A graphic depiction of the selected data query is available by selecting the “Graphic”tab above the data table. When the query is specific to trends over time, a line graph is displayed (Figure 18). When one or two categories are selected, bar graphs are used to depict the data (Figure 19).
Resources
The Resources section of the NHQR/NHDR integrated Web site contains a catalog of publications and Web tools. These resources provide information that is useful to quality improvement and disparities reduction activities. They are categorized into seven focal areas—
- Raising awareness
- Collecting data
- Analyzing data
- Reporting data
- Identifying best practices
- Tracking success
- Focusing on specific vulnerable populations
The source, title, and a brief description are provided for each resource. Figure 20 shows an example of the resources under Focusing on Specific Vulnerable Populations.
Acknowledgments
The 2012 NHQR/NHDR Integrated Web Site was developed from information in the 2012 National Healthcare Quality and Disparities Reports through a team effort. The team included Ernest Moy, Karen Ho, Doreen Bonnett, and Biff LeVee from the Agency for Healthcare Research and Quality; Sheryl Davies and Kathryn McDonald from Stanford University; Rosanna Coffey, Anika Hines, Laurie MacCallum, Anne Pfuntner, and Natalia Coenen from Truven Health Analytics; Brian Eppert from Coding Leap; Marguerite Barrett from M.L. Barrett, Inc.; and Dana Douglas from User Works.
End Notes
Internet Citation
NHQR/NHDR Web Site: Methods Report. November 2013. Rockville, MD: Agency for Healthcare Research and Quality.