Module 2: Data — Understanding the Foundation of Quality Improvement

"Health care is crucial to our quality of life and is one of the biggest, and probably the fastest growing financial burdens for government, business and individuals. It is complicated, and we are learning by experience. Good decisions will make (the State) healthier and the State economically competitive, poor decisions will not. We need reliable and current data to make good decisions." — Robert Huefner, Ph.D., Member, Utah Health Data Committee Testimony to the Health and Human Services Appropriations Committee, January 10, 2002

Module Overview: Quality Measurement Background Diabetes-Related Quality Measures in the NHQR Sources of NHQR Data on Diabetes Care Process Measures—BRFSS and MEPS Data Outcome Measures—NHANES, HCUP, and NHDS Data Filling Local Data Gaps Developing an Inventory of Local Data Sources Using Published Studies and Readily Available Data To Develop State or Local Estimates Summary and Synthesis Resources for Further Reading List of Associated Appendixes for Use With This Module Key Ideas in Module 2: Data are essential to quality improvement — essential for identifying and measuring problems and setting goals for improvement. The first two steps are: 1) identifying measures and 2) identifying data sources to support those measures. The NHQR is a valuable resource for consensus-based measures, national and State-level data sources, and estimates for tracking diabetes care quality. State leaders must understand the limitations of data sources to be able to handle challenges who will say that "the data are the problem, not the health care system." States also have a wide array of other data sources. Gaps in State-level data can be filled by using methods from published national studies and available State-level data, such as that collected or analyzed by State DPCP staff.

A key ingredient to improving health care quality is data. The term data usually refers to values or estimates generated to describe a concept and to track it over time, space, and populations. Data reveal the extent of a problem, the subpopulations involved, and the geographic disparities in outcomes and processes of care. Data are necessary to make the case for diverting scarce State resources (staff or budgets) to a quality improvement initiative.

Exploring available data is a productive way to begin the process of identifying quality problems and selecting and defining an improvement project. Furthermore, the quality improvement process is a cycle (explained in Module 5: Improvement) that rests on the backbone of data. Data are necessary to assess the situation at a baseline and ultimately to determine whether an intervention is accomplishing what was intended or whether objectives and actions need to be changed to improve quality.

The National Healthcare Quality Report, with national and sometimes State-level data, is a valuable resource for reviewing and comparing health care quality across the States. It is a source of accepted measures and benchmarks for comparison. (Benchmarks are explained in Module 3: Information.)

This module discusses the basic building blocks of quality improvement — measurement and data. The Module describes the diabetes-related data available in the NHQR and other relevant data sources that States can use.

Even when data are not readily available, estimates can be generated by assembling information from various sources. Two practical examples of this are developed in this module for the Medicaid and State populations. The results of research studies combined with national and State databases are used to estimate the Medicaid spending on diabetes care and the cost burden of diabetes to each State.

What this module does not address are the wide-ranging possibilities, constrained only by resources, of collecting data through surveys tailored to planned projects and aimed at measuring the scope of the quality problem and evaluating the effectiveness of planned interventions. Each State has a cadre of health statisticians and analysts who should be recruited to be part of any quality improvement project aimed at the health care system in the State.

Return to Contents

Quality Measurement

Background

This section reviews the concept of quality measurement, available diabetes-related measures in the NHQR, and the importance of using multi-dimensional measure sets. All of this is from the perspective of State quality improvement programs.

Conceptual design of quality measures is necessary before data collection can begin. What is to be measured? How should it be measured? How will it be analyzed?

Fortunately, finding measures of health care quality is not difficult. Much work has been done over the past 30 years to advance the field of quality measurement. In fact, the plethora of measures can actually frustrate health care providers and analysts: Which should be used to guide and evaluate a quality improvement program? What do the measures mean? How should individual values be interpreted?

Quality measures cover a large range, from crude measures (e.g., unadjusted mortality rates) to more refined measures (e.g., percent of an at-risk population achieving glycemic control as evidenced by HbA1c levels). While a full range of measures is essential for a complete picture of health care quality, specific process measures are needed to move a health care team toward delivering quality care. For example, the number of deaths at a hospital can suggest poor quality of treatment at that hospital, but knowing the number of deaths does not tell the hospital staff how to improve. Quality measures of processes of care that are linked to increases or decreases in deaths or other medical outcomes help medical staff know how to change care in order to improve patient outcomes.

There is a distinction between quality measures and guidelines for quality care. The health care quality measures used in the NHQR and used for State, regional, or local planning for quality improvement initiatives relate to populations. Such measures are often rates (e.g., percentages) which indicate the number achieving a goal (e.g., glycemic control) relative to a population base (e.g., all people with diabetes in the Nation).

By contrast, guidelines for quality care are recommendations devised via consensus processes of clinical experts that describe standards of care for individual patients. In general, guidelines for quality care of individual patients are used as the theoretical underpinning to develop population-based quality measures.

Most quality improvement efforts focus on process and outcome measures (go to text box below). Process measures often reflect evidenced-based guidelines of care for specific conditions. Outcome measures often relate to patient health status. Ideally, improvement in a particular process measure yields improvement in the associated outcome measure. Structural measures of the health care infrastructure are a third type of quality measure, less directly related to quality of care.

Types of Quality Measures: Process measures often are based on guidelines of care for a specific condition. Process measures are generally considered to be within the control of the provider and, therefore, are considered performance indicators. They also are more likely to reveal actions that can be taken to improve quality (for example, whether a necessary test or medication is given). Outcome measures generally are based on patient health status. They are considered to be the ultimate objective of quality improvement — improving the patient’s health (for example, mortality rates, hospitalization rates, and test results). Structural measures reflect aspects of health care infrastructure that generally are broad in scope, system wide, and difficult to link to short-term quality improvement (for example, the staff-to-bed ratio in a hospital). The NHQR does not use structural measures.

Diabetes-Related Quality Measures in the NHQR

Although many process measures exist for diabetes care, those listed below survived an extensive consensus process developed for the NHQR and could be estimated from national databases. (Go to Appendix C for more information on national quality measurement activities and the NHQR measure selection process). The NHQR uses five process measures and seven outcome measures; the outcome measures are of two types—test results and avoidable hospitalizations.

Process Measures

• HbA1c test—Percent of adults with diabetes who had a hemoglobin A1c measurement at least once in the past year.
• Lipid profile—Percent of patients with diabetes who had a lipid profile in the past 2 years.
• Eye exam—Percent of adults with diabetes who had a retinal eye examination in the past year.
• Foot exam—Percent of adults with diabetes who had a foot examination in the past year.
• Flu vaccination—Percent of adults with diabetes who had an influenza immunization in the past year.

Outcome Measures

• Test results—The NHQR uses the three measures listed below:
  • HbA1c levels—Percent of adults with diagnosed diabetes with HbA1c levels >9.5 percent (poor control); <9.0 percent (needs improvement); and < 7.0 percent (optimal control)
  • Cholesterol levels— Percent of adults with diagnosed diabetes with most recent LDL-C level <130 mg/dL (needs improvement); <100 (optimal)
  • Blood pressure—Percent of adults with diagnosed diabetes with most recent blood pressure <140/90 mm/Hg

• Avoidable hospitalizations—The NHQR uses the four measures listed below:
  • Hospital admissions for adults with uncomplicated, uncontrolled diabetes per 100,000 population
  • Hospital admissions for adults with short-term complications of diabetes per 100,000 population
  • Hospital admissions for adults with long-term complications of diabetes per 100,000 population
  • Hospital admissions for lower extremity amputations for patients of all ages with diabetes per 1,000 population

Ideally, improvement in a process measure will yield improvement in an associated outcome measure. An example of this, used by the NHQR is the diabetes process measure of an annual HbA1c test to monitor blood glucose levels. Control of blood glucose in people with diabetes has been connected with the delay of complications. Such complications often result in hospitalization. Hospitalizations for uncontrolled (long-term and short-term) complications of diabetes are outcome measures used in the NHQR. In this case, improvement in the process of monitoring HbAlc is expected to decrease the number of such hospitalizations, as diagramed in Figure 2.1. Of course, the connections are never that simple or direct. An HbA1c test does not necessarily mean that a patient will self-manage the disease sufficiently or the clinician will provide the appropriate intervention to lower an HbA1c level and decrease long-term complications. Effective patient and provider education is a crucial link.

Return to Contents

Sources of NHRQ Data on Diabetes Care

This section describes actual estimates for the diabetes measures above from the NHQR as well as other sources of data that may help States generate estimates or analyze factors related to the quality of diabetes care. The quality of the data itself is discussed throughout this section, because State leaders in quality improvement must understand issues that will be raised in the improvement process. Health care providers may argue that the data, due to limitations, do not reflect reality. They may say: "The data are the problem and not the health care system." Understanding data limitations leads to responsible use of data.

The NHQR uses many different data sources (go to Appendix B for a complete list). Different sources use different methods, definitions, and classifications. Some sources produce estimates by State and some by national population subgroup, such as race/ethnicity, gender, age, and income.

The diabetes data in the NHQR come from five data sources:

• Behavioral Risk Factor Surveillance System, a telephone survey designed by the CDC and conducted by individual States. BRFSS data are the only diabetes-related data reported by State in the NHQR (except for a special analysis using HCUP data discussed in Module 3: Information). BRFSS provides State-level estimates for four of the five process measures.

• Medical Expenditure Panel Survey–Household Survey, a national in-person survey, conducted by AHRQ. MEPS data are used for all five process measures and report data by national population subgroup.

• National Health and Nutrition Examination Survey, a physical examination survey conducted by clinicians who staff a tractor-trailer clinic that travels to sampled communities under the auspices of the National Center for Health Statistics (NCHS/CDC). NHANES is used for two laboratory value-related outcome measures that require clinical data from physical examinations.

• Healthcare Cost and Utilization Project (HCUP), a census of hospital discharge records for States (29 in 2000) in a Federal-State-Industry partnership, sponsored by AHRQ. HCUP data are used to report on three outcome measures of avoidable hospitalizations.

• National Hospital Discharge Survey (NHDS), a national sample of hospitals and a sample of their discharges, conducted by NCHS. NHDS is used for one outcome-related avoidable hospitalization.

General information on each data source and its limitations are presented next. The most detail is presented on BRFSS because it is the only NHQR diabetes data that reports by State. Following those discussions, Table 2.1 presents the State-by-State rates for the four BRFSS process measures. Appendix C includes a more in-depth discussion of each data source and other NHQR data tables. Data tables in Appendix C from sources other than BRFSS present national rates and data by subgroup.

Process Measures—BRFSS and MEPS Data

Behavioral Risk Factor Surveillance System

BRFSS data used in the NHQR are from 2001; in that year, 41 States collected data for three of the five diabetes process measures in the NHQR. Those measures include annual HbA1c testing, foot exams, and eye exams. All 50 States collected data on receipt of influenza vaccination in the past year.

The BRFSS data are based on telephone surveys developed by the CDC but administered by each State independently. The survey consists of a core set of questions developed by CDC, additional questions developed by the States, and separate, optional modules for States to use. The diabetes module, which contains the quality-of-care questions, is optional for State use. More information about the BRFSS data and methods as well as interactive databases with some State and local level diabetes data are available at: http://www.cdc.gov/brfss/.

Limitations of BRFSS data: Every data source has limitations. They relate to the population represented, methods used to collect the data, definitions, and analytic approaches. These factors affect the estimates generated from a data set. When similar measures from two data sets differ, the cause can usually be traced to the limitations of the data sets. By understanding the limitation of a data set, the strengths and weakness of estimates from the data set can be assessed and the estimates can be used more responsibly. Limitations of BRFSS data include the following:

• BRFSS samples are kept small to minimize survey costs for States. The State BRFSS samples for the year 2001 range from 1,888 to 8,628 respondents (go to: http://www.cdc.gov/brfss/technical_infodata/surveydata/2001/codebook_01.rtf). For respondents with diabetes the sample is even smaller, generally around 200 (Mukhtar, Murphy, Mitchell, 2003; Safran, Mukhtar, Murphy, 2003). Small samples increase the variance of estimates and decrease the size of the difference between two subpopulations that can be detected through the survey responses.

• The BRFSS survey excludes people without a residential phone and people who are institutionalized. This means that the total population of interest—all people with diabetes—will not be represented in the estimates that come from the survey (Nelson, Holtzman, Bolen, et al., 2001). This weakness can be dealt with by carefully discussing BRFSS results in relation to the population it represents.

• BRFSS data are self-reported and reflect the perceptions of respondents. An advantage of self-reports is that they can reveal information that cannot be obtained from other sources; for example, the receipt of flu vaccinations for people who don’t see a doctor during the year. A disadvantage of self-report data is that respondents may have difficulty recalling events, understanding or interpreting questions, or responding truthfully to questions about socially unacceptable behaviors. Furthermore, cultural and language barriers and limited health knowledge can affect the quality of self-reported data (Nelson, Holtzman, Bolen, et al., 2001). These problems may occur with different propensity for different subgroups.

• BRFSS data, like most surveys, are limited by budget constraints. Because BRFSS is funded by State which vary considerably in the levels of their budgets allocated to health surveys, these fiscal disparities may affect the quality of the data across States. Such data quality shortcomings can include bias from differential response rates, varying followup periods, and variations in interviewer protocols (e.g., extent of probing for answers).

Addressing small sample size limitations: One way to deal with small samples is by pooling data over two or three years. In 1999, when the CDC incorporated evaluation and program accountability requirements for the State diabetes control programs, it provided baseline estimates of State rates for HbA1c testing, eye exams, foot exams, and self-monitored blood glucose by pooling the data from 1997 through 1999. A more stable baseline facilitated comparisons among the States and enabled States to monitor improvements (Safran, Mukhtar, Murphy, 2003). (Tables C.6 through C.9 in Appendix C include these baseline estimates and BRFSS trends for various years.

Because the NHQR uses data from only one year, Module 3: Information takes sample size into account when interpreting the data on diabetes quality measures from BRFSS.

Despite limitations, BRFSS diabetes data are widely used by State DPCP coordinators. Seventy percent of State coordinators surveyed reported that they used those data for program evaluation, publications, or program implementations. When rating the usefulness of the questions in the diabetes module, State coordinators rated HbAlc testing, eye exams, foot exams, self-monitoring of blood glucose, and diabetes education as "highly useful" (Mukhtar, Murphy, Mitchell, 2003).

BRFSS estimates for diabetes care quality: Table 2.1 shows estimates for the four NHQR measures derived from BRFSS and includes estimates for the revised HP2010 objective for HbA1c testing at least twice annually. These estimates are reported nationally (over all 41 contributing States) and by individual State. Each of the four measures includes the estimate of the rate per 100 people (or percent) and the standard error of the rate (which is affected by the sample size).

Table 2.1 also indicates statistical significance for each State compared to the national average and the top decile of States. (The top decile or "best in class" benchmark is explained in Module 3: Information.) Two different statistical significance tests are represented in symbols as follows:

• Test of difference from the national average—For this test, the symbols + and - represent the State rate that is statistically above (+) or below (-) the national average. States with no adjacent symbol are not statistically different from the national average.

• Test of difference from the average of the best-in-class States—For tests of difference from the top-decile States, the symbol ‡ indicates States that are indistinguishable from the best-in-class States. States without the ‡ symbol are statistically different from the best-in-class average.

The maps in Figure 2.2 summarize the five BRFSS measures found in Table 2.1 in relation to the national average rates. The hues show which States are statistically significantly below or above the average, those that are not different from the average statistically, and those that do not collect data.

Medical Expenditure Panel Survey

The NHQR uses data from the Medical Expenditure Panel Survey to report national rates by national subgroup for five process measures. Four measures are the same as those from BRFSS—HbA1c testing, eye exams, foot exams, and influenza immunizations. The fifth measure is lipid profile—the percentage of people with diabetes who reported receiving a test for lipid profiles in the past 2 years.

MEPS is a family of surveys, including a household survey and surveys of related health care providers. Information is collected annually on health care utilization, expenditures, and health insurance coverage. For the most part, MEPS data are collected using computer-assisted, in-person interviews. The diabetes component is collected via a separate paper and pencil questionnaire distributed to respondents who report that they have been diagnosed with diabetes. More information about MEPS data and methods are available at http://www.meps.ahrq.gov/mepsweb/data_stats/data_overview.jsp.

Differences between MEPS and BRFSS: MEPS reports on the same process measures as BRFSS nationally but does not produce State-level estimates. Notable differences exist between MEPS and BRFSS national rates for HbA1c testing and influenza immunization. The HbA1c MEPS-BRFSS difference (90 percent versus 79 percent) is due to different survey response options and the order of the questions. The MEPS-BRFSS influenza immunization difference (55 percent versus 37 percent) is due to different age-group definitions between the two surveys; the MEPS rate is for adults age 18 and over; the BRFSS rate is for adults age 18 to 64. Since flu shots are less likely to be given to younger than to elderly people, the BRFSS rate is lower than the MEPS rate. More information on differences between MEPS and BRFSS is provided in Appendix C.

Outcome Measures—NHANES, HCUP, and NHDS Data

National Health and Nutrition Examination Survey

The NHQR uses data from the National Health and Nutrition Examination Survey for two outcome measures related to diabetes—the average blood glucose level over the prior 2 to 3 months and blood pressure at examination. NHANES, which uses a relatively small sample size because of the costliness of conducting physical examinations in communities, does not support State-level estimates. NHANES does provide estimates for the Nation that could be used as benchmarks over time. These benchmarks would be valuable to a State that has the same clinical measures for some population within the State (such as health systems with electronic medical records) or if the State establishes special data collection through health care providers for such measures. (Note: To be comparable to data from providers, the NHANES HbA1c and blood pressure values would have to be recalculated to exclude people who do not use the health care system during a year.) Additional information on NHANES is available at: http://www.cdc.gov/nchs/about/major/nhanes/NHANES99_00.htm.

Healthcare Cost and Utilization Project

The NHQR uses inpatient discharge data from the Healthcare Cost and Utilization Project for national estimates of three outcome measures of avoidable hospitalizations related to diabetes. HCUP is a public-private partnership sponsored by AHRQ with 29 participating States that covers about 80 percent of U.S. discharges in the United States in 2000, the time for which data are included in the first NHQR. While national diabetes estimates from HCUP are included in the NHQR, State-level data are not, except for one special analysis of admissions for uncomplicated uncontrolled diabetes (discussed in Module 3: Information). Additional information on HCUP data is available at: http://www.hcup-us.ahrq.gov/overview.jsp.

AHRQ also has developed the Quality Indicators (AHRQ QIs) for use with HCUP and other hospital administrative data (AHRQ, 2001, 2002, 2003). The AHRQ QIs use sophisticated clinical algorithms of inclusions and exclusions to define patient groups at low risk of poor health outcomes and then calculate the outcomes of these low risk groups across different settings and populations. The algorithms have been tested, reviewed, and hewn by clinical consensus panels under AHRQ sponsorship. The AHRQ QIs include the Prevention Quality Indicators, which estimate rates of avoidable admissions, including diabetes admissions, as an indirect measure of the quality of ambulatory diabetes care in the United States. As tools for local quality improvement, the AHRQ QIs can be used as screens for quality problems that call for more in-depth local study; they are not considered definitive measures of local quality of care. As national measures they capture trends in quality as well as coding of diagnoses. National estimates of the Prevention Quality Indicators are part of the first NHQR and NHDR. Additional information on the AHRQ QIs is available at: http://www.qualityindicators.ahrq.gov/.

Limitations of HCUP data: The main limitation of HCUP data (or any administrative billing data) is that the data are collected for the purpose of payment, and what is coded as clinical diagnoses and procedures can be affected by reimbursement incentives (Keating, Landrum, Landon, 2003). Such incentives can encourage or discourage coding of specific types of conditions or treatments. Nevertheless, HCUP data can be used for many purposes, provided that the bias of coding is considered and ruled out as inconsequential. Thus, while administrative hospital data can be mined for clues to quality of care, analysts should be alert for whether the data contain incomplete entries or inadequate clinical detail.

National Hospital Discharge Survey

The NHQR used the National Hospital Discharge Survey for one outcome measure—lower extremity amputations. The NCHS at CDC uses a national sample of hospitals and a sample of their discharges to collect administrative hospital records for the NHDS (similar to HCUP). The sample consists of about 270,000 inpatient records from about 500 hospitals and is representative of inpatient discharges nationally. Additional information on NHDS data is available at: http://www.cdc.gov/nchs/about/major/hdasd/nhdsdes.htm.

Limitations of NHDS data: The limitation of NHDS data are similar to those for HCUP data (described above) because NHDS also uses discharge records or inpatient claims for reimbursement. In addition, although NHDS is a true probability sample, it has a much smaller sample size than HCUP. As a result, many subgroup estimates that can be made with HCUP cannot be supported with NHDS data.

Return to Contents
Proceed to Next Section