Summary
Public recognition of health care quality issues has spiked remarkably
in the past ten years, driven by a series of high-profile reports
from the Institute of Medicine (IOM), the RAND Corporation, and
other organizations.
1-4
These reports showed, among other facts, that preventable medical errors in hospitals
result in as many as 98,000 deaths per year; preventable medication errors occur at
least 1.5 million times per year; and on average, only 55% percent of recommended care
is delivered. In response, a variety of stakeholders from across the spectrum of health
care deliveryincluding providers, professional and hospital associations,
accreditation organizations, employers and business groups, insurance companies, and
state and federal governmentshave focused on monitoring and improving the
quality of care. These efforts have focused on avoiding unnecessary deaths and poor
health, while also encouraging better quality and value for health care spending. In
the current environment, the quality of health care is increasingly recognized as a
product of systems, not individuals, and there is widespread agreement that systematic
measurement, monitoring, and reporting are needed to make meaningful advances in
improving quality.
Health care quality indicators provide an important tool for measuring
the quality of care. Indicators are based on evidence of "best practices"
in health care that have been proven to lead to improvements in health status
and thus can be used to assess, track, and monitor provider performance.
While the results of quality measurement were originally not typically shared
outside the provider organization conducting the quality improvement project,
more recent assessments using the indicators have been included in public reports
intended to steer patients toward higher-quality care and drive providers to
improve their scores in order to bolster their public reputation. Indicators
have also been used to link quality of care to financial incentives, either
in the form of pay-for-performance (i.e., paying more for good performance on
quality metrics), or in the form of tiered insurance products, which steer patients
towards higher-quality providers by charging higher copayments for visits
to providers with poorer quality scores.
The Agency for Healthcare Research and Quality (AHRQ) has been a pioneer in the
development of health care quality indicators. In 1994 its Healthcare Cost and
Utilization Project (HCUP) developed a publicly available set of quality indicators
for hospital care based on discharge data. AHRQ updated the HCUP indicators in 2001,
which were then renamed the AHRQ Quality Indicators (AHRQ QIs). Today, AHRQ maintains
four sets of QIs:
- Inpatient Quality Indicators (IQIs), which reflect the quality of care
provided in hospitals.
- Patient Safety Indicators (PSIs), which reflect potentially
avoidable complications or other adverse events during hospital care.
- Prevention Quality Indicators (PQIs), which consist of hospital admission
rates for 14 ambulatory care-sensitive conditions; and
- Pediatric Quality Indicators (PDIs),
which combine components of the PSIs, IQIs, and PQIs, as applied to the
pediatric population.
The AHRQ QIs are publicly distributed and supported by AHRQ, with regular
updates. They are widely used by a variety of organizations for many different
purposes. Meanwhile, many other organizations, both public and private, have
developed and used their own sets of quality indicators. Given the rapid
growth of and robust demand for quality indicators, it is important to assess
the position of the AHRQ QIs in the quality indicator "market."
- Who is using the AHRQ QIs today, and for what purposes?
- What have users' experiences been, and what unmet needs do they still
have?
- Who else is developing and/or distributing indicators similar to the AHRQ
QIs?
- Most importantly, what has been the impact of the AHRQ QIs on the quality
of care delivered to patients?
To answer these and related questions, the RAND Corporation was asked to
conduct a one-year evaluation to assess user experiences with the AHRQ
QIs and to identify users of other quality indicators, vendors of quality
measurement products using the AHRQ QIs, and developers of quality
indicators comparable to the AHRQ QIs. The
results of this study are intended to inform decisions by AHRQ
on future priorities for the QI program.
This report has three main objectives:
- Provide an overview of the market for the AHRQ QIs as well as indicators
and quality measurement tools developed by other organizations that are
similar to the AHRQ QIs or that incorporate the AHRQ QIs.
- Provide an overview of the range of ways in which the AHRQ QIs are used
by various organizations.
- Assess the market demand for the AHRQ QIs, identify unmet needs,
and discuss implications for future activities by AHRQ.
Return
to Contents
The Market and Uses for AHRQ QIs
While AHRQ has developed four sets of QIs, all the QIs share certain key characteristics.
- Based on administrative data.The AHRQ QIs are based
on hospital discharge data and can be used with existing hospital administrative
databases.
- Outcome-focused. Most of the AHRQ QIs reflect health
care outcomes, not the rates at which evidence-based processes of care
are followed.
- Hospital-focused. Most of the indicators focus
on inpatient care, and all of the indicators are based on hospital data.
- Reviewed for scientific soundness. The AHRQ QIs
were tested by the Evidence-Based Practice Center at the University of
California San Francisco and Stanford University, and detailed documentation
of technical information is available in the public domain.
- Available for public use. The AHRQ QIs and associated
code for SAS, SPSS,a and
Windows are publicly available for download at no cost to the user.
To understand the market for the AHRQ QIs, we conducted a series of interviews
with users of AHRQ QIs, users of other products, developers of similar indicator
sets, and vendors of quality measurement products that include AHRQ QIs. This
review found that the AHRQ QI program fills a unique niche in the market for
QIs since there are no other sources of hospital care quality indicators that
represent both a national standard and are also publicly available, transparent,
and based on administrative data. Many of our interviewees stressed
that the AHRQ QIs fill an important void in this respect.
AHRQ's unique place in the market for quality indicators has led to
a wide proliferation of uses for the AHRQ QIs. Our environmental scan
of users of the AHRQ QIs identified 114 users of the indicators and a range
of different uses, including public reporting, quality improvement/benchmarking,
pay-for-performance, and research. The most common uses of the AHRQ QIs include:
- Research. We identified 43 uses of the AHRQ QIs
for research. For example, Leslie Greenwald and colleagues used the
AHRQ QIs to compare the quality of care provided in physician-owned specialty
hospitals and competitor hospitals.5
- Quality improvement. We identified 23 organizations
that use the AHRQ QIs as part of a quality improvement activity, including
reports benchmarking performance against peers, but do not release the quality
information into the public domain.
- Public reporting. We identified 20 organizations
using the AHRQ QIs for public reporting. We classified an activity
as "public reporting" if a publicly available report was published
that compares AHRQ QI results between hospitals (IQIs and PSIs) or geographic
areas such as counties (PQIs).
- Pay-for-Performance. We identified 4 organizations
that are using the AHRQ QIs in pay-for-performance programs. Three
were health plans and one was a Centers for Medicare and Medicaid Services
(CMS) demonstration project.
As part of our environmental scan for users of the AHRQ QIs, we conducted
a limited review of international uses. We found that the most visible
current endeavor that attempts to make use of the AHRQ QIs is the Organization
for Economic Cooperation and Development's (OECD) Health Care Quality
Indicators (HCQI) Project. The OECD is an intergovernmental economic research
institution headquartered in Paris, France, with a membership of 30 developed
countries that share a commitment to democratic government and the market economy.
The organization recently convened a meeting to work on the development and
implementation of QIs at the international level. Preliminary discussions
indicate that there is interest in using the AHRQ QIs internationally as well
as sufficient data and technical capability to implement them.
Return
to Contents
Key Findings
The results of our interviews and environmental scan focused on four key factors
that can be used as criteria for evaluating quality indicators: importance,
usability, scientific soundness, and feasibility.
Importance
Nearly all of the organizations interviewed stressed the importance of the
AHRQ QI program. AHRQ was frequently mentioned as a "national leader" in
measurement development and research. Many interviewees stated very strongly
that they rely on AHRQ as one of the only sources for publicly available, transparent
indicators based on readily available data. They stressed that without
the AHRQ QIs, they would have few alternatives and would likely have to drastically
change or eliminate their quality reporting and/or measurement activities. Interviewees
generally felt that it was important that a federal agency like AHRQ, which
is regarded as credible and respected, develop and support a quality indicator
set for public use. AHRQ's credibility and the transparency of
the AHRQ QI methods were considered to be key in overcoming opposition to quality
measurement and reporting by stakeholders, particularly providers. Overcoming
this type of opposition is particularly important for public reporting and
pay-for-performance initiatives, where providers' reputations and revenues
are at stake.
Although only one organization in our interviews had formally measured the impact
of AHRQ QIs on the quality of care delivered to patients, many interviewees provided
anecdotal evidence of the effect of the use of indicators on quality. When
asked whether they had measured the impact of using the AHRQ QIs, many interviewees
responded that indicator use began too recently to allow for observation of any
impact. However, many interviewees reported anecdotally that their or their
clients' use of the AHRQ QIs was having some type of impact on quality
of care. The impacts observed usually consisted of an activity such as
putting a new quality improvement process in place, rather than an improvement
in outcomes.
Scientific Soundness
On the whole, our interviewees were impressed by the quality and level of
detail of the AHRQ documentation on the face validity of the indicators and
stated that the indicators captured important aspects of clinical care. Very
rarely were indicators challenged on conceptual grounds. Users largely
felt that the AHRQ QIs can be reliably constructed from hospital discharge
data, but that there was a certain learning curve during which hospital coding
departments had to adjust to the requirements for the QIs. Thus far, coders
had mainly been trained to apply coding rules to fulfill reimbursement requirements,
but now they had to understand that coding practices also had implications
for quality reporting. In selected instances, we heard concerns about ambiguity
in the coding rules that would not provide sufficient guidance on whether to
code an indicator-relevant diagnosis.
Sample size issues (whether due to the rarity of certain procedures or the
infrequency with which some procedures are conducted at certain facilities)
were repeatedly mentioned as a threat to the validity of the indicators,
particularly the PSIs. Most users stated that the indicators were correctly
operationalized within the constraints of the underlying data source. Isolated
findings of specification errors were brought to our attention, but interviewees
emphasized that AHRQ was always able to address those quickly. The limitations
of administrative data were frequently mentioned as a threat to validity.
Usability
Most interviewees stated that the AHRQ QIs provide a workable solution for
their needs and were very appreciative of the support that the AHRQ QI team
provides for implementation and ongoing use. Despite these overall favorable
impressions of the usability of the QIs, three needs related to their usability
for reporting were raised repeatedly: development of reporting templates, development
of composite indicators, and clearer guidance on the use of the AHRQ QIs for
public reporting and pay-for-performance programs.
Standard reporting format. Nine of 54 interviewees
highlighted the need for a standard format for reporting AHRQ QI results as
a top priority. At the simplest level, some interviewees requested AHRQ-supported,
standard, basic names for the AHRQ QIs in plain language, as some of the current
indicator names are difficult for non-clinical audiences to understand. Other
interviewees expressed a desire for more guidance and support on other aspects
of presentation.
Composite indicators. Twelve of 54 interviewees
expressed a desire for an AHRQ-supported methodology for constructing a composite
indicator. Forming composites would allow the results to be summarized into
one statistic, which is easier to grasp and communicate, in particular for
non-expert audiences. Composites would also overcome sample size limitations,
as indicators could be pooled.
Guidance on using AHRQ QIs for public reporting and pay-for-performance.
Interviewees who are currently using the AHRQ QIs for public reporting and
pay-for-performance generally felt that they provided a workable basis for their
activities. Still, interviewees stated that additional standards and guidance on
the reporting of AHRQ QI results were needed. Many interviewees expressed dissatisfaction
with the current AHRQ guidance on the appropriateness of the AHRQ QIs for public
reporting. They felt that clearer guidance from AHRQ would help to counter
opposition from those who argue that the AHRQ QIs should only be used for quality
monitoring and improvement and research, but not as a public reporting or pay-for-performance
tool. Taking the opposing view were several interviewees (mostly hospitals)
who would like to see AHRQ make a clear statement that the AHRQ QIs are not
appropriate for use in public reporting, pay-for-performance, or other reporting activities
because of the limitations of the underlying administrative data.
Feasibility
We were told consistently that a major advantage of the AHRQ QIs was the feasibility
of their implementation. They require only administrative data in the UB-92
format to which many users have routine access, since those data are already
being used for billing and other administrative purposes and have to be collected
and reported by hospitals in most states.
Interviewees emphasized that another substantial advantage of the AHRQ QIs is that
the indicators have clearly defined and publicly available specifications, which
helps with implementation of measurement. These specifications were regarded as
of particular importance for hospitals, as the originators of the data, because the
specifications enable hospitals to work with their coding departments to ensure that
the required data elements were abstracted from medical records consistently and with
high reliability. In addition, users who analyze data with the QIs, such as researchers,
appreciated the fact that they could dissect the indicator results and relate them back
to individual records. That ability helped researchers gain a better understanding
of the indicator logic and distinguish data quality issues from actual quality
problems.
Return
to Contents
Lessons Learned for Future Activities
Interviewees' perspectives provided lessons in three areas: current,
anticipated, and potential development projects involving the QIs; AHRQ's
role as a measures developer and the ways in which users speculate this role
could evolve; and market demand for quality indicators, in particular, user
willingness to pay for QIs.
Priorities for Future Development of QIs
We asked interviewees to prioritize three categories of AHRQ development projects:
- Improvements in the current QI product line.
- Addition of new product lines.
- Improved support for the QI products.
Improving the current products was most frequently seen as the highest priority,
followed by both the addition of new products and improvements in service, outreach,
and user support for the measures. The most commonly requested improvement to
the current QIs was the addition of data elements to the QI specifications,
notably a flag for conditions present on hospital admission (currently in development
by AHRQ), a flag for patients under do-not-resuscitate orders, and clinical
data elements. The most commonly requested new product line was indicators
of ambulatory care. As far as service improvements, the most frequently
mentioned activities were a template and guidance for public reporting of the
QIs, and guidance on next steps in quality improvement following identification
of a potential quality problem using the QIs.
The Future Role of AHRQ Compared to Other Players
Our interviewees held AHRQ in very high regard. The work of the AHRQ
QI team was described as technically sound, sensitive to the limitations of
the underlying data, and transparent. AHRQ is regarded as an intellectual
leader and "go-to" institution for health services research and
the use of administrative data for hospital quality measurement. Several other
organizations, especially the Joint Commission on Accreditation of Healthcare
Organizations (JCAHO), CMS, the Hospital Quality Alliance (HQA), and the Leapfrog
Group, are seen as prominent sources for measures, but their indictors are
generally regarded as complements to the AHRQ QIs. Interviewees were quite
comfortable with AHRQ having a leading role in national quality indicator development.
It was generally viewed as positive that a trustworthy federal institution
had defined open-source and well-documented quality measurement standards. These
standards were viewed as contributing to the transparency of health care quality
measurement and reducing the measurement burden for health care providers by
limiting the number of measurement tools they must use to satisfy various reporting
requirements.
We discussed whether it could be a viable option for AHRQ to give up parts of
the current QI program in order to free up resources and set different priorities.
Almost unanimously, interviewees rejected a model under which AHRQ would develop
and distribute the QI software without supporting it. We received mixed reactions
to a model under which AHRQ would develop and release indicators and their technical
specifications, but no longer provide or support software. Interviewees were
generally wary of the idea of delegating user support and/or software development
and distributions to vendors, fearing that vendors would be prohibitively expensive
or incapable of providing the same quality of support as the original developers.
Willingness to Pay for the AHRQ QIs
As an alternative to AHRQ realigning current funds, we asked interviewees
whether AHRQ might consider financing program growth by generating additional
revenues by charging users. Not unexpectedly, this proposal was not met with
enthusiasm. However, almost half of interviewees (44%) were willing to pay
a "reasonable fee" for access to the full QI resources.
Return
to Contents
Discussion
Limitations
The majority of the interviewees in this evaluation were users of the AHRQ
QIs. Non-users may have more negative opinions of the AHRQ QIs. The
few non-users we did interview did not have express substantially negative
opinions about the AHRQ QI program, but a larger sample of non-users may have
produced different results. This study also focused on uses of the AHRQ
QIs that were publicly discussed or released, so that the results likely do
not fully reflect the use of AHRQ QIs for non-public uses such as confidential
quality improvement activities by hospitals.
What is AHRQ's Current Market Position?
The AHRQ QIs have achieved a strong position in their market segment and no
obvious alternative or competitor could be identified, although some organizations
(notably JCAHO, CMS, HQA, and Leapfrog) have complimentary indicator sets.
This is unlikely to change: new users have an incentive to adopt the prevailing
product, because it makes their results comparable to a large number of other
users. Widespread use lends legitimacy to the product, which is
critical in the often-politicized debates about selecting quality indicators
for public reporting and pay-for-performance.
Where are the Growth Opportunities for the AHRQ QI Program?
There are now a substantial number of users of the AHRQ QIs for public reporting
and pay-for-performance programs. As the prevalence of those activities increases,
we expect the number of users to increase substantially both for the programs
themselves and for internal quality improvement programs and projects that
will attempt to align their target measures with standards for external accountability.
Our interviewees wanted expansion of the AHRQ QI program. They were largely
aware and appreciative of AHRQ's current efforts to improve and expand the program,
but expressed an interest in scaling up, and speeding up, those activities.
How Could Growth Be Financed?
Most interviewees stated that federal funding should be used to support future
AHRQ QI activities, even though they realized that this was a difficult proposition
given the pressure on public budgets in general, and on AHRQ's budget
in particular. Interviewees were reluctant to see AHRQ give up software
development and/or user support. As an alternative, we discussed the option
of AHRQ continuing to provide specifications, software and user support but
starting to charge for those services. While there was little enthusiasm for
user fees, only a few stated that they would stop using the AHRQ QI product
in that case. Most interviewees seemed to be willing to pay a "reasonable"
amount. However, if AHRQ were to implement a charge-based model for the QIs, it would
face the challenge of finding a business model that would generate sufficient
revenue and still be consistent with AHRQ's mission and values as a public agency.
Return
to Contents
List of Symbols
| Symbol |
Definition |
| ACOG |
American College of Obstetricians and Gynecologists |
| AHRQ |
Agency for Healthcare Research and Quality |
| AMI |
Acute myocardial infarction |
| APR-DRGs |
All patient refined diagnosis related groups |
| BCBSMA |
Blue Cross Blue Shield of Massachusetts |
| CMS |
Centers for Medicare and Medicaid Services |
| CHSC |
Center for Studying Health System Change |
| DFW |
Dallas-Fort Worth |
| DFWHC |
Dallas-Fort Worth Hospital Council |
| DI |
DFWHC Data Initiative |
| DRGs |
Diagnosis related groups |
| DVT |
Deep vein thrombosis |
| EC |
European Commission |
| ESQH |
European Society for Quality in Healthcare |
| ETGs |
Episode Treatment Groups |
| GIC |
Group Insurance Commission (State of Massachusetts) |
| HCQI |
Health Care Quality Indicators Project |
| HCUP |
Healthcare Cost and Utilization Project |
| HQA |
Hospital Quality Alliance |
| ICD-9-CM |
International Statistical Classification of
Diseases and Related Health Problems - Version 9 - Clinical Modification |
| IHI |
Institute for Healthcare Improvement |
| IOM |
Institute of Medicine |
| ISQua |
International Society of Quality in Healthcare |
| JCAHO |
Joint Commission on Accreditation of Healthcare
Organizations |
| IQIs |
Inpatient Quality Indicators |
| Mass-DAC |
Massachusetts Data Analysis Center |
| MHA |
Massachusetts Hospital Association |
| NQF |
National Quality Forum |
| OECD |
Organization for Economic Cooperation and
Development |
| PDIs |
Pediatric Quality Indicators |
| PE |
Pulmonary embolism |
| PQIs |
Prevention Quality Indicators |
| PSIs |
Patient Safety Indicators |
| QI |
Quality indicator |
| THCIC |
Texas Health Care Information Collection |
| VBAC |
Vaginal birth after cesarean section |
| WHO |
World Health Organization |
a. Support for SPSS will be discontinued by
AHRQ in 2007.
Return to Contents
Proceed to Next Section
540 Gaither Road Rockville, MD 20850