Executive Summary
The measurement of health care efficiency has lagged behind the measurement of health care
quality. Providers, payers, purchasers, consumers, and regulators all could benefit from more
information on value for money in health care. Purchasers, particularly large employers, have
been demanding that health plans incorporate economic profiling into their products and
information packages. Despite the importance, there has not been a systematic and rigorous
process in place to develop and improve efficiency measurement as there has been for other
domains of performance. Recognizing the importance of improving efficiency measurement, the
Agency for Healthcare Research and Quality (AHRQ) has sponsored this systematic review and
analysis of available measures. Our work was designed to reach a wide variety of stakeholders,
each of which faces different pressures and values in the selection and application of efficiency
measures. Thus, we anticipate that some sections of the report will be less useful to some
readers than others. This report should be viewed as the first of several steps that are necessary
to create agreement among stakeholders about the adequacy of tools to measure efficiency.
Methods
Typology
Because we found that many stakeholders attach different meanings to the word "efficiency,"
we first developed a definition of efficiency. We believe that being explicit about how the term
is being used is helpful in advancing the dialogue among stakeholders. In this report, we define
efficiency as an attribute of performance that is measured by examining the relationship between
a specific product of the health care system (also called an output) and the resources used to
create that product (also called inputs). Under our definition, a provider in the health care system
(e.g., hospital, physician) would be efficient if it was able to maximize output for a given set of
inputs or to minimize inputs used to produce a given output.
Building on this definition, we created a typology of efficiency measures. The purpose of the typology is to make explicit the content and use of a measure of efficiency. Our typology has three levels:
- Perspective: who is evaluating the efficiency of what entity and what is their objective?
- Outputs: what type of product is being evaluated?
- Inputs: what resources are used to produce the output?
The first tier in the typology, perspective, requires an explicit identification of the entity that is evaluating efficiency, the entity that is being evaluated, and the objective or rationale for the assessment. We distinguish between four different types of entities:
- Health care providers (e.g., physicians, hospitals, nursing homes) that deliver health care services.
- Intermediaries (e.g., health plans, employers) who act on behalf of collections of either providers or individuals (and, potentially, their own behalf) but do not directly deliver health care services.
- Consumers/patients who use health care services.
- Society, which encompasses the first three.
Each of these types of entities has different objectives for considering efficiency, has control
over a particular set of resources or inputs, and may seek to deliver or purchase a different set of
products. Efficiency for society as a whole, or "social efficiency," refers to the allocation of
available resources; social efficiency is achieved when it is not possible to make a person or
group in society better off without making another person or group worse off. The perspective
from which efficiency is measured has strong implications for the measurement approach,
because what looks efficient from one perspective may look inefficient from another. For
example, a physician may produce CT scans efficiently in her office, but the physician may not
appear efficient to a health plan if a less expensive diagnostic test could have been substituted in
some cases. The intended application of an efficiency measure (e.g., pay-for-performance,
quality improvement) offers another way of assessing perspective.
The second tier of the typology identifies the outputs of interest and how those will be
measured. We distinguish between two types of outputs: health services (e.g., visits, drugs,
admissions) and health outcomes (e.g., preventable deaths, functional status, clinical outcomes
such as blood pressure or blood sugar control). The typology addresses the role of quality (or
effectiveness) metrics in the assessment of efficiency. A key issue that arises in external
evaluations of efficiency is whether the outputs are comparable. Threats to comparability arise
when there is (perceived or real) heterogeneity in the content of a single service, the mix of
services in a bundle, and the mix of patients seeking or receiving services. Pairing quality
measures with efficiency measures is one approach that has been suggested by AQA and others
to assess comparability directly.
In this typology, we do not require that the health service outputs be constructed as
quality/effectiveness metrics. For example, an efficiency measure could consider the relative
cost of a procedure without evaluating whether the use of the procedure was appropriate.
Similarly, an efficiency measure could evaluate the relative cost of a hospital stay for a condition
without considering whether the admission was preventable or appropriate. However, the
typology allows for health service outputs to be defined with reference to quality criteria. That
is, the typology is broad enough to include either definition of health services. We deliberately
constructed the typology in this way to facilitate dialogue among stakeholders with different
perspectives on this issue.
The third tier of the typology identifies the inputs that are used to produce the output of
interest. Inputs can be measured as counts by type (e.g., nursing hours, bed days, days supply of
drugs) or they can be monetized (real or standardized dollars assigned to each unit). We refer to
these, respectively, as physical inputs or financial inputs. The way in which inputs are measured
may influence the way the results are used. Efficiency measures that count the amounts of
different inputs used to produce an output (physical inputs) help to answer questions about
whether the output could be produced faster, with fewer people, less time from people, or fewer
supplies. In economic terms, the focus is on whether the output is produced with the minimum
amount of each input and is called technical efficiency. Efficiency measures that monetize the
inputs (financial inputs) help to answer questions about whether the output could be produced
less expensively—whether the total cost of labor, supplies, and other capital could be reduced. A
focus on cost minimization corresponds to the economic concept of productive efficiency, which
incorporates considerations related to the optimal mix of inputs (e.g., could we substitute nursing
labor for physician labor without changing the amount and quality of the output?) and the total
cost of inputs.
This typology provides a framework within which stakeholders can have an explicit
discussion about the intended use of measures, the choice and measurement of outputs, and the
choice and measurement of inputs. Requesting that groups use a standard format, such as that
suggested by the typology, allows stakeholders to systematically examine what is being
measured and whether the measure (and available data) is appropriate for the purpose.
Evidence Sources and Searches
We searched Medline® and EconLit for articles published between 1990 and 2005 describing
measures of health care efficiency. Titles, abstracts and articles were reviewed by two
independent reviewers, with consensus resolution. We focused on studies reporting efficiency of
U.S. health care, and excluded studies focusing on other countries. Data were abstracted onto
Evidence Tables and also summarized narratively.
Because we expected some of the most commonly used efficiency measures might not
appear in the published literature, we developed a list of organizations that we knew had
developed or were considering developing their own efficiency measures. We contacted key
people at these organizations in an attempt to collect the information necessary to describe and
compare their efficiency measures to others we abstracted from articles.
A Technical Expert Panel (TEP) advised the project staff on the typology and sources of information, and reviewed a draft of this report. The TEP is listed in Appendix D of this report.
Return to Contents
Results
We found little overlap between the peer-reviewed literature that describes the development,
testing, and application of efficiency measures and the vendor-based efficiency metrics that are
most commonly used. From the perspective of policymakers and purchasers, the published
literature provides little guidance for solving current challenges to managing rising health care
costs. From the perspective of measurement experts, the vendor-based metrics are largely
untested and as such the results may be problematic to interpret accurately. These observations
have implications for the recommendations we make at the end of the report regarding future
research.
Published Literature
In total, RAND reviewers examined 4,324 titles for the draft version of this report. Of these, 563 articles were retrieved and reviewed. There were 158 articles describing measures of health care efficiency in the United States.
The majority of peer-reviewed literature on health care efficiency has been related to the
production of hospital care. Of the 158 priority articles abstracted, 93 articles (59%) measured
the efficiency of hospitals. Studies of physician efficiency were second most common (33
articles, 21%), followed by fewer articles on the efficiency of nurses, health plans, other
providers, or other entities. None of the abstracted articles reported the efficiency of health care
at the national level, although two articles examined efficiency in the Medicare program.
Almost all of the measures abstracted from the articles used health services as outputs.
Common health service types used as inputs included inpatient stays, physician visits, and
procedures. Only four measures were found that included health outcomes as outputs. In
addition, none of the outputs explicitly accounted for the quality of service provided. A small
subset of measures attempted to account for quality by including it as an explanatory variable in
a regression model in which efficiency was the dependent variable. Some articles also
conducted analyses of outcomes separately from analyses of efficiency.
The health care efficiency measures abstracted were divided between measures using physical or financial inputs. There were more articles that used physical inputs than financial inputs. No articles were found containing measures of social efficiency.
Most of the measures abstracted from the peer-reviewed literature used econometric or
mathematical programming methodologies for measuring health care efficiency. Two
approaches were most common: data envelopment analysis (DEA) and stochastic frontier
analysis (SFA). DEA is a non-parametric deterministic approach that solves a linear
programming problem in order to define efficient behavior. SFA is a parametric approach that
defines efficient behavior by specifying a stochastic (or probabilistic) model of output and
maximizing the probability of the observed outputs given the model. These techniques can
explicitly account for multiple inputs and multiple outputs. For example, DEA and SFA could
be used to measure the efficiency of hospitals that use nursing labor and supplies to produce
inpatient stays and ambulatory visits. DEA and SFA differ in a number of respects. DEA makes
fewer assumptions than SFA about how inputs are related to outputs. DEA compares the
efficiency of an entity to that of its peers (rather than an absolute benchmark) and typically
ignores statistical noise in the observed relationship between inputs and outputs.
Some measures were ratio-based. Ratios were more common for physician efficiency
measures than hospital efficiency measures. The main difference between the various
measurement approaches is that ratio-based measures include only single inputs and outputs
(although various elements are sometimes aggregated to a single quantity), whereas SFA, DEA,
and regression-based approaches explicitly account for multiple inputs and outputs.
An example of a measure that uses multiple physical inputs and multiple health services
outputs comes from Grosskopf.1 This DEA-based measure used the following inputs (counts): physicians; nurses; other personnel; and hospital beds. As outputs it used (again, counts):
outpatient procedures; inpatient procedures; physician visits in outpatient clinics; hospital
discharges; and emergency visits. In comparison, a typical example of a measure that uses a
single physical input and health services output (ratio) was the number of hospital days (input)
divided by the number of discharges (output)—the average length of stay.2
Vendors and Stakeholder Interviews
Thirteen organizations were selected using a purposive reputational sampling approach. The
results presented here are based on information gathered from eight vendors and five
stakeholders who responded to our request for an interview. The TEP, which included various
stakeholders and experts on efficiency measurement, also provided input into the search and
reviewed this report. The TEP members are listed in Appendix D.
Most of the measures used by purchasers and payers are proprietary. The main application of
these measures by purchasers and plans is to reduce costs through pay-for-performance, tiered
product offerings, public report, and feedback for performance improvement. These measures,
for the purpose of assessing efficiency, generally take the form of a ratio, such as observed-to-expected
ratios of costs per episode of care, adjusting for risk severity and case-mix. Efforts to
validate and test the reliability of these algorithms as tools to create relevant clinical groupings
for comparison are documented in either internal reports or white papers. External evaluations
of performance characteristics of these measures are beginning to emerge from the Medicare
Payment Advisory Commission (MedPAC), the Centers for Medicare and Medicaid Services
(CMS), and other research groups including RAND. Our scan identified seven major developers
of proprietary software packages for measuring efficiency, with other vendors providing
additional analytic tools, solution packages, applications, and consulting services that build on
top of these existing platforms.
The proprietary measures fall into two main categories: episode-based or population-based.
An episode-based approach to measuring efficiency uses diagnosis and procedure codes from
claims/encounter data to construct discrete episodes of care, which are a series of temporally
contiguous health care services related to the treatment of a specific acute illness, a set time
period for the management of a chronic disease, or provided in response to a specific request by
the patient or other relevant entity. On the other hand, a population-based approach to
efficiency measurement classifies a patient population according to morbidity burden in a given
period (e.g., one year).
We contacted a sample of stakeholders to seek their insights on efficiency measurement. We
used their input to cross-validate our selection of vendors described above. Our sample included
two coalitions on the national level; two coalitions on the state level; and an accrediting agency.
We asked these stakeholders to provide the definition of efficiency they used to guide their
efforts; describe desirable attributes they considered as they searched for available measures;
comment on their interest or objectives in developing and/or implementing efficiency measures;
and list proprietary measures they have considered.
While the stakeholders used different definitions of "efficiency," they shared a number of
common concerns related to efficiency measurement. Many concerns were related to
methodological issues such as data quality, attribution of responsibility for care to providers, risk
adjustment, and identification of outliers. The stakeholders also shared a number of concerns
related to the use of efficiency measures, including the perceptions of providers and patients, and
the cost of using proprietary measures and transparency of the methods used to construct the
measures.
Return to Contents
Evaluation
Measures of any construct can rarely be evaluated in the abstract. The evaluation must take into account the purpose or application of the measure; some measures that work well for research, for example, may be unusable for internal quality improvement.
We suggest that measures of health care efficiency be evaluated using the same framework as measures of quality:
- Important—is the measure assessing an aspect of efficiency that is important to
providers, payers, and policymakers? Has the measure been applied at the level of interest to those planning to use the measure? Is there an opportunity for improvement? Is the measure under the control of the provider or health system?
- Scientifically sound—is the measure reliable and reproducible? Does the measure appear
to capture the concept of interest? Is there evidence of face, construct, or predictive validity?
- Feasible—are the data necessary to construct this measure available? Is the cost and
burden of measurement reasonable?
- Actionable—are the results interpretable? Can the intended audience use the information
to make decisions or take action?
An ideal health care efficiency measure does not exist, and therefore the selection of
measures will involve tradeoffs between these criteria. We summarize the results of our review
of measures below.
Important
The measurement of efficiency meets the test of importance because of the interest and intent
among stakeholders in finding and implementing such measures for policy and operations.
Although we found differences in the content of measures from peer-reviewed versus vendor–developed
sources, they have in common the specification of one or more outputs and one or
more inputs in constructing a measure.
The "importance" of measures abstracted from peer–reviewed literature appears low because
these have not generally been used in practice and there is no apparent consensus in the academic
literature of an optimal method for measuring efficiency. Some academic experts have indicated
skepticism that the construct can be adequately measured. Although many peer-reviewed
articles identified factors that were found to influence efficiency, the findings appear to be
difficult to translate into policy. We found no clear evidence that efficiency measures developed
by academics had influenced policy decisions made by providers or policymakers.
The vendor-developed measures meet the importance criterion because they are being widely used by purchasers and plans to inform operational decisions. Some of the vendor-developed measures are based on methods originally developed in the academic world (e.g., Adjusted Clinical Groups).
Scientifically Sound
Very little research on the scientific soundness of efficiency measures has been published to
date. This includes measures developed by vendors as well as those published in the peer–reviewed
literature. Although academics are more likely to publish articles evaluating scientific
soundness, we found little peer–reviewed literature on the reliability and validity of efficiency
measures. Several studies have examined some of the measurement properties of vendor–developed
measures, but the amount of evidence available is still limited at this time. Vendors
typically supply tools (e.g., methods for aggregating claims to construct episodes of care or
methods for aggregating the costs of care for a population) from which measures can be
constructed; thus, the assessment of scientific soundness requires an evaluation of the application
as well as the underlying tools. Significant questions about the scientific soundness of efficiency
measures have been raised. The lack of testing of the scientific soundness of efficiency measures
reflects in part the pressure to develop tools that can be used quickly and with relative ease of
implementation.
Feasible
The focus of vendor–developed measures is on producing tools that are feasible for routine
operational use. Most of the measures abstracted from the peer-reviewed literature were based
on available secondary data sources (i.e., claims data). These measures could feasibly be
reconstructed at little cost and measurement burden. The vendor–developed measures also rely
largely on claims data. Most of the vendor–developed measures require that the user obtain and
pay for a license either directly or through a value added reseller. This has prompted some
organizations to begin developing open–source, public domain measures of efficiency. This
work is at an early stage.
Actionable
For efficiency metrics to have the effects intended by users, the information produced from measures must be actionable. We found little research on the degree to which the intended audiences for these measures (e.g., consumers, physicians, hospitals) were able to readily use the information to choose or deliver care differently.
Return to Contents
Conclusions
We found little overlap between the measures published in the peer-reviewed literature and
those in the grey literature suggesting that the driving forces behind research and practice result
in very different choices of measure. We found gaps in some measurement areas, including: no
established measures of social efficiency, few measures that evaluated health outcomes as the
output, and few measures of providers other than hospitals and physicians.
Efficiency measures have been subjected to relatively few rigorous evaluations of their
performance characteristics, including reliability (over time, by entity), validity, and sensitivity
to methods used. Measurement scientists would prefer that steps be taken to improve these
metrics in the laboratory before implementing them in operational uses. Purchasers and health
plans are willing to use measures without such testing under the belief that the measures will
improve with use.
The lack of consensus among stakeholders in defining and accepting efficiency measures that
motivated this study was evident in the interviews we conducted. An ongoing process to develop
consensus among those demanding and using efficiency measures will likely improve the
products available for use. A major goal of the AQA has been to develop a consensus around
use of language in describing measures of economic constructs. The National Quality Forum is
similarly working to achieve consensus on criteria for evaluating measures. Both groups support
the use of clear language in describing particular metrics, which may be easier to implement than
a consensus definition of efficiency.
Return to Contents
Future Research
Research is already underway to evaluate vendor-developed tools for scientific soundness, feasibility, and actionability. For example, we identified studies being done or funded by the
General Accounting Office, MedPAC, CMS, Department of Labor, Massachusetts Medical
Society, and the Society of Actuaries. A research agenda is needed in this area to build on this
work. We summarize some of the key areas for future research here but do not intend to signal a
prioritization of needed work.
Filling Gaps in Existing Measures
Several stakeholders recognize the importance of using efficiency and effectiveness metrics together but relatively little research has been done on the options for constructing such approaches to measurement. Much of the developmental work currently underway at AQA is focused on this gap.
We found few measures of efficiency that used health outcomes as the output measure.
Physicians and patients are likely to be interested in measures that account for the costs of
producing desirable outcomes. We highlight some of the challenges of doing this that are
parallel to the challenges of using outcomes measures in other accountability applications; thus, a
program of research designed to advance both areas would be welcome.
We found a number of gaps in the availability of efficiency measures within the classification
system of our typology. For example, we found no measures of social efficiency, which might
reflect the choice of U.S.-based research. Nonetheless, such measures may advance discussions
related to equity and resource allocation choices as various cost containment strategies are
evaluated.
Evaluating and Testing Scientific Soundness
There are a variety of methodological questions that should be investigated to better
understand the degree to which efficiency measures are producing reliable and valid information.
Some of the key issues include whether there is enough information to evaluate performance
(e.g., do available sample sizes allow for robust scores to be constructed?); whether the
information is reliable over time and in different purchaser data sets (e.g., does one get the same
result when examining performance in the commercial versus the Medicare market?); methods
for constructing appropriate comparison groups for physicians, hospitals, health plans, markets;
methods for assigning responsibility (attribution) for costs to different entities; and the use of
different methods for assigning prices to services. Remarkably little is known about these
various methodological issues and a program of systematic research to answer these questions is
critical given their increasing use in operational applications.
Evaluating and Improving Feasibility
One area of investigation is the opportunities for creating easy-to-use products based on methods such as DEA or SFA. This would require work to bridge from tools used for academic research to tools that could be used in operational applications.
Another set of investigations is identifying data sources or variables useful for expanding inputs and outputs measured (e.g., measuring capital requirements or investment, accounting for teaching status or charity care).
Making Measures More Actionable
Considerable research needs to be conducted to develop and test tools for decisionmakers to
use for improving health care efficiency (e.g., relative drivers of costs, best practices in efficient
care delivery, feedback and reporting methods) and for making choices among providers and
plans. Research could also identify areas for national focus on reducing waste and inefficiency in
health care. The relative utility of measurement and reporting on efficiency versus other
methods (Toyota’s Lean approach, Six Sigma) could also be worthwhile for setting national
priorities.
Return to Contents
Return to Preface
Proceed to Next Section