Chapter 5: Discussion
Limitations
Publication Bias
Our literature search procedures were extensive and included canvassing experts from
academia, industry, and our peer reviewers regarding studies we may have missed. However, we
can never be sure that we identified all the relevant published literature. We also excluded
studies from non-U.S. data sources, primarily because we judged the studies done on U.S. data
would be most relevant. It is possible, however, that adding the non-U.S. literature would have
identified additional measures of potential interest.
Study Quality
An important limitation common to systematic reviews is the quality of the original studies.
A substantial amount of work has been done to identify criteria in the design and execution of
the studies of the effectiveness of health care interventions, and these criteria are routinely used
in systematic reviews of interventions. However, we are unaware of any such agreed-upon
criteria that assess the design or execution of a study of a health care efficiency measure. We did
evaluate whether or not studies assessed the scientific soundness of their measures (and found
this mostly lacking).
Return to Contents
Conclusions
We found little overlap between the measures published in the peer-reviewed literature and
those in the grey literature suggesting that the driving forces behind research and practice result
in very different choices of measure. We found gaps in some measurement areas including: no
established measures of social efficiency, few measures that evaluated health outcomes as the
output, and few measures of providers other than hospitals and physicians.
Efficiency measures have been subjected to relatively few rigorous evaluations of their
performance characteristics, including reliability (over time, by entity), validity, and sensitivity
to methods used. Measurement scientists would prefer that steps be taken to improve these
metrics in the laboratory before implementing them in operational uses. Purchasers and health
plans are willing to use measures without such testing under the belief that the measures will
improve with use.
The lack of consensus among stakeholders in defining and accepting efficiency measures that
motivated this study remained evident through the interviews we conducted. An ongoing
process to develop consensus among those demanding and using efficiency measures will likely
improve the products available for use.
Return to Contents
Future Research
Research is already underway to evaluate vendor-developed tools for scientific soundness,
feasibility, and actionability. For example, we identified studies being done or funded by the
General Accounting Office, MedPAC, CMS, Department of Labor, Massachusetts Medical
Society, and the Society of Actuaries. A research agenda is needed in this area to build on this
work. We summarize some of the key areas for future research but do not intend the order to
signal any particular priority.
Filling Gaps in Existing Measures
Several stakeholders recognize the importance of using efficiency and effectiveness metrics
together but relatively little research has been done on the options for constructing such
approaches to measurement.
We found few measures of efficiency that used health outcomes as the output measure.
Physicians and patients are likely to be interested in measures that account for the costs of
producing desirable outcomes. We highlight some of the challenges of doing this that are
parallel to the challenges of using outcomes measures in other accountability applications; thus, a
program of research designed to advance both areas would be welcome.
We found a number of gaps in the availability of efficiency measures within the classification
system of our typology. For example, we found no measures of social efficiency, which might
reflect the choice of U.S.-based research. Nonetheless, such measures may advance discussions
related to equity and resource allocation choices as various cost containment strategies are
evaluated.
Evaluating and Testing Scientific Soundness
There are a variety of methodological questions that should be investigated to better
understand the degree to which efficiency measures are producing reliable and valid information.
Some of the key issues include whether there is enough information to evaluate performance
(e.g., sample sizes); whether the information is reliable over time and in different purchaser data
sets (e.g., does one get the same result when examining performance in the commercial versus
the Medicare market?); methods for constructing appropriate comparison groups for physicians,
hospitals, health plans, markets; methods for assigning responsibility (attribution) for costs to
different entities; and the use of different methods for assigning prices to services.
Evaluating and Improving Feasibility
One area of investigation is the opportunities for creating easy-to-use products based on
methods such as DEA or SFA. This would require work to bridge from tools used for academic
research to tools that could be used in operational applications.
Another set of investigations is identifying data sources or variables useful for expanding
inputs and outputs measured (e.g., measuring capital requirements or investment, accounting for
teaching status or charity care).
Making Measures More Actionable
Considerable research needs to be conducted to develop and test tools for decision makers to
use for improving health care efficiency (e.g., relative drivers of costs, best practices in efficient
care delivery, feedback and reporting methods) and for making choices among providers and
plans. Research could also identify areas for national focus on reducing waste and inefficiency in
health care. The relative utility of measurement and reporting on efficiency versus other
methods (Toyota's Lean approach, Six Sigma) could also be worthwhile for setting national
priorities.
Return to Contents
Proceed to Next Section