Chapter 2: Methods

A principal task was to create an analytic framework, or typology, of efficiency. The typology serves two major functions: (1) to provide a structured way to consider the content and use of efficiency measures; (2) to guide our literature review. Additional information can be found in Appendix A.

Analytic Framework–A Typology of Efficiency

We begin with a definition. Efficiency is an attribute of performance that is measured by examining the relationship between a specific product of the health care system (also called an output) and the resources used to create that product (also called inputs). A provider in the health care system (e.g., hospital, physician) would be efficient if it was able to maximize output for a given set of inputs or to minimize inputs used to produce a given output.

Building on this definition, we created a typology of efficiency measures. The purpose of this typology is to make explicit the content and use of a measure of efficiency. Our typology has three levels (go to Figure 1):

• Perspective: who is evaluating the efficiency of what entity and what is their objective?
• Outputs: what type of product is being evaluated?
• Inputs: what resources are used to produce the output?

Considering each of these questions in turn will clarify the intended use of an efficiency measure, the definitions of the key elements, and the validity of the metrics that are proposed for use.

Perspective

The first tier in the typology requires an explicit identification of the entity that is evaluating efficiency, the entity that is being evaluated, and the objective or rationale for the assessment. The diagram illustrates four different types of entities:

• Health care providers (e.g., physicians, hospitals, nursing homes) that deliver health care services.
• Intermediaries (e.g., health plans, employers) who act on behalf of collections of either providers or individuals (and, potentially, their own behalf) but do not directly deliver health care services.
• Consumers/patients who use health care services.
• Society, which encompasses the first three.

Each of these types of entities has different objectives for considering efficiency, has control over a particular set of resources or inputs, and may seek to deliver or purchase a different set of products. Health care providers can act directly to change the way in which different products are produced whereas intermediaries can typically just change how much is paid or what will be purchased. Individuals often operate through intermediaries to access the products delivered by providers; they are two steps removed from the production process of the health care providers. (High deductible health plans are designed in part to make consumers more cost conscious in their decisionmaking.) Society as a whole includes the perspectives of all providers, intermediaries and consumers. Efficiency for society as a whole, or "social efficiency," refers to the allocation of available resources; social efficiency is achieved when it is not possible to make a person or group in society better off without making another person or group worse off. Thus, perspective is the lens through which an entity views efficiency; entities will select measures that reflect their objectives, the outputs of interest, and the inputs necessary to produce those outputs.

Performance on efficiency may be evaluated internally by a firm (could we perform better?) or be externally driven by agents and individuals (could we get a better deal?). Stating the purpose or intended use of the results of an evaluation is critical for evaluating the utility and appropriateness of measures. The requirements for conducting a fair internal evaluation are often less demanding than those for an external evaluation.

The perspective from which efficiency is evaluated has strong implications for the measurement approach, because what is efficient from one perspective may not be efficient from another. For example, a physician may produce CT scans efficiently in her office, but the physician may not appear efficient to a health plan if a less expensive diagnostic could have been substituted for some cases. We will illustrate how these different perspectives operate in the examples provided later.

Outputs

Efficiency measures should explicitly identify the outputs of interest and how those will be measured. We distinguish between two types of outputs: health services (e.g., visits, drugs, admissions) and health outcomes (e.g., preventable deaths, functional status, blood pressure control). Both represent reasonable ways of defining the products of the health care system. Health care services can be considered an intermediate output in the production of health outcomes.⁷

Health service outputs can be measured in a variety of ways:

• Individual units of service (e.g., procedures, prescriptions).
• Bundles of services within a single entity (e.g., hospital stay).
• Bundles of related services provided by one or more entities (e.g., episodes of care).

In this typology, we do not require that the health service outputs be constructed as quality/effectiveness metrics. For example, an efficiency measure could consider the relative cost of a procedure without evaluating whether the use of the procedure was appropriate.

Similarly, an efficiency measure could evaluate the relative cost of a hospital stay for a condition without considering whether the admission was preventable or appropriate. However, the typology allows for health service outputs to be defined with reference to quality criteria. That is, the typology is broad enough to include either definition of health services. We deliberately constructed the typology in this way to facilitate dialogue among stakeholders with different perspectives on this issue.

More recently, suggestions have been made about incorporating other quality measures (i.e., effectiveness of care or patient experience) into efficiency assessments. The AQA, a consortium of physician professional groups, insurance plans, and others, has adopted a principle that measures can only be labeled "efficiency of care" if they incorporate a quality metric; those without quality incorporated are labeled "cost of care" measures.⁸ The AQA has noted the potential unintended consequences of measurement focusing solely on one dimension or the other of quality.

The methods for incorporating quality into efficiency measurement are not well developed at this time, however. The most common (and simplest) approach has been to perform comparisons of quality measures alongside comparisons of efficiency measures for the same provider, medical condition treated, or procedure. For example, blood glucose monitoring frequency for diabetic patients could be reported in conjunction with the cost of an episode of diabetes care. Another (more difficult) approach would be to adjust the outputs of efficiency measures for quality by directly incorporating quality metrics into the specification of the output. The method would be analogous to how quality-adjusted life years (QALYs) weight years of life using a health-related quality of life scale. For example, comparisons of the efficiency of producing coronary artery bypass graft surgical procedures would give less weight to procedures resulting in complications. This approach poses significant methodological challenges and is not well-developed at this time.

Health outcome outputs may include health status at a point in time, changes in health status over a period of time, or changes in health status associated with a particular intervention (e.g., mortality following surgery). The use of health outcomes measures as outputs more directly incorporates quality metrics into efficiency measurement. For many clinicians, this information is important for assessing whether efficient patterns of care (e.g., relatively high rates of generic drugs used for treatment of hypertension) are also effective (e.g., as measured by the proportion of the population with good blood pressure control).

A number of methodological challenges arise in using health outcomes as outputs, including: defining the time period for evaluation (i.e., whether the time frame for costs and outcomes must be identical), identifying the responsible entities, taking account of the role of individuals in "producing" their own health outcomes, adjusting for the expected trajectory of the patient (particularly for outputs measured over a longer period of time), and accounting for factors outside the scope of the health delivery system (e.g., air pollution, education). For many of the same reasons that outcomes measures can be challenging to develop for quality measurement, they are likely to be challenging to use in evaluating efficiency. Although we focus in this section on health outcomes, an extension of this typology could include customer satisfaction.

The approach to measurement may be influenced by the way in which the output is purchased. For example, if physician services are paid fee-for-service, then the purchaser may consider an evaluation of efficiency at the service unit level. If a hospital is paid for a bundle of services, such as under the Medicare inpatient prospective payment system, then the purchaser may be more likely to evaluate efficiency for the bundle. Thus, the perspective of the evaluator may be shaped by the way in which the outputs of interest are paid for.

A key issue that arises in external evaluations of efficiency is whether the outputs are comparable. Threats to comparability arise when there is (perceived or real) heterogeneity in the content of a single service, the mix of services in a bundle, and the mix of patients seeking or receiving services. Thus, one way to evaluate efficiency measures is by determining whether the methods used truly allow for apples-to-apples comparisons. Some of the methods used today include peer-to-peer comparisons (e.g., by specialty for physicians, by bed size and/or location for hospitals, by profit vs. non-profit status for health plans), geographical controls, case-mix or severity adjustments for heterogeneity among patients, and consistent inclusion/exclusion criteria for constructing bundles of services. Whether or not these approaches adequately define comparable groups is an ongoing area for research. For example, a common way to identify physician peer groups is by specialty but this fails to account for the heterogeneity of practice within specialty (e.g., cardiologists who specialize in electrophysiology versus those with a general practice). Suggestions have been made to define peer groups empirically on the basis of patterns of practice but these approaches have not been fully developed or tested.

Inputs

Efficiency measures must also explicitly identify the inputs that are used (or will be counted) to produce the output of interest. Inputs can be measured as counts by type (e.g., nursing hours, bed days, days supply of drugs) or they can be monetized (real or standardized dollars assigned to each unit). We refer to these, respectively, as physical inputs or financial inputs. The measurement objectives should guide the method for measuring inputs.

Efficiency measures that count the amounts of different inputs used to produce an output (physical inputs) help to answer questions about whether the output could be produced faster, with fewer people, less time from people, or fewer supplies. In economic terms, the focus is on whether the output is produced with the minimum amount of each input and is called technical efficiency.

Efficiency measures that monetize the inputs (financial inputs) help to answer questions about whether the output could be produced less expensively-could the total cost of labor, supplies, and other capital be reduced? A focus on cost minimization corresponds to the economic concept of productive efficiency, which incorporates considerations related to the optimal mix of inputs (e.g., could we substitute nursing labor for physician labor without changing the amount and quality of the output?) and the total cost of inputs.

Questions similar to those discussed in the section on outputs have been raised regarding the comparability of inputs. For example, the method of paying physicians or other providers (e.g., fee-for-service versus capitation) may affect the comparability of the input costs. The allocation of dollars across services can vary considerably depending on the cost structure of the medical group, hospital or physician practice. For this reason, many users have elected to create standardized prices based on fee schedules or some other method that are applied to utilization patterns to remove variable pricing or differential cost structures from an evaluation.

An Example

To make the typology more concrete, we offer a simple example shown in Table 2. Let's assume that a health plan has decided to create a tiered network where patients who see physicians in the top tier pay 20 percent of charges and patients who see physicians in the bottom tier pay 50 percent of charges. Physicians are assigned to tiers based on efficiency metrics. The health plan is evaluating the performance of physicians with the objective of steering patients to the physicians who produce cataract surgeries at the lowest charge. In the example, the outputs are identical whether the health plan examines services or outcomes; each physician performs the same number of procedures per day with identical patient outcomes and satisfaction.

Table 2. An example of efficiency measures where outputs are identical

Type	MD1	MD2	MD3
Input (per procedure)
MD Labor	15 minutes	20 minutes	15 minutes
MD Cost	$100/hour	$100/hour	$100/hour
RN Labor	60 minutes	45 minutes	45 minutes
RN Cost	$40/hour	$40/hour	$40/hour
Anesthesia dose	40cc	40cc	40cc
Anesthesia cost	$0.10/cc	$0.10/cc	$0.05/cc
Total input cost	$69	$67	$57
Total charge	$80	$75	$65
Output (total)
Cataract Surgeries	8/day	8/day	8/day
Visual Functioning	+10 points	+10 points	+10 points
Patient Experience	89	89	89

From the health plan's perspective, the relevant input is the charge for the service, so MD3 would be rated highest followed by MD2 and then MD1. From the perspective of the physicians (e.g., internal evaluations of their practices' efficiency), the total input cost may be a more relevant metric than the total charge. If they could successfully lower costs at a given level of charges, they could increase their practices' profits. The relationship between costs and charges is not constant, so that a physician with the lowest total costs could possibly also have the highest total charges. The physician would be the most efficient from her perspective, but least efficient from the health plan's perspective. For example, a physician practicing in a region with only one major health plan may be less able to negotiate favorable payment rates than a physician practicing in an area with heavy competition between health plans.

The different results that would be obtained by using total costs instead of total charges as the relevant input in an efficiency metric underscores the importance of perspective in efficiency measurement. In this case, one way to control for market differences in physicians' charges would be to use standardized prices. In cases where standardized prices are used, the measure reflects a mixture of efficiency of physical inputs (technical efficiency), efficiency of financial inputs (productive efficiency), and some degree of measurement error.

If we examine physical inputs, we note that MD2's labor time is longer than MD1 or MD3 whereas MD1's nursing labor hours are longer than those of MD2 and MD3. From the physical input perspective, MD3 has the most efficient practice (least amount of physician and nursing time and no more of any other inputs). If MD2 could reduce his labor time without changing the number of procedures performed or his results, he was operating inefficiently. On the other hand, if reducing his time would reduce either the volume of procedures or the outcomes, his practice was operating efficiently. Similarly, if MD1 could reduce his nursing time without sacrificing quantity or quality of service, he was operating inefficiently. All physicians use the same amount of anesthetic (the only supply in this example); on the basis of physical inputs, no physician uses this input less efficiently. But, examining inputs from a financial perspective, MD3 would be more efficient because his use of a generic anesthetic gives him the lowest total input cost. Note that in this example we have not assumed substitution across inputs, but in many real world circumstances this would be another way to achieve efficiency.

In Table 3 we provide a variation on the preceding example to illustrate the real-world challenge of making comparisons when the outputs vary. The number of cataract surgeries and outcomes now differ between MD1 and MD2. MD2 produces more procedures but with a lower visual functioning score and a lower patient satisfaction score. To compare their efficiency, we would need a model to tell us what MD2's physical or financial efficiency would have been if his outputs were adjusted to equal those of MD1 (or vice versa). One approach would be to combine the outputs into a single output measure (e.g., a procedure count that is weighted for visual functioning and patient experience). More complex methodologies, including regression analysis, data envelopment analysis (DEA), and stochastic frontier analysis (SFA), can be used to model efficiency using multiple inputs and outputs. The various methods are described in Box 1 in Chapter 3. In any event, all of these methods face the challenge that some inputs or outputs may be difficult to measure, raising potential concerns about the usefulness or fairness of their results.

Table 3. An example of efficiency measures where outputs vary

Type	MD1	MD2
Input (per procedure)
MD Labor	15 minutes	20 minutes
MD Cost	$100/hour	$100/hour
RN Labor	60 minutes	45 minutes
RN Cost	$40/hour	$40/hour
Anesthesia dose	40cc	40cc
Anesthesia cost	$0.10/cc	$0.10/cc
Total input cost	$69	$67
Total charge	$80	$75
Output (total)
Cataract Surgeries	8/day	10/day
Visual Functioning	+10 points	+8 points
Patient Experience	89	80

Return to Contents

Applying the Typology

In this section, we use some general examples of approaches to measuring efficiency (or measures that have been labeled as efficiency measures) to show how the typology can be applied and what questions might arise in doing so. The purpose of this section is to illustrate at a high level the identification of perspective (including objective), outputs, and inputs so that one can identify the issues that might arise in drawing conclusions from the metric sufficient to drive action on the objective.

Table 4 summarizes how seven common efficiency measurement approaches fit into our typology. We describe these measures at a very general level in order to highlight some features of the typology. A key consideration is the tradeoff between broad measures that are heterogeneous or narrow measures that are more homogeneous. The advantage of broad composite measures is the ability to acquire a large enough number of observations to construct a robust measure, however, the presence of heterogeneity increases the need for case-mix adjustment. Narrower measures may have fewer problems with heterogeneity but may suffer from small sample sizes.

Table 4. Some common approaches to efficiency measurement

Metric	Perspective	Outputs	Inputs
Cost per episode	Health plan as evaluator Physicians evaluated Objective: reduce costs	Bundle of health services related to care for a condition, procedure, event	Monetized total cost
Cost per discharge	Health plan as evaluator Hospitals evaluated Objective: reduce costs	Bundle of health services used to treat patients while in the hospital	Monetized total cost
Cost per covered life	Employer as evaluator Health plan evaluated Objective: set premium prices	# of employees with health insurance, by type	Premium price charged by health plan
Cost per health improvement	Medicare as evaluator Health plans evaluated Objective: maximize production of healthy lives	Change in functional status	Total costs of care
Labor utilization	Hospital as evaluator Hospital evaluated (internal) Objective: optimize labor mix	Total number of discharges	Total number of nursing hours by level of training
Productivity	Physician as evaluator Physician practice evaluated (internal) Objective: maximize output	Number of patients seen in time period	Number of physician hours in patient care
Generic prescribing rate	Health plan as evaluator Physicians evaluated Objective: minimize medication costs	Number of days of medication supplied (total)	Number of days of generic medication supplied (total)

Cost per Episode

Episode methods are being used primarily by health plans or employers to identify variations in the amount of money spent on patients with similar health problems with the objective of reducing costs. Currently, employers and health plans are evaluating the performance of physicians and physician groups. The output in this case is in the health service category and includes a bundle of services (e.g., visits, medications, procedures, urgent care services) that are associated with care for a particular condition, procedure, or event. The inputs in this example are financial, usually expressed as monetized total costs. The usual application of this measure is to examine average costs per episode among physicians in the same specialty and identify those whose costs are higher than average. Thus, the evaluation of efficiency is relative and the question being asked is whether care could be delivered less expensively; variation in costs for like episodes provides the evidence that this is possible. The major threats to the validity of the comparison are whether the episodes are comparable across the entities being evaluated (similar quality, patient risk, etc.) and whether the attribution of responsibility for the cost of the episode is made properly.

Cost per Discharge

Health plans may evaluate the average cost they pay different hospitals for a discharge with the objective of reducing costs. In this case, the output is the bundle of services used to treat patients while in the hospital (health service) and the input is the price paid by the health plan for the discharge (financial input). If the health plan is undertaking this evaluation alone, the price paid will reflect any discounts that have been negotiated with hospitals. Because this is an external evaluation, the actual costs to the hospital of producing the discharge are not part of the assessment. If the discharges are bundled together (average cost across all discharges), then the measure can be affected by the mix of types of discharges, which might vary by hospital.

Cost per Covered Life

Employers may evaluate the costs of providing different types of health insurance coverage for their employees and dependents with the intent of minimizing the total cost of labor as an input to their own production processes. The output in this example is the number of health plan enrollees and the input is the premium price charged by the health plan. If the benefit package across plans is identical, the employer might conclude that lower premium prices signal greater efficiency. Large national employers may have some difficulty accounting for differences in market prices and state mandated benefit packages that affect the actuarial value of the package and thus the premium charged.

Cost per Health Improvement

Medicare could evaluate the health improvement and costs of beneficiaries enrolled in private Medicare Advantage health plans, potentially comparing the efficiency of the plans to traditional fee-for-service Medicare, with the objective of maximizing the health of the Medicare population at current funding levels. The output in this example is the change in physical functioning of beneficiaries over a given period of time and the input is the amount Medicare spends for those beneficiaries over the time period. One measurement challenge would be to ensure that the Medicare beneficiaries in the different comparison groups were similar.

Labor Utilization

Hospitals may evaluate their use of nursing labor to produce discharges with the intent of minimizing labor costs (generally the largest component of hospital costs). The output in this case would be discharges (health service) and the input would be the total number of nurse labor hours or days used (physical). From an efficiency perspective, the question is whether the same number of discharges could be produced with fewer nursing hours (with the caveat that the results would have to be the same, say functional status at discharge). Another use of such measures would be to consider whether a different mix (by level of training) of nursing hours could produce equivalent outcomes.

Productivity

Physicians are frequently paid based on their productivity so a physician practice may conduct an internal evaluation of productivity for the purpose of maximizing reimbursement. The output in this case would be the number of visits (health service) and the input would be the total number of hours the physician spent in patient care (physical). A challenge to this measure is whether the visits (output) are equivalent across different levels of physician labor hours. Substituting less costly labor (e.g., nursing time) for physician time offers one approach to improving efficiency on this metric.

Generic Prescribing Rate

To minimize the amount spent on prescription drugs, some large purchasers are measuring generic prescribing rates at the health plan or physician level. The output in this case is a health service (total days supply of a medication) and the input is a physical input (total days supply of generic medications). This measure focuses on a narrow set of outputs and inputs (prescription drugs), omitting other aspects of care delivered. The bases of the measure are the dual assumptions that (1) the output is identical regardless of whether generic or brand name drugs are prescribed; (2) generics are always less expensive, implying that a higher ratio of generic to brand name drugs is preferable; and (3) availability of generic substitutes is consistent across conditions. As with most rate measures, the preferred proportions are often unknown. A number of factors could influence the metric including reductions in the total days supplied of medications (the optimal number is likely not known).

Return to Contents

Approaches That We Do Not Classify as Efficiency Measures

Table 5 presents three approaches, readmission, procedure rates, and cost-effectiveness, that have been used to measure "efficiency" but would not be classified as efficiency measures under our definition and typology. Although they may indirectly reflect the efficiency of health care providers and may be useful for evaluating other problems in practice patterns, they do not directly measure efficiency by comparing the inputs and outputs of health care or otherwise are not appropriate for this application.

Table 5. Measures we would not classify as efficiency measures

Metric	Perspective	Outputs	Inputs
Readmission rate	Purchaser as evaluator Hospitals evaluated Objective: change reimbursement method	Not specified	Not specified
Rate of CABG surgery	Employer as evaluator Health plans evaluated Objective: minimize costs	Total number of CABG procedures	Not specified
Cost-effectiveness	Any perspective Objective: minimize costs	Change in outcome	Change in cost of producing outcome

Readmissions

Large purchasers such as Medicare have used readmissions (admission to a hospital for the same diagnosis within a short time period following a discharge) as a measure of efficiency. While readmissions are certainly a signal of a quality problem (for example, premature discharge) and represent a cost to Medicare, it is less clear how they can be used as an efficiency measure. Neither the output nor the input is clearly specified and the readmission itself is only one sign of a problem (death prior to readmission or admission to an urgent care or other facility being two other examples). In our typology, these measures would not be included as efficiency measures.

Procedure Rates

To minimize costs, purchasers have requested information from health plans on the rates at which certain high cost procedures (e.g., coronary artery bypass graft surgery) are performed. The rate may be constructed within age groups in the population. The surgical procedure rate could be considered the output, but no inputs are specified. Purchasers may intend to interpret higher rates as being a sign of an efficiency problem. Alternatively, purchasers may interpret higher rates as indicators of economies of scale. The rate, by itself, is difficult to interpret. Previous work has shown no relationship between the rates at which procedures are performed and the proportion of such procedures that are clinically inappropriate.⁹

Cost-Effectiveness

We specifically did not include cost-effectiveness as a type of efficiency measure. The methods for assessing cost-effectiveness typically answer the question-is this technology a good value relative to the alternatives? The judgment is made by reference to a standard threshold such as $200,000 spent per life year saved and the evaluation is generally done on a narrow question (procedure A versus medication B for condition C) for a particular setting or set of assumptions. The answer may be monetary (the dollars spent per life year saved) or dichotomous (yes or no). But within that general framework the analysis does not produce information about whether one institution or provider does the intervention more efficiently than another. The results of a CEA analysis could be used to construct an efficiency measure, for example, "the proportion of people with an episode of care for condition C who are treated with procedure A instead of medication B" where the input is the use of either procedure A or medication B and the output is an episode of care for condition C, and the specifications for A, B, and C are all defined. We did not believe that an assessment of the large literature on CEA would provide new measurement tools.

Return to Contents
Return to Previous Section
Proceed to Next Section