Key Concepts in Evaluating Outcomes of ATP Funding of Medical Technologies

Andrew J. Wang
Advanced Technology Program, National Institute of Standards and Technology
Technology Administration, U.S. Department of Commerce
Gaithersburg, MD 20899

Abstract

The Economic Assessment Office (EAO) of the Advanced Technology Program (ATP) commissions economic studies by outside research organizations. A recent study by the Research Triangle Institute (RTI) outlines a framework for evaluating ATP funding of medical technologies. The RTI study employs a standard approach to cost-benefit analysis, while utilizing a number of methods unique to health care assessment to measure the benefits brought about by the new medical technologies. This article highlights important analytical points and explicates key concepts forming the basis of understanding for the approach. Concepts of economic returns, such as private and social returns, and the returns to public investment, are discussed in the framework of cost-benefit analysis and net present value. For analyzing the benefits of new medical technology, methods from health care assessment and concepts such as QALYs and the statistical value of life are identified and explained.

Introduction

As part of its effort at economic evaluation and research, the Economic Assessment Office (EAO) of the Advanced Technology Program (ATP) commissions economic studies conducted by outside research organizations and economists. One recent study, completed by the Center for Economic Research at the Research Triangle Institute (RTI), is described in the research report A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies  (RTI 1998). This report concludes an evaluation effort conducted by RTI for EAO, focusing on a set of ATP-funded projects in the developing medical technology field of tissue engineering.

The RTI study examined a set of seven ATP projects in tissue engineering with an aim to developing and demonstrating an evaluative framework that would enable the estimation of economic benefits from ATP funding of new medical technologies. In general terms, the overall framework of analysis can be considered to be cost-benefit analysis. But beyond the general framework, the substantive issue of interest is what are the details and specifics of how the costs and benefits are estimated and analyzed, and what is the framework for estimating and analyzing costs and benefits associated with an investment. For evaluating the impact of new medical technologies, the key question is how to evaluate health benefits to patients that are brought about by medical innovation.

This article will outline the general methodology taken in the RTI study, and will highlight and explicate the most important conceptual points critical to a proper understanding of the methodology. Section II defines and clarifies concepts of economic returns. Section III identifies social benefits from new technologies and explains how public investment may affect overall returns. Section IV explains the key concepts of quality-adjusted life year (QALY) and "value of life" that are central to the evaluation of benefits associated with new health care or medical technologies. Section V provides a brief conclusion.

Concepts of Economic Returns

Net Present Value

The correct measure of the economic value of an investment that produces a stream of benefits is the net present value. Net present value (NPV) is defined as the "discounted" present value of all net benefits associated with an investment project. Net benefits NBt in any time period t is total benefits minus total costs in time period t. Net benefits that occur in later periods are "discounted" to make them comparable with net benefits in the present. The net present value of an investment project is then defined as the sum over all time periods of discounted net benefits.

The rationale for discounting net benefits that occur in the future is straightforward. A dollar today is not the same as a dollar tomorrow. A dollar today can be invested at a rate of return r, and is therefore worth (1+r) dollars tomorrow. More generally, a dollar in the present, invested at a rate of return of r per period, is equivalent to (1+r)^t in a future period t. Similarly, a dollar in a future period t is equivalent to 1/(1+r)^t dollars in the present period, and so the discounted value, or present value, of net benefit NB_t occurring in time period t is equal to NB_t/(1+r)^t.

Net present value can be expressed as

(1) ,
Click here for full size version of equation.

where NBt is net benefits in period t, and r is the discount rate. The discount rate r is specified to be the currently prevailing rate at which dollars today are traded for dollars tomorrow--it is the intertemporal price for dollars traded between time periods, which is the "opportunity cost" for funds available for either investment or consumption. Equation (1) shows that costs and benefits in all periods are first translated into net present value terms, using the discount rate r, and then the separate net benefit terms are summed to give the value of the project.

It is important to note that the NPV calculation already accounts for the best alternative use of funds by discounting net benefits at discount rate r. Thus, any project with an NPV>0, by definition, has economic value greater than prevailing economic opportunities. In other words, if NPV = $x is greater than zero, then the project is worth $x more than existing economic opportunities. The key point here is that a project with an NPV>0 is better than existing economic opportunities and is therefore worth doing.

Private Returns and Social Returns

Consider a private-sector investment decision. In calculating the NPV of an investment project, the investor will only take into account costs incurred by the investor and benefits that accrue to the investor. For a business company, these would be company costs and revenues associated with an investment project. This NPV calculation gives the private value of the investment to the private investor--it is the private return for the investment.

But when a private investor undertakes an investment, costs may be imposed on others as well, and benefits may also accrue to others. Consequences to others (i.e., costs and benefits), resulting from a given private action, are called externalities, or spillover effects. An NPV calculation that takes into account costs and benefits not only to the investor, but also to all others affected, gives the social value of the investment--it is the social return for the investment. The calculation for social return includes not only private costs and benefits to the private investor, but also all positive and negative externalities that affect others in society. Net benefits in the calculation of social return therefore includes "private" net benefits as a component of "social" net benefits.

Incremental Social Return on Public Investment

The social value of a given public-sector investment can be derived from a comparison of scenarios with and without the public investment. For concreteness, take the ATP to be the public investment program under consideration. Without ATP, some level of R&D investment is undertaken by the private sector, and the net present value of social benefits and social costs defines the social return associated with the private investment.

With ATP, a different level of R&D investment may be undertaken by the private and public sectors together, and the net present value of all social benefits and costs (including the public investment costs) defines the social return associated with the combined private and public investment.

In each time period t, incremental social net benefit can be defined to be the difference between social net benefit with ATP and social net benefit without ATP. The net present value of this stream of incremental social net benefits defines the "incremental social return on public investment." Figure 1 shows that social return and private return can each be defined for a withATP scenario and a withoutATP scenario. The incremental social return on public investment is based on a comparison between social return with ATP and social return without ATP, that is, a comparison between cell A and cell C in Figure 1.

Social Returns and the Impact of ATP

Social Returns from New Technologies

As discussed above, social returns from investment can differ from private returns because of externalities, or spillover effects. Three classes of spillovers have previously been identified to ATP as relevant for evaluating the social benefits of new technologies (Jaffe 1996). First, there are market spillovers, which are benefits consumers receive that are greater than what they pay for, what economists call "consumer surplus." A consumer purchasing a good usually places a higher value on the item than the price paid this excess of benefit or value over price is consumer surplus and is a market spillover benefit to the consumer. Second, there are knowledge spillovers, or benefits to other firms doing R&D that are able to learn from the R&D and innovation of a given firm. Third, there are network spillovers, or benefits to firms or consumers that stem from complementarities or interaction effects that may characterize a set of related technologies.

In the RTI study, the main focus in the estimation of social net benefits is the measurement of health benefits resulting from the application of new medical technologies. For the most part, these health benefits to individuals can be categorized as market spillovers to consumers. The RTI effort does not attempt to measure social benefits from knowledge spillovers or network spillovers.

Estimating Net Benefits of New Medical Technologies

Net benefits of new medical technology innovation may be disaggregated for analytical purposes into net benefits occurring in the health care delivery sector, and net benefits occurring in the medical technology sector. For the health care delivery sector, RTI's estimation of net benefits of new medical technology incorporates a measure of the value of health benefits to patients brought about by the medical innovation, and an estimate of changes in the costs of medical treatment due to the new technology. For the medical technology sector, net benefits include revenues resulting from sales of new medical technology products, less all investment and production costs associated with bringing the new technology to market (e.g., R&D investment, physical capital investment, commercialization costs, actual production costs).

In short, net benefit in the health care delivery sector is consumer surplus benefit to individuals--the value of increased health benefits to patients, and the decrease in medical treatment costs. And net benefit in the medical technology sector is total revenues minus total costs, or total profits generated in this sector. More accurately, net benefit is equal to the change in total profits in the sector, after revenues and costs for the old displaced technologies are properly accounted for. Overall, total social net benefit is equal to the sum of consumer surplus benefit from the health care delivery sector, and the change in total profits from the medical technology sector.

The Impact of ATP on Social Returns

The rationale for a public investment program such as the ATP is that public investment can have a significant impact on social returns. As outlined above, the incremental social return on public investment is based on a comparison of social returns for scenarios with and without the particular public investment under consideration. For evaluating the impact of ATP investment, RTI identifies three ways in which ATP funding may increase social returns:

• Acceleration of R&D
• Increase in probability of R&D success
• Expansion of scope of R&D.

These three channels of possible ATP impact on the social return to R&D investment are discussed in turn

1. ATP funding may accelerate R&D and thus lead to earlier introduction of the new technology. In the NPV calculation, more years of net benefits or earlier years of net benefits will tend to increase project NPV.
2. ATP funding may increase the intensity of R&D, which may lead to a higher probability of R&D success. When project success is uncertain, all net benefits for the project are expressed in terms of "expected value" in the NPV calculation. The expected value of net benefits for any given year is defined to be the probability that benefits will occur (i.e., the probability of project success), multiplied by the value of net benefits in the case of project success. Therefore, if ATP funding increases the probability of project success, the expected value of all net benefits will increase, and project NPV will tend to be higher.
3. ATP funding may broaden the scope of R&D to include a wider range of potential applications. With wider application of the technology, a greater number of patients will receive health benefits, and the increase in net benefits will tend to raise project NPV.

Valuing Health Benefits

Quality Adjusted Life Years (QALYs)

As discussed earlier, in estimating net benefits of medical technologies, a central focus is measuring the health benefits to patients. The concept of Quality Adjusted Life Year (QALY) was developed to make possible a quantification of health benefits to an individual in terms of the quantity and the quality of life (Torrance & Feeny 1989). The basic idea is that one year of life at less than full health can be considered equivalent to less than one year of life in full health. So a year of life in full health will be given a QALY value of 1.0, a year of life at less than full health will be given a QALY value between 0.0 and 1.0, and death will have a QALY value of 0.0.

Using QALY values assigned to health states, it is possible to quantify health improvements for an individual, as well as to aggregate health benefits across different individuals. A medical intervention that extends the life of person A by one year at an existing QALY level of 0.50, and produces a health improvement in person B equal to 0.25 QALY for one year, is considered to have produced 0.75 QALY in health benefits in total. Note that a 1 QALY gain to a young person is considered equivalent to a 1 QALY gain to an old person, and a gain of 0.50 QALY for one person is equivalent to gains of 0.25 QALY for two people. The concept of QALY is understood as a means to quantifying health outcomes, a method of accounting for years of life and quality of life in a single measure.

Where do QALYs come from? QALY values assigned to different health states are derived from surveys of individuals from a relevant population. Individual survey responses are personal subjective judgments on the quality of life in various health states. A population "average" response is taken to define the QALY values for health states. QALY values are therefore reflective of the population on which they are defined, and are not meant to be viewed as fundamental or immutable characteristics.

Various survey methods have been used to elicit QALY values from individuals. The "standard gamble" method is a conventional method often used that is considered to be most theoretically consistent. Figure 2 illustrates the standard gamble method for eliciting the QALY value for a health state A. The individual surveyed is presented with a choice between Choice A (health state A with certainty), and Choice B (a lottery or gamble on full health with probability p and immediate death with probability 1_p). At some level of probability p, the individual surveyed will express indifference between Choice A and Choice B. That level of probability p defines the subjective QALY value for health state A, as expressed by the particular individual surveyed. As already mentioned, the QALY value for the population as a whole will be taken to be some average of the individual personal subjective QALY values. Table 1 shows QALY values for selected health states that have been reported in the health assessment literature.

Value of Life, or the Pricing of Fatality Risk

Perhaps the most controversial, or most "controversial sounding," aspect of cost-benefit analysis is the idea of placing a value on life. As is often the case, what may seem to be controversial at first, turns out to be not so controversial, once ideas and language are clarified. In fact, in this case, the "value of life" is defined to be the value of a "statistical" life, which is itself just a measure of the observed price of fatality risks (Viscusi 1992).

The idea of the "statistical" life is based on the observation that a 1 in 100,000 risk of death to an individual is equivalent in statistical terms to 1 death in a society or community of 100,000 people. What the community is willing to pay collectively, to reduce deaths in the community by 1, is an appropriate measure of the value that society places on one life, or one "statistical" life. And what a community of 100,000 is willing to pay in aggregate, for a reduction in deaths by 1, is just equal to what a typical person in the community is willing to pay for a 1 in 100,000 reduction in the risk of individual death, multiplied by the number of people in the community, or 100,000. If, for example, each person in the community is willing to pay $50 for a 1 in 100,000 reduction in individual death risk,then the value of a statistical life in this community is $50 ´ 100,000 = $5 million.

The value of statistical life is then a reflection of individual willingness to pay for very small reductions in very small risks of death. A $5 million value of statistical life does not imply that an individual would be willing to accept certain death for $5 million, or a 0.50 increase in individual death risk for a payment of $2.5 million. For the same reason, the value of statistical life can be much greater than a person's total lifetime earnings potential. A person with total lifetime earnings well below $5 million may indeed be willing to pay $50 for a 1 in 100,000 reduction in death risk, implicitly valuing statistical life at $5 million.

Estimates of value of statistical life can be derived from observed data on the willingness to pay for reductions in fatality risks. Most of the empirical studies use labor market data to derive estimates of the wage premium (also known as the "compensating differential") associated with higher-risk jobs. The observed pricing of fatality risk in the labor market provides a good measure of the value of statistical life.

The empirical studies of the value of life based on labor market data place the value of life of workers in typical jobs in the range of $3 to $7 million. Like QALY values, the value of life must be properly understood as an "average" value for a given relevant population. For some classes of workers, the price on fatality risk, and consequently the implied value of life, is higher; for other groups of workers, the price of risk and the implied value of life will be lower. The purpose of using QALYs and estimates of value of life is to allow for a common basis of measurement. To the extent that QALY values and the estimated value of life are based on a population that is relevant to the analysis, the measurement function is served.

Monetary Value of QALYs

QALY values need to be converted to monetary terms for the purpose of analysis of net benefits and calculating economic returns. If the estimated value of life is taken to be $5 million, and this estimate is for an average worker at age 40 in full health with 36 years of remaining life expectancy, then the money value of a QALY might be calculated as $5 million ÷ 36 years = $139,000. But as explained earlier in the discussion of NPV, a dollar tomorrow is worth only 1/(1+r) today, and a dollar at time period t is worth only 1/(1+r)^t in the present, given the discount rate r. Thus, for 36 years of life in full health to be valued at $5 million today, the money value of each year of life must be equal to v defined by the equation

(2) .
Click here for full size version of equation.

Or rearranging, the money value of a QALY is given as

(3) .
Click here for full size version of equation.

For example, if the discount rate r is 0.03, then the money value of a QALY is v = $222,000.

Conclusion

In developing a framework for evaluating ATP funding of medical technologies, the RTI study employs a standard approach to cost-benefit analysis, while utilizing a number of methods unique to health care assessment to measure the benefits brought about by the new medical technologies. This article highlights important analytical points and explicates more fully key concepts that form the basis of understanding for the approach. Specifically, concepts of economic returns such as private and social returns, and returns to public investment, are discussed in the framework of cost-benefit analysis and net present value. And for analysis of benefits of new medical technology, methods from health care assessment and concepts such as QALYs and statistical value of life are identified and explained. Overall, the basic framework of evaluation presented by RTI in their study report is well grounded in established economics theory and is widely accepted in practice.

References

Jaffe, Adam B. Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program. Prepared for Advanced Technology Program, National Institute of Standards and Technology, U.S. Department of Commerce. October 1996.

Research Triangle Institute (RTI). A Framework for Estimating the National Economic Benefits of ATP of Medical Technologies. Prepared for Advanced Technology Program, National Institute of Standards and Technology, U.S. Department of Commerce. July 1998 (expected).

Sloan, Frank A., ed. Valuing Health Care: Costs, Benefits, and Effectiveness of Pharmaceuticals and Other Medical Technologies. New York: Cambridge University Press, 1995.

Torrance, George W., and David Feeny. "Utilities and Quality-Adjusted Life Years." International Journal of Technology Assessment in Health Care 5, 1989, pp. 559_575.

Viscusi, W. Kip. Fatal Tradeoffs: Public and Private Responsibilities for Risk. New York: Oxford University Press, 1992.

Author Biography

Andrew Wang, an Economist in the Economic Assessment Office of the Advanced Technology Program, National Institute of Standards and Technology, holds a Ph.D. in Economics from Harvard University.

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)