NIST IR 7319 - Toward a Standard Benefit-Cost Methodology for Publicly Funded Science and Technology Programs
III. SIMILARITIES AND DIFFERENCES IN ATP STUDIES
ATP has used aggregates of results of benefit-cost studies in comparing benefits from the limited number of projects for which such studies have been done with the entire costs of the Advanced Technology Program to date. These comparisons have often showed that estimated benefits exceed program costs to date by more than an order of magnitude. However, the results of this portfolio analysis were known to be very rough approximates of portfolio minimum performance, given that the studies were performed at different times and at different stages of project life cycles, a mix of retrospective and totally prospective benefits were quantified, and somewhat different metrics were reported. In addition, assumptions used in highly prospective studies sometimes proved too optimistic over time.
In recognition of these difficulties, ATP economists seek to develop a more comprehensive, integrated approach to benefit-cost studies that will support aggregation of results and comparability. With the passage of time, the possibilities for more retrospective analysis improve.
As a foundation for such an integrated effort, ATP has documented the similarities and differences across its benefit-cost studies that appear to affect the ability to combine results of individual studies to describe overall impacts of the program to date.
Similarities
ATP's benefit-cost studies have much in common:
- All address ATP's mission. They aim to show, and succeed in showing, that ATP enables technology development that benefits industry and end users and generates broad-based benefits to the U.S. economy.
- The studies follow the analytical paradigms of Mansfield and Jaffe,
which emphasize (1) the large size of public benefits relative to private
benefits that can often be expected from enabling new technologies
(Mansfield et al. 1977), (2) the role for federal funding in addressing
market failures in early-stage technology development (Jaffe 1996), (3)
the substantial gap between social returns on technology investment
(public plus private) and private returns alone (Mansfield et al. 1977;
Jaffe 1996), and (4) the need to determine where federal funding is
needed for these benefits to occur (Jaffe 1996, 1998).
- Substantial "bottom-up" interviews with ATP-funded company
representatives and customers are involved, as well as other market
research, to estimate benefits of ATP-funded technologies (relative to
existing or anticipated defender technology) and market demand. In
general, benefits are estimated on the basis of a single product unit sold
and extended to an estimated market. In instances in which the project
would most likely have been funded at some level without ATP, the
studies assess the effect of ATP funding on scope and timing of the
projects compared with a hypothetical counterfactual situation without
ATP funding.
- The studies implement cash-flow analysis techniques consistent with
public finance literature and good practice in both public and private
investment analysis.
- The same basic metrics are computed: net present value, benefit-to-cost
ratio, and internal rate of return.
- Considerable uncertainties exist about impacts being measured.
- The studies are consistent with OMB Circular A-94 in computing the net
present value of outcomes as the key metric of cost-effectiveness.
- They consistently use the OMB-mandated rate of 7% (real rate) in dis
counting benefits and costs to a common base year.
- They consistently estimate benefits and costs in constant dollars, gen
erally using the dollar value at the time the study was conducted.
- They address uncertainties and include some sensitivity analysis.
- The studies follow similar approaches to estimating period of economic
benefits. None assume more than 10 years; all consider competing
developments and assess the economic life of the technology.
- ATP projects a priori believed to be economically successful and
therefore worth the investment in the study are selected for these studies,
although some studies cover a broader cluster or portfolio of projects.
- Study results are not assumed to be representative of the outcomes for
the portfolio of ATP projects and are not presented as such. Instead they
are presented as impacts of some of ATP's more successful projects. In
aggregation, the estimated impact of these projects alone is assumed to
be no better than a minimum of impact for the entire portfolio. ATP uses other evaluation tools, including high-level overview case studies and a composite performance rating system, to examine all projects— successful and unsuccessful. The quantitative benefit-cost studies examine only some of the more successful projects for which advance screening suggested substantial measurable economic and social impact.
Differences
Comparability of studies, potential for aggregating results across studies, and actual utility in program evaluation are affected by a number of factors. From hindsight, ATP economists are learning how to compensate for some nonoptimal conditions and plan to address some key sources of difference in ongoing efforts such as this methodological study and further case studies. Differences can be summarized as follows:
- Timing of studies and uncertainties about projected future outcomes
- Which metrics? Social returns (that is, public and private benefits on
combined public-private investments) and/or public return on ATP
investment
- Identification of specific counterfactual to ATP funding
- Attribution to ATP given multiple funding sources
- Different base years and different constant dollar years (base year and
constant dollar year often differ in a given study and across studies)
Framework of Analysis
The rest of this report examines these differences as evidenced in our portfolio of published studies. We draw on results of the workshop and the more comprehensive analysis of differences across our portfolio to start laying a foundation for a standard approach to benefit-cost analysis for publicly funded R&D projects. Three criteria are applied in considering an appropriate standard approach:
- ATP's evaluation objective of measuring program impacts against the
program's mission while also more broadly considering tools appropriate
to the evaluation of public-private science and technology programs
- Quality relative to analytical models established in the economic and
public finance literature and accuracy as provided by established cashflow analysis procedures used by the business community
- Effects on consistency and comparability across studies and the ability to
aggregate results of different studies
The matrix shown in Table 2 illustrates this framework.
Table 2. Framework of Analysis
|
Evaluation Objective |
Quality and Accuracy |
Comparability and Consistency |
Timing of study and uncertainties about benefit outcomes |
|
|
|
Which metrics: Social and/or public? |
|
|
|
Role of ATP: Identifying the counterfactual and appropriate attribution to ATP |
|
|
|
Adjusting for time differences: Base year Constant dollars |
|
|
|
The analysis is limited to studies that used standard cash flow-based financial analysis techniques envisioned in OMB Circular A-94, although many of the issues addressed apply similarly to other quantitative case study approaches.
Return to Table of Contents or go to next section.
Date created: July 11, 2006
Last updated:
July 12, 2006
|