NIST Advanced Technology Program
Return to ATP Home Page
ATP Historical Award Statistics Business Reporting System Surveys EAO Economic Studies and Survey Results ATP Factsheets ATP Completed Projects Status Reports EAO Home Page
NIST IR 7319 - Toward a Standard Benefit-Cost Methodology for Publicly Funded Science and Technology Programs

III. SIMILARITIES AND DIFFERENCES IN ATP STUDIES

ATP has used aggregates of results of benefit-cost studies in comparing benefits from the limited number of projects for which such studies have been done with the entire costs of the Advanced Technology Program to date. These comparisons have often showed that estimated benefits exceed program costs to date by more than an order of magnitude. However, the results of this portfolio analysis were known to be very rough approximates of portfolio minimum performance, given that the studies were performed at different times and at different stages of project life cycles, a mix of retrospective and totally prospective benefits were quantified, and somewhat different metrics were reported. In addition, assumptions used in highly prospective studies sometimes proved too optimistic over time.

In recognition of these difficulties, ATP economists seek to develop a more comprehensive, integrated approach to benefit-cost studies that will support aggregation of results and comparability. With the passage of time, the possibilities for more retrospective analysis improve.

As a foundation for such an integrated effort, ATP has documented the similarities and differences across its benefit-cost studies that appear to affect the ability to combine results of individual studies to describe overall impacts of the program to date.

Similarities

ATP's benefit-cost studies have much in common:

  • All address ATP's mission. They aim to show, and succeed in showing, that ATP enables technology development that benefits industry and end users and generates broad-based benefits to the U.S. economy.
  • The studies follow the analytical paradigms of Mansfield and Jaffe, which emphasize (1) the large size of public benefits relative to private benefits that can often be expected from enabling new technologies (Mansfield et al. 1977), (2) the role for federal funding in addressing market failures in early-stage technology development (Jaffe 1996), (3) the substantial gap between social returns on technology investment (public plus private) and private returns alone (Mansfield et al. 1977; Jaffe 1996), and (4) the need to determine where federal funding is needed for these benefits to occur (Jaffe 1996, 1998).
  • Substantial "bottom-up" interviews with ATP-funded company representatives and customers are involved, as well as other market research, to estimate benefits of ATP-funded technologies (relative to existing or anticipated defender technology) and market demand. In general, benefits are estimated on the basis of a single product unit sold and extended to an estimated market. In instances in which the project would most likely have been funded at some level without ATP, the studies assess the effect of ATP funding on scope and timing of the projects compared with a hypothetical counterfactual situation without ATP funding.
  • The studies implement cash-flow analysis techniques consistent with public finance literature and good practice in both public and private investment analysis.
  • The same basic metrics are computed: net present value, benefit-to-cost ratio, and internal rate of return.
  • Considerable uncertainties exist about impacts being measured.
  • The studies are consistent with OMB Circular A-94 in computing the net present value of outcomes as the key metric of cost-effectiveness.
    • They consistently use the OMB-mandated rate of 7% (real rate) in dis­ counting benefits and costs to a common base year.
    • They consistently estimate benefits and costs in constant dollars, gen­ erally using the dollar value at the time the study was conducted.
    • They address uncertainties and include some sensitivity analysis.
  • The studies follow similar approaches to estimating period of economic benefits. None assume more than 10 years; all consider competing developments and assess the economic life of the technology.
  • ATP projects a priori believed to be economically successful and therefore worth the investment in the study are selected for these studies, although some studies cover a broader cluster or portfolio of projects.
  • Study results are not assumed to be representative of the outcomes for the portfolio of ATP projects and are not presented as such. Instead they are presented as impacts of some of ATP's more successful projects. In aggregation, the estimated impact of these projects alone is assumed to be no better than a minimum of impact for the entire portfolio. ATP uses other evaluation tools, including high-level overview case studies and a composite performance rating system, to examine all projects— successful and unsuccessful. The quantitative benefit-cost studies examine only some of the more successful projects for which advance screening suggested substantial measurable economic and social impact.

Differences

Comparability of studies, potential for aggregating results across studies, and actual utility in program evaluation are affected by a number of factors. From hindsight, ATP economists are learning how to compensate for some nonoptimal conditions and plan to address some key sources of difference in ongoing efforts such as this methodological study and further case studies. Differences can be summarized as follows:

  • Timing of studies and uncertainties about projected future outcomes
  • Which metrics? Social returns (that is, public and private benefits on combined public-private investments) and/or public return on ATP investment
  • Identification of specific counterfactual to ATP funding
  • Attribution to ATP given multiple funding sources
  • Different base years and different constant dollar years (base year and constant dollar year often differ in a given study and across studies)

Framework of Analysis

The rest of this report examines these differences as evidenced in our portfolio of published studies. We draw on results of the workshop and the more comprehensive analysis of differences across our portfolio to start laying a foundation for a standard approach to benefit-cost analysis for publicly funded R&D projects. Three criteria are applied in considering an appropriate standard approach:

  • ATP's evaluation objective of measuring program impacts against the program's mission while also more broadly considering tools appropriate to the evaluation of public-private science and technology programs
  • Quality relative to analytical models established in the economic and public finance literature and accuracy as provided by established cash­flow analysis procedures used by the business community
  • Effects on consistency and comparability across studies and the ability to aggregate results of different studies

The matrix shown in Table 2 illustrates this framework.

Table 2.    Framework of Analysis

 

Evaluation Objective

Quality and Accuracy

Comparability and Consistency

Timing of study and uncertainties about benefit outcomes

 

 

 

Which metrics: Social and/or public?

 

 

 

Role of ATP: Identifying the counterfactual and appropriate attribution to ATP

 

 

 

Adjusting for time differences: Base year Constant dollars

 

 

 

The analysis is limited to studies that used standard cash flow-based financial analysis techniques envisioned in OMB Circular A-94, although many of the issues addressed apply similarly to other quantitative case study approaches.

Return to Table of Contents or go to next section.

Date created: July 11, 2006
Last updated: July 12, 2006

Return to ATP Home Page

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)

Return to NIST Home Page
Return to ATP Home Page Return to NIST Home Page Go to the NIST Home Page