NIST Advanced Technology Program
Return to ATP Home Page
ATP Historical Award Statistics Business Reporting System Surveys EAO Economic Studies and Survey Results ATP Factsheets ATP Completed Projects Status Reports EAO Home Page
NIST IR 7319 - Toward a Standard Benefit-Cost Methodology for Publicly Funded Science and Technology Programs

VI. WHICH METRICS: SOCIAL RETURN ON INVESTMENT AND/OR PUBLIC RETURN ON PUBLIC (ATP) INVESTMENT?

What Are the Issues?

Some studies and researchers emphasize return on ATP investment metrics; others emphasize social return on investment metrics. The social return metrics include both public and private benefits of a given project compared with all investment costs incurred in the R&D and product development phases. The public return on ATP investment metrics focus on the public benefits attributable to ATP relative to ATP's investment. (Return on ATP investment metrics can be defined to include both public and private benefits; however, most of the studies that compute these measures have focused on public returns, in keeping with ATP's mission.) The social return metrics and public return on ATP investment metrics measure different things, and there is no consistent relationship between them.

Return to Investment Measures

  • Public return on ATP investment: Return to nation attributable to ATP (defined in most ATP studies to exclude benefits to ATP project participants) on ATP investment
  • Social return: Return to ATP project participants and nation on total investment from all sources Private return: Return to ATP project participants on total investments from all sources

Contrasting Examples

Tables 5 and 6 illustrate the variation across just the more recent studies. The metrics for each of the eight component-based software case studies are presented in Table 5. The metrics for the comparison studies are shown in Table 6. The component-based software study and digital video study computed social return metrics. The amorphous silicon detector for digital mammography study and the composites and photonics cluster study cases, performed by a different contract researcher, computed public return on ATP investment metrics. In addition to public return on ATP investment metrics, the photonics case studies estimated social rates of return, but not the other social return metrics.
Both sets of metrics have utility and merit. Social return metrics are rooted in the literature on the economics of innovation. Public return on program investment metrics have stronger roots in public finance and program evaluation. Tassey (2003, p. 19) suggests that "when the objective is to estimate the impact of the government role, two calculations should be made; one for the total impact (i.e., social return) and one for the contribution of the government subsidy (i.e., return on government investment)."

Table 5.   Which Metrics? Contrasting Projects in Component-Based Software

Component-based Software

Social Return Metrics

Public Return Metrics

NPV ($M)

B:C

IRR (%)

NPV ($M)        B:C        IRR (%)

Aesthetic Solutions

-1.2

.37

NC

 

 

 

Not computed for this study

Commerce One JV

789

39

363

Extempo

-1.22

.63

NC

Intermetrics

29.6

9.6

103

Real-Time Innovations 2.06 1.8 31

Sci Comp

21

7.6

51

Tom Sawyer

52

18

136

Xerox PARC

1.2

1.2

13

Note: IRR = internal rate of return; NC = not computed; NPV = net present value.

Table 6.   Which Metrics? Other Contrasting Studies

Other Contrasting Studies

Social Return Metrics

Public Return I Metrics

NPV ($M)

B:C

IRR (%)

NPV ($M)

B:C

IRR (%)

Digital video (midpoint)

165.9

4.24

28.6

Not computed forthis study

A-Si Detector (base case)

NC

NC

NC

219

125

69

Composites (base cases)
      Applied Sciences
      Lincoln Comp.


NC
NC


NC
NC


NC
NC


552
510


221
187


57
58

Photonics (base cases)
      X-Ray Optics
      Ion Optics


NC
NC


NC
NC


43
51


184
143


75
174


49
75

Note: IRR = internal rate of return; NC = not computed; NPV = net present value.

Work stemming from Edwin Mansfield in the late 1970s consistently finds evidence that social rates of return on R&D investments exceed private rates of return and private hurdle rates. The "spillover gap," or difference between the social and private rates of return, is indicative of a market failure in funding R&D. This body of work underlies the case for government support of R&D and national innovation policy. R&D projects that are cost shared between public and private sources, such as ATP projects, further imply that a relevant benefit-cost comparison involves both public and private benefits and investments.

The component-based software and digital video studies illustrate the difference between what is being measured in the social return metrics and what is measured in public return metrics and the variation in the effect of those differences. Although neither study presents separate public and private return metrics, the component-based software study provides measures of producer surplus (benefits captured by innovator firms) and consumer surplus (benefits captured by customers), which can be considered proxies for private benefits and public benefits, respectively. For the component-based software study, across all eight project cases, the consumer surplus was more than twice the producer surplus; however, there was a huge variation among projects. Surprisingly, the estimated producer surplus exceeded the consumer surplus for many of the small-company projects; however, the consumer surplus vastly exceeded the producer surplus for the successful Commerce One Joint Venture and for the Xerox PARC project. In the digital video study, the private benefits to U.S. companies were estimated to be very, very small compared with the public benefits. A public return on ATP investment analysis alone would fail to capture this variation among the separate private benefit and public benefit components of social benefits. And a public return on ATP investment analysis would further require a more in-depth assessment of the degree to which the consumer surplus, or public benefits, are attributable to ATP than was provided in the social return analysis. (See the prior section, "Identifying the Counter factual and Attribution of Benefits," for further discussion of this issue.)

Of course, the cost side, as well as the benefits side of the analysis, is different for the social return metrics compared with the public return on ATP investment metrics. Full social costs span the earlier stages of R&D, such as ATP funds, product development, and potentially production scale-up. Of course, this life cycle of costs affects computed rates of return. In addition, it greatly complicates the computation. The analysis requires considering a vast amount of historical data. The historical stream of investment cost data is difficult to come by. Private profit data, on a product basis, for estimating the private portion of benefits may not exist or may be highly confidential.

Researchers work hard to build a positive relationship with company representatives, including CEOs and presidents of small companies. In response, companies have been cooperative with these ATP studies, even though they generally require a full day of on-site interviews plus a number of phone calls and/or preparation of responses to written questions. However, the atmosphere changes when researchers probe explicitly for profit and cost of sales data. Across the board, researchers have settled for less in this area in return for forthcoming responses in other areas.

The social return metrics that include investments from multiple sources of funding of R&D may tend to foster a broader definition of the R&D project being studied than is most useful for program evaluation. One source of funding can overlap another one, and one product generation or R&D project outcome may overlap another. As a result, a narrow R&D project nucleated with specific federal funding can become buried among a chain of related R&D projects, with outcomes partially in common. If the project is interpreted broadly, the federal program's effect on outcomes— that is, its additionality effect—becomes invisible in the resulting metric. Unless the project evaluated is traceable directly and more or less entirely to a single program that enabled the project, the social return metrics provide a general picture of the return on investment from multiple sources but may say little if anything about programmatic impact from a program evaluation perspective.

This potential muddiness concerning project definition with the social return metrics does not appear to have been a major problem with ATP's component-based software and digital video studies. The two large joint ventures included in these studies—the Sarnoff-led joint venture covered by the digital video study and the Commerce One Joint Venture in the component-based software study—were uniquely ATP. It is unlikely that the resulting products would have come to be in a similar time frame without ATP. Five of the seven remaining projects covered in the component-based software study were start-up or otherwise very small companies that had received little funding besides ATP and whose only products were closely derived from their ATP project. Although several of these analyses included investment costs considerably beyond ATP's funding and company cost share on the ATP project, most of the additional funding directly supported commercialization of the ATP-funded technology. Therefore, the resulting social return metrics can be interpreted as "ATP project impact." The social return metric for the photonics project in which ATP and NSF provided similar levels of funding in a similar period of time would need to be interpreted more broadly.

Summary of Return on Investment Measures

The best strategy would appear to be to compute both social return metrics and metrics that show the impact of the specific federal program.

Metrics that focus specifically on the incremental effect of the specific federal program (for ATP, the public return on ATP's investment) have obvious utility for program evaluation. They also involve substantially less data. The challenge is to clearly define the funded project's outcomes compared with a hypothetical counterfactual in a way that pinpoints differences in outcomes attributable to the federal program being evaluated. (See the prior section, "Identifying the Counterfactual and Attribution of Benefits.")

Social return metrics capture the return to all investors, which is particularly important for projects that are cost shared with industry. Industry proposes projects for federal cost share from which it anticipates a future return on its investment in the form of profits from resulting products and processes. Industry's return is of concern to the federal program because it provides the incentive to propose. Differences in the private component and public component of the social return further assist in assessing the potential market failure that justifies federal funding.

Given the confidentiality of some of the data needed for computing social return metrics, public industry data may need to supplement company data. Relatively simple estimation techniques for social rates of return that draw on public data may be more practical than comprehensive analysis of actual private investment and profit data. Such simplified techniques may actually lead to more consistent methodologies and more comparable social return metrics across projects. However, little work has been done to develop practical social return methodologies, apply them illustratively, and test them for reasonable accuracy.

Return to Table of Contents or go to next section.

Date created: July 11, 2006
Last updated: July 12, 2006

Return to ATP Home Page

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)

Return to NIST Home Page
Return to ATP Home Page Return to NIST Home Page Go to the NIST Home Page