V. IDENTIFYING THE COUNTERFACTUAL AND ATTRIBUTION OF BENEFITS

What Are the Issues?

The issues are: What is the project being analyzed? And what is the counterfactual to the project that enables definition of project impact? Both questions entail considerable definitional and measurement challenges.

For a benefit-cost study, the project analyzed spans the life cycle from R&D through long-term outcomes and economic impact. Investment costs and benefits included in the analysis should reflect the increment over, and separable from, the counterfactual, or hypothetical situation that would have existed without the project being analyzed. Typically, the counterfactual situation is assumed to be the continuation of profits and products from a defender technology. The counterfactual describes the "losses" in the process of shifting to a newer technology from the older defender technology. The analysis seeks to capture the benefits of the newer technology less the losses from displacing the older one. In a dynamic environment, the counterfactual may be evolving in parallel with the technology being analyzed, and the defender technologies may be new technologies competing with it.

In practice, data to support analysis of actual outcomes relative to a counterfactual, as a controlled experiment, probably do not exist. Analysts use different approaches, or a mix of approaches, to define the project of study, from investment costs to long-term outcomes and economic impacts, and compare the project with a hypothetical alternative situation that reflects the researcher's best judgment as to what would have occurred in the absence of the project of analysis.

In S&T federal program evaluation, "the project" is the R&D that "the program" funded. In practice, the economic outcomes, or "benefits," often are the result of many investments of varying criticality. For example, public and private benefits from the sale and use of a product embodying specific new technology can derive from multiple public and private investments. And benefits may evolve from a stream of product generations that use increasing or decreasing amounts of the new technology resulting from the project.

The project could potentially be defined differently depending on whether one seeks to emphasize the return on investment from the point of view of a single government funding agency or the return on all investments that lead to a particular stream of benefits. Obviously, when both social returns (returns to all beneficiaries, including innovators, on all investments) and public returns (in ATP studies, public return is computed as returns, excluding those to innovators who received ATP funds, on ATP's investment) are presented together, the project analyzed for public return and social return purposes must be the same project, and benefits need to be traceable to the project. A mix is often needed: ATP-funded technology is traced forward into the marketplace and products in the product pipeline or in the marketplace are traced backward through the technology development process to see whether ATP played a role. In instances in which the stream of benefits clearly depended on ATP among other external investment sources, some method of benefit allocation may be needed. A social return analysis will need to include industry's full costs for R&D, including early-stage R&D and product development costs.

Contrasting Examples

In practice, researchers first try to identify product outcomes with a direct connection to ATP-funded technology. They then conduct interviews to attempt to construct a counterfactual situation and elicit the ATP effect. Some companies have been working in the ATP area before applying to ATP. Some of them indicate that they would pursue similar goals without ATP but at a slower pace and with a poorer chance of success. Others indicate they could have done little or nothing without ATP and/or other government funding. The latter may be true for large companies pursuing new directions with ATP funding or for small start-up companies.

Cash flow-based benefit-cost studies have used a variety of mechanisms to model economic effects of an ATP project relative to a hypothetical situation in which ATP funding was absent. In general, this occurs in two stages: The researcher identifies the economic benefits from products and processes related to the ATP project and then, through a counterfactual analysis, establishes the portion of these benefits that are attributable to ATP. Table 4 summarizes many of the approaches used to define the benefits attributable to ATP.

The tissue engineering study defined project benefits in regard to acceleration of benefits and increased probability of technical success relative to a hypothetical project without ATP. The return on the ATP investment was the resulting incremental effect of acceleration of R&D and greater technical success. Benefits of the general technical capabilities (in regard to improved patient outcomes anticipated when the technology was put to use) were estimated independently of these effects. However, the amount of benefit attributed to ATP was the portion of benefits estimated to be accelerated with ATP funding, the portion attributable to the bigger research project enabled by ATP funding and collaboration opportunities, or both.

The component-based software study took a very different approach. By focusing on ATP projects that were already in commercialization, researchers could identify market product outcomes that were related to ATP funding. The study estimated all public and private benefits of these products and compared them with all costs to bring the products to market, from R&D through product development. The resulting metrics showing public plus private return on combined sources of investment did not attempt to quantify ATP's direct contribution. Most of the projects analyzed, however, were undertaken by tiny companies with little access to private or other public financing. The products assessed were their first and only products. ATP funding was obviously critical. The study discussed the need for ATP funding to achieve these benefits and the effects of ATP on acceleration as qualitative effects.

The amorphous-silicon detectors for digital mammography study and the composites cluster case studies all concluded from in-depth company interviews that the research project funded by ATP would not have been undertaken without ATP. These studies identified specific products or product or cost improvements and related user benefits resulting from the ATP project directly. Benefits to industry customers and end users of these products or improvements were attributed fully to ATP based on a counterfactual analysis in computing the return on ATP's investment. For example, for the amorphous-silicon detector project, the counterfactual was the existing, higher-cost digital mammography system. Benefits attributable to the ATP project were limited to those deriving from the ATP-funded manufacturing process that reduced costs of production and delivery of digital mammography systems and thereby enabled more patients to benefit from digital mammography than occurred with the earlier system.

In the study of a cluster of photonics projects, it was determined that ATP and NSF provided similar levels of funding to one of the projects. Company officials indicated a strong belief that neither funding would have occurred without the other and that the two agencies were jointly responsible for realizing any economic benefits from the company's first products. The study attributed to ATP 50% of estimated benefits from these impending products in computing the return on ATP's investment.

Note that computing social returns on investment (public plus private) on combined sources of investment does not require as strong an assumption about "causation" as does public return on ATP's investment. The social return set of metrics compares all benefits from a specific product/group with all investments required to generate those benefits. The estimation challenge is less one of ATP attribution than one of identifying product(s)/economic outcomes to target in the analysis and obtaining all relevant data for those products compared with the counterfactual products.

Obviously, when both social returns and public returns are presented together, "the project" analyzed for public return and social return purposes must be the same project, and benefits need to be traceable to "the project." If benefits to the project are traceable back to ATP among other projects or sources of investment, for purposes of computing the returns on the ATP investment, benefits may need to be allocated to different funding sources based on acceleration or other effects due to ATP funding or a pro rata share of investment costs provided by ATP.

Summary of Counterfactual and Attribution of Benefits Issues

Studies use a variety of mechanisms to model incremental benefits of an ATP-funded technology relative to a counterfactual situation in which ATP funding did not exist. Acceleration of benefits, increased likelihood of achieving benefits, and allocation in accordance with the amount or importance of individual funding sources (given multiple sources) have all been used.

The attribution question increases in complexity over the technology life cycle. Studies performed later in the commercialization phase encounter a complex weave of threads from and to ATP. The complexity may increase with the size of the company ATP funded (or its successor) if technology products are numerous, or past R&D investments were numerous, or both. The ATP lineage is clearer for most studies performed earlier; however, those studies face much more uncertainty about ultimate product benefits and unit sales.

Isolating the benefits of the ATP project can be equally or more difficult in retrospective analysis of fully commercialized technologies than analysis of R&D outcomes at an earlier stage. Ultimately, ATP-funded technology may be embodied in multiple products of a company, each of which resulted from a number of different public and/or private funding sources and different R&D projects. A mix of "tracing ATP-funded technology forward" into the marketplace and tracing products in the product pipeline or in the marketplace backward through the technology development process to see whether ATP played a role is often needed to (1) define the ATP project over its life cycle and (2) establish its impact relative to a hypothetical situation in which ATP funding was not available.

Return to Table of Contents or go to next section.

Date created: July 11, 2006
Last updated: July 12, 2006

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)