From the start of the Advanced Technology Program in 1990, the Economic Assessment Office began building an internal system of longitudinal surveys and sponsoring rigorous and groundbreaking economic and policy studies, all of which have advanced the understanding of the process of technology-based innovation. For these evaluation activities, the National Research Council recognized ATP for setting “a high standard for assessment involving both internal and independent external review.”

Evaluation has provided an objective, analytical and empirical basis for assessing ATP’s operations and impacts of operations. Cumulatively, these evaluations highlight the value of applying multiple evaluation methods to complex problems, building a body of credible evidence over time that ATP is achieving its objectives.

An Emerging Knowledge of ATP

ATP-sponsored evaluation studies have produced an emerging knowledge of the overall performance of the program, firm behavior, collaboration, spillover effects, and interfaces with state and international technology programs.

Overall ATP Performance

• Benefit-cost case studies provide evidence that the benefits of the program far exceed its costs. These studies collectively attributed to ATP more than $18 billion in expected present value social benefits from about 40 projects, much greater than the total amount spent by the program of $2.3 billion to date, a greater than 8 to 1 return on investment.
• As expected, not all of the projects are strong performers, but several years after project end an estimated 10% of completed ATP-funded projects showed strong progress toward creating and disseminating knowledge and commercializing projects and processes, and another 31% also showed substantial progress. Slightly less than 10% of all funded projects failed to start or were terminated prior to completion for a variety of reasons.

Firm/Industry Effects

• ATP funding is complementary to, not a substitute for, private sources of R&D funds. Industry would not have undertaken 40% of ATP projects and another 40% would have proceeded on a much slower scale.
• ATP funding leverages and accelerates R&D, refocuses R&D on more technically challenging problems and enabling platforms of technologies, and fills a significant funding gap in early-stage technology development. ATP funding accelerates the R&D cycle for 9 out of 10 companies and over half are ahead by 1 to 3 years.

Collaboration Effects

• High rates of collaboration in ATP projects are the result of encouraging the formation of joint venture and the use of subcontractors and strategic partners by single applicant companies. Four out of five ATP projects involved collaborative relationships, ranging from R&D partnerships with other firms, universities, and non-profit labs, to alliances with other firms to pursue commercialization.
• Collaborations with universities were frequent and they enhanced the research capabilities of the firms and provided an avenue of knowledge diffusion from and through the universities.

Spillover Effects

• Considerable evidence that ATP-funded projects generate outputs—publications, patents, patent citations, collaborative linkages, and products—that potentially lead to knowledge and market spillovers.
• ATP selects projects with attributes conducive to generating large knowledge spillover effects. Those attributes included linkages to other organizations, and a positive attitude of award winners toward information sharing.

Source: The framework for presenting these results is from a recent publication, A Toolkit for Evaluating Public R&D Investment: Models, Methods, and Findings from ATP’s First Decade: Models, Methods, and Findings from ATP’s First Decade (NIST GCR 03-857). This factsheet updates the findings in the Toolkit and adds findings for the most recent 5 years of evaluation work.

Factsheet 1.A1 (April 2005 by Connie Chang)