The Advanced Technology Program, Its Evaluation Plan, and Progress In Implementation*

Rosalie T. Ruegg
Advanced Technology Program, National Institute of Standards and Technology
Technology Administration, U.S. Department of Commerce
Gaithersburg, MD 20899

Abstract

The Advanced Technology Program (ATP) partners with U.S. businesses in high-risk research to develop enabling technologies with strong potential for economic benefits to the nation. Universities, nonprofit research laboratories, and firms of all sizes participate in ATP-funded projects. ATP's evaluation effort seeks not only to measure the impacts of the technology development projects it funds, but also to understand the underlying relationships among research, technological change, and economic impact, and to provide feedback to the program to increase its broad-based benefits. Evaluation activities encompass developing models, conducting surveys, compiling databases, conducting micro- and macro-economic case studies, and performing statistical and econometric analyses. Program metrics include private rates of return, social rates of return, and public rates of return the social-rate-of-return component attributable to the ATP. Topics of special interest, in addition to performance metrics, include spillover pathways, benefits and costs of collaboration, financing issues, and new models of impact assessment.

Overview of the ATP

To accomplish its mission, the ATP partners with U.S. businesses of all sizes in high-risk scientific research to develop enabling technologies with strong potential for broad-based economic benefits. ATP's focus is on technologies that offer, in civilian applications, the potential for substantial increases in productivity and competitiveness of firms, provide consumers with new, better, and lower-cost products and services, and increase high-wage employment in the U.S.

The multi-year ATP awards are made both to individual companies and to joint research ventures comprised of two or more companies, often in combination with universities and nonprofit research laboratories. Most of the single-company awards actually resemble joint ventures in their involvement of other organizations. But the single-company awards are limited to $2 million and three years, while the joint-venture projects have no mandated limit on the amount of award, and their period of performance is limited to five years instead of three.

ATP awards are made through fair and rigorous competitions. The ATP held its first competition for awards in 1990, and has held a total of 30 competitions through 1997. Seven of the competitions one each year were "General Competitions," which are open to all technologies and applicants. The remainder were "Focused Program Competitions," through which, beginning in 1994, ATP has funded suites of related projects to achieve pre-identified sets of technological and economic goals developed in concert with industry. The General Competitions allow the ATP to cast a wide net and fund a wide array of good ideas. The Focused Competitions enable the ATP to deepen its funding to address larger problems and opportunities requiring concentrated, coordinated efforts.

From 1990 through 1997, the ATP made multi-year awards for a total of 352 projects, including over 100 joint ventures, and involving more than 800 participants (not including the many subcontractors and informal partners and collaborators that participate in many of the projects). These projects entail approximately $2.3 billion of research, of which industry committed slightly more than half, and ATP the remainder. Some of the earlier funded projects are now completed and an increasing number of them are moving into the commercialization phase. Most of the funded projects at this time are still in the research phase, reflecting the fact that the program's budget permitted more projects to be funded in the latter half of the 1990s than during the early years.

A rigorous peer review process is used to select all awards. Panels of technical, business, and economic experts carry out evaluations to assess the technical and economic merit of applicant proposals. Selection criteria include the potential of the research to contribute significantly to the scientific and technical knowledge base of the country; the potential of the technology to generate broad-based benefits to the nation and the difference ATP funding would make to the effort; the proposed pathways to market and to broader diffusion; the commitment of the proposers to follow through with future development of products and processes if they are successful in overcoming the challenging technical barriers addressed in their ATP research project; the organizational structure of the project, and its management plan; and the proposer's experience, qualifications, and resources.

The ATP relies on the presence of expected private returns to induce companies to be willing to plan, propose, and cost-share research with the ATP, and, if the research is successful in overcoming the technical hurdles, to pursue commercial development of the new technology with private capital. To the many proposals it receives from companies, the ATP applies its criteria to identify which are expected to accomplish ATP's public-interest mission and objectives. It selects for award projects whose potential social rate of return (the return to the nation) is expected to far exceed the private rate of return on investment, and for which the private sector is unable without ATP assistance either to do the project at all, or to do it within the critical time frame, scale, or scope necessary to realize the social benefit potential.

The ATP is not a customer for the results of the technology development projects it funds. The ultimate outcomes are the results of the interplay of demand and supply forces in the marketplace. For this reason, the ATP is concerned not only with the benefits potential of the technologies from a hypothetical standpoint, but also from a very practical standpoint in terms of the proposers' ability, commitment, and plans actually to bring the technology into use in a timely way.

The ATP is not a funder of streams of basic research in the traditional mode of the U.S. National Science Foundation. It is NOT a procurer of mission-related applied research in the mode of the U.S. Defense Department and the U.S. Department of Energy. It funds research, not product development. It operates in the middle ground between basic research and product development.

Having industry conceive, propose, conduct, and cost-share the projects keeps them anchored in the applied world, oriented towards commercial potential, and run with attention to efficiency. Having highly qualified research scientists plan and conduct the research helps to extend the nation's scientific and technical knowledge base by taking on cutting-edge research with a reasonable level of feasibility. And funding only those emerging and enabling technologies which promise large benefits beyond the direct funding recipients, and which require ATP assistance to happen, ensures that the public interest is served. As expressed by Spender, ". . . by focusing on cost-sharing R&D within broadly conceived programs which are anticipated to have major public goods consequences, ATP harnesses private firms' resources to the public interest without seeking to outguess market forces" (Spender 1997, p. 46).

More information about the ATP is readily available at its web site, http://www.atp.nist.gov. There one can find summary statistics, application guidelines, selection criteria, project descriptions, participating organizations, funding amounts, and other information about the program. This information can, of course, also be obtained by contacting the ATP directly (1-800-ATP-Fund).

What and When to Measure

Because ATP's mission is economic in nature, its evaluation emphasizes economic impacts of the program. But there are a number of sub-objectives and constraints to the program that condition the program and its evaluation. For example, in addition to providing economic benefits, projects must entail high-risk research. Hence, one aspect of the ATP's evaluation concerns the scientific and technical contributions of funded research. Because the research is high-risk, it is understood that not all projects will be fully successful. Only a fraction will likely accomplish all the goals scientific knowledge creation, timely commercialization of products and processes, and widespread diffusion of the technology leading to large spillover benefits. Most will be at least partial successes given that scientific knowledge often is gained even from research failures. Many will likely yield a sufficient return to pay back their costs. A few likely will be "home runs."

Since accelerating the development and commercialization of technology is a program mission, ATP's evaluation tracks the degree of speed up of technology development and the rate of commercial progress of award recipients. It also investigates the economic value of accelerating technology development projects.

Since the program is for national benefit, the evaluation is concerned with generating and measuring spillover effects beyond the direct benefits to innovators including market spillovers, knowledge spillovers, and network spillovers. It seeks to fund technology development projects for which the spillovers are large, as reflected in gaps between the resulting social rates of return and the private rates of return to the innovators.

Because the ATP is charged with promoting the formation of research joint ventures, collaborative research is another topic of particular interest for ATP's evaluation. The objectives and progress of the individual joint-venture members are tracked, as well as those of the overall entity. Efficiency issues, the internalization of spillover effects, and technology diffusion effects are examples of research topics of interest to the ATP that concern collaborations.

Because it is critically important to the realization of benefits from the program that the ATP make a net contribution to the nation's economy leveraging rather than displacing private sources of capitalATP's evaluation seeks to measure the differential impacts attributable to the ATP, in addition to the overall impact of projects and groups of projects. With- and without-ATP scenarios are posed to help get at the effects attributable to the ATP. This entails the use of counterfactual and the attendant uncertainties thereof.

Overview of ATP's Evaluation Program

The ATP initiated evaluation at the outset of the program, first, to develop a management tool to make the program better meet its mission and operate more efficiently; and, second, to meet the many external requirements and requests for ATP program results. Demands for performance measures for the ATP are intense. Requests for evaluation results come frequently from individual members of Congress and their staff, from Congressional subcommittees, the General Accounting Office, the Executive Office of the President, the Office of Management and Budget, the Office of Inspector General, the Press, think tanks, industry groups, and others.

Title II of the American Technology Preeminence Act of 1991 (P.L. 102-245), enacted in 1992, directed that a comprehensive report on the results of the ATP be submitted to each House of the Congress and the President not later than 1996. This report was delivered in April 1996 (The Advanced Technology Program 1996).

In addition, the ATP, like other federal programs, is subject to the evaluation requirements of the 1993 Government Performance and Results Act (GPRA). The GPRA resulted from a bipartisan effort to improve accountability, productivity, and effectiveness of federal programs through strategic planning, goal setting, and performance assessment. The ATP/NIST is developing assessment plans and techniques, and carrying out evaluation studies in compliance with the GPRA. The ATP receives many inquires about its evaluation tools and methodologies from other agencies, as well as from similar programs in other countries.

To square the often urgent demands in the short run for evaluation results with the reality that patience is required to realize and validate empirically long-run program outcomes, the ATP has adopted a multicomponent evaluation strategy. Its main components include (1) descriptive (statistical) profiling of applicants, projects, participants, technologies, and target applications; (2) progress measures derived principally from surveys and ATP's "Business Reporting System;" (3) real-time monitoring of project developments by ATP's staff; (4) "status reports" on completed projects; (5) microeconomic and macroeconomic case studies of project impacts; (6) methodological research to improve the tools of longer term evaluation; (7) special-issues studies to inform program structure and evaluation; and (8) econometric and statistical analyses of the impacts of projects and focused programs.

Overview of ATP's Evaluation Implementation

The ATP's Economic Assessment Office has developed ATP's approach to economic evaluation in consultation with leading economists in the field. The Economic Assessment Office holds periodic workshops to obtain feedback on its approach and plans, to review work in progress, and to solicit advice on future directions. For example, workshops on ATP's evaluation, co-chaired by Professor Zvi Griliches of Harvard University, were held at NIST in December 1994, September 1995, and January 1997; and at the National Bureau of Economic Research (NBER) in August 1997.

In conjunction with planning future directions of its evaluation effort, the ATP commissioned several background reports, including an early report on performance measures by Link (unpublished), a report by Jaffe on spillovers what they are, how they arise, and how ATP can increase its effectiveness by enhancing spillover effects (Jaffe 1996), and a report by Mansfield on problems and opportunities in estimating social and private returns from ATP projects (Mansfield, 1996).

Periodically the ATP conducts economic review boards to consider proposed evaluation studies in the context of its evaluation plan. Guidelines are provided to evaluation researchers on proposing studies to the ATP (Ruegg 1996). In order to access leading academic researchers, the ATP has funded many of its recent evaluation studies through a collaborative arrangement with the NBER.

One activity in support of ATP's evaluation is the compilation and analysis of databases mainly the ATP "Awards Database" and the ATP "Business Reporting System"to provide answers to numerous questions about what, where, and who ATP is funding, and to measure early results. Descriptive information important to assessing the portfolio of ATP projects comes from the Awards Database. The Business Reporting System, an integrated set of databases comprised of data compiled by electronic survey of project participants, is used to track the evolution of projects towards achieving their business and economic goals. It is described further by Powell in a paper in this collection. The Business Reporting System database has considerable potential for use in evaluation research.

Surveys are another tool used by ATP's evaluation. The ATP sponsored two broad surveys of funded companies by third-party contractors using telephone interviews to assess the progress of early projects not included in its Business Reporting System (Solomon 1993 and Silber 1996). The later survey included an assessment of the satisfaction of participants with the program. Another survey, the results of which are featured in a paper in this collection, focused on research acceleration (Laidlaw 1997). In a study now underway, Hall, Link, and Scott conducted a survey to collect information on the roles universities are playing in ATP-funded projects.

The case study method is an important tool for developing an understanding of how the projects actually function and how they yield results. Two case studies of ATP projects are featured in this issue (Link 1997 and CONSAD 1997), and another case study is treated indirectly in the paper by Wang, who expounds on key methods used to evaluate the impact of medical technologies. Several other studies now underway also entail case study, including a study by Vonortas on research joint ventures, and a study by Gompers and Lerner on financing issues.

A recently completed study used a case study approach to develop preliminary estimates of the potential benefits of an inclusive portfolio grouping of ATP projects (RTI 1998). Seven individual case studies of tissue engineering projects were conducted. The study entailed development of an evaluation framework that the ATP could consider for possible adoption with or without further modifications and extensions for evaluating a wide variety of technologies with medical applications. This work by Research Triangle Institute has stimulated further examination of the framework in preparation for future evaluations of medical technologies.

The ATP has recently launched its first effort at focused program evaluation, targeting its Digital Video Focused Program. The first phases of the effort entail identifying potential spillover pathways, developing an evaluation model, developing a data collection plan, and establishing a baseline for comparison. Two alternative approaches proposed by different contractors are being tested simultaneously, one approach by the Center for Economic Research at the Research Triangle Institute, and one by economists at the University of Kansas. It is too early in the process to report results from these efforts.

Another activity underway is to prepare status reports for all completed ATP projects. These reports provide a "snapshot" of developments after the ATP-funded phase of the research has finished. The emphasis of these reports is on technical and commercialization progress, publications, patents, outside recognition, and the outlook for future developments. The first set of status reports covered the first 38 ATP projects completed as of March 1997, plus approximately a dozen projects that were terminated during this time without completing. The next set of status reports is underway. Even though the status reports do not alone provide a thorough assessment of project impact, they update developments while the information is still fresh and relatively easy to document, and they are expected to be an important step towards individual project assessment, and towards the ability better to characterize ATP's portfolio of projects.

Most industrialized countries have programs similar to the ATP, and these other programs offer potential insights for the ATP. Hence, one of ATP's evaluation activities has been to collect information on foreign programs. We can learn from the experience of the other programs, as well as use the information to perform "determinations of eligibility of foreign-owned U.S. subsidiaries" to participate in the ATP, a requirement of Congress.

In addition to the studies mentioned above and treated in this collection, the ATP has conducted other evaluation studies and has a number of other studies underway. Research topics include spillover identification and evaluation; research collaborations; inter- and intrepidity diffusion mechanisms, patterns, and rates; development and application of new and improved qualitative and quantitative models for measuring economic impacts of publicly funded, privately executed technological advances; impact on firm productivity of government-funded research; technology financing issues; and organizational issues affecting project structure, participants, and outcomes. Some of these studies provided the basis of presentations at an international conference, "The Economic Evaluation of Technological Change," Washington, DC, June 15-16, 1998, sponsored by the ATP in conjunction with the NBER.

Evaluation Is an Ongoing Activity

Early evaluation studies suggest that the ATP is on track, meeting its objectives, and delivering results for the U.S. economy. But at this time only rough quantitative and qualitative projections of project impacts are possible due to limited information and uncertainties about the ultimate outcomes. These measures will become better informed as commercialization and diffusion activities progress. By tracking developments as they unfold, we expect over time to be able to reduce the estimating errors, extend the scope of analysis, and provide better measures. Over the coming years, the ATP expects to contribute significantly to the body of work on technology impact assessment and to build towards a more comprehensive view of the impacts of the ATP.

References

CONSAD Research Corp. Advanced Technology Program, A Case Study: The Development of Advanced Technologies and Systems for Controlling Dimensional Variation in Automobile Body Manufacturing. NIST Contract Report 97-709, March 1997.

Jaffe, Adam. Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program. NIST Contract Report 97-708, December 1996.

Laidlaw, Frances J. Acceleration of Technology Development by the Advanced Technology Program. National Institute of Standards and Technology Report NISTIR-6047, September 1997.

Link, Albert N. Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End. NIST Contract Report 97-722, November 1997.

Mansfield, Edwin. Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities. January 1996.

National Institute of Standards and Technology. The Advanced Technology Program: A Progress Report on the Impacts of an Industry-Government Technology Partnership. NIST-ATP-96-2, April 1996.

Ruegg, Rosalie. Guidelines for Economic Evaluation of the Advanced Technology Program. NIST Report NISTIR-5896, November 1996.

Ruegg, Rosalie. "The Advanced Technology Program's Evaluation Plan and Progress." Proceedings of the 7th International Forum on Technology Management. Kyoto, Japan: November 1997.

Spender, J-C. "Publicly Supported Non-Defense R&D: The USA's Advanced Technology Program." Science and Public Policy, February 1997, pp. 45_52.

Research Triangle Institute (RTI) Center for Economic Research. A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies; Preliminary Applications to Tissue Engineering Projects Funded form 1990 to 1996. NIST Contract Report, publication, July 1998 (expected).

Author Biography

Rosalie T. Ruegg, Director of the Economic Assessment Office of the Advanced Technology Program (ATP), develops and leads ATP's evaluation efforts to meet GPRA and program management requirements, advises on economic and business issues, and oversees ATP's peer review process performed by non-government industry, business, and economics experts. She has designed and presented in the U.S. and abroad numerous courses on economic topics for universities and government agencies, and has more than 50 publications, including a text book. Her interests center on technology policy, the economics of technological change, and performance measurement, and she is frequently an invited speaker on these and other topics.

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)