 |
|
|
|
|
|
|
|
|
 |
 |
 |
 |
 |
 |
 |
 |
|
|
|
||||||||||||||||||
|
Table of Contents
Abstract
Endnotes
List of References
Abstract
These guidelines are intended as a resource for members of the economic evaluation community who wish to perform economic evaluation studies of the Advanced Technology Program (ATP). To help members of this community frame the performance evaluation problem in the context of the ATP, background information is provided on the program, on its evaluation efforts and accomplishments to date, as well as its evaluation objectives and principal areas of interest. An overview is provided of ATP's procedures for considering proposals on program evaluation and the criteria against which proposed studies are scored. Suggestions are given for the content and format of proposals and reports on evaluation. The guidelines in no way constitute a solicitation for proposals on evaluation or an obligation to fund any proposals that might be submitted; they are provided to improve the quality and relevancy of evaluation studies, and the efficiency and effectiveness of the ATP/evaluator interface.
Keywords
advanced technology program; benefit-cost analysis; economic analysis; evaluation; impact assessment; performance metrics; research and development; technology assessment
Acknowledgments
For their helpful comments on the draft guidelines, appreciation is extended to Dr. Harold Marshall, Ms. Jeanne Powell, Dr. Richard Spivack, Ms. Connie Chang, and Dr. Gregory Tassey -- all economists at NIST. Appreciation is also extended to Dr. J.-C. Spender, Chair of the Department of Management, New York Institute of Technology, for the many insights he provided the author on technology policy and evaluation during the year he spent with the ATP. In addition, credit is due all of those who participated in two ATP evaluation planning workshops held in conjunction with the National Bureau of Economic Research -- particularly to Professors Zvi Griliches and Adam Jaffe -- for their excellent ideas and recommendations on developing ATP's evaluation program.
Executive Summary
The Advanced Technology Program (ATP) is one component of the nation's strategy to promote the economic well-being of the country. The ATP encourages industry to accelerate the creation and commercialization of enabling technologies that are expected to yield large economic benefits to the nation, extending significantly beyond the direct benefits to award recipients. The ATP provides this catalyst through research awards to companies for overcoming challenging technical barriers that currently stand in the way of realizing the potential benefits; in fact, ATP funding goes only for research that presses the scientific or technical state-of-the-art in the given industry sector. Businesses conceive, propose, and carry out the research projects for advanced technology development that are cost-shared by the ATP. The ATP awards are made to individual companies and to research joint ventures -- comprised of two or more companies, often in combination with universities and non-profit research laboratories -- in intense competition.
The ATP held its first competition in 1990, and to date has funded 280 projects totaling about $2 billion in advanced research, with more than half the funds coming from industry in cost share. Some of the early projects are now moving into the commercialization phase, but most of the projects were funded in the last two years and are still in the research phase. Though it is still early in the program, there is considerable interest in the likely outcomes -- both to meet internal management requirements for information, and to satisfy external policy related demands.
The ATP has emphasized program evaluation from the outset. To this end, the ATP utilizes the assistance of highly qualified academic and consulting economists and other experts in evaluation, in addition to its in-house staff -- in planning, modeling, and developing databases, in conducting surveys, case studies, statistical and econometric analyses, and in carrying out other studies that enhance program performance and contribute to the quality and reliability of performance metrics.
This summary report is provided to help the economics\evaluation communities better understand the ATP and its on-going activities in evaluation. The goal is to provide an efficient and effective means of communicating to those communities what the ATP is doing and what it wishes to do in the field of evaluation. All the economic evaluation work that the ATP supports is needs-driven and geared either towards increasing the expected net benefits of the program or measuring the net benefits, or both. Economic research of a general nature, not directly relevant to the ATP, is not supported. The guidelines provided herein are for researchers proposing and conducting ATP evaluation work. The document is merely explanatory and does not constitute a solicitation for proposals.
To help researchers frame the evaluation problem in the context of the ATP, background information is provided on the program and its evaluation goals, activities, accomplishments, and principal areas of interest. A model of the program is provided as a framework for thinking about the evaluation needs of the ATP. Researchers are reminded of the importance of tying proposed evaluation studies directly to ATP's mission and project goals; of bridging the gap between interesting theoretical work and the need for practical relevancy to ATP's evaluation problems.
The ATP expects that the evaluation studies it funds will not only further the understanding of the program directly, but will push the state-of-the-art in evaluation, resulting in tools, methods, and data of broad interest and leading to publishable papers in refereed journals. Examples of evaluation topics of current interest to the ATP include predicting and measuring spillover effects; assessing the benefits and costs of collaboration; understanding organizational effects of projects on participating firms that may extend beyond the project walls; R&D financing issues and the impact of the ATP on private investment in R&D; the development of new models and tools for measuring inputs and impacts; as well as other topics. Recommendations are given for the format and contents of proposals and reports. From time to time, the ATP plans to issue a supplement to the Guidelines updating evaluation studies underway and completed. Users of the Guidelines are invited to share their advice on ways to improve the document's usefulness as a reference for evaluation researchers interfacing with the ATP.
1. Evaluation Studies in the Advanced Technology Program (ATP) Context
The ATP's mission is to partner with U.S. businesses in high-risk scientific and technical research to develop enabling technologies with strong potential for producing broad economic benefits for the U.S. Its activities in the fields of economics and evaluation are purely in support of this mission. The ATP does not fund general or basic economic research outside of this scope.
In the context of its mission, the ATP has evaluation efforts underway in support of increasing and measuring the short- and long-run impacts of the technology development projects it funds, and, ultimately, of the entire program. As part of that effort, the ATP utilizes the assistance of academic and consulting economists and other experts in evaluation, in addition to its small in-house staff of economists. The assistance of outside experts is used in its planning activities, development of evaluation methods, models, and databases, conducting surveys, case studies, statistical and econometric analyses, and in carrying out other studies that enhance its performance and contribute to the quality and reliability of its performance metrics.
1.1 Relevancy to the ATP
Although it is the ATP's hope that the studies it funds will contribute significantly to the broad fields of R&D economics and performance evaluation, and be of keen interest to those working in these fields, the real purpose of ATP-funded studies is to advance the understanding of the workings and outcomes of the ATP. It is expected that researchers will be well informed about the ATP, that evaluation topics will have direct relevancy to the ATP, and that researchers will clearly draw potential implications of their research for the ATP in some detail. It is expected that successful studies funded by the ATP will both satisfy the researchers' interest in pressing the state-of-the-art of evaluation -- leading to publishable papers in peer-reviewed journals -- and satisfy the ATP's need for better evaluation tools and more reliable performance metrics. It is expected that ATP's management, policy makers, the broader R&D community, the economics community, and, in fact, all those interested in performance evaluation will benefit from the resulting studies.
1.2 In Tune with ATP's Mission
To evaluate a program or project, it is essential to understand the program mission or project goals, and to tailor the evaluation accordingly. Thus, proposers of evaluation studies are strongly encouraged to take the time to review and understand the ATP's mission. Researchers performing case studies of given projects and sets of projects in focused programs are strongly encouraged, in addition, to review their particular goals and objectives.
1.3 ATP's Mission
The ATP was created by the Technology Competitiveness Act of 1988 and received its first budget in 1990. According to the Statute establishing the ATP, the program was designed specifically for the purpose of "assisting United States businesses in creating and applying the generic technology and research results necessary to (1) commercialize significant new scientific discoveries and technologies rapidly, and (2) refine manufacturing technologies." It was further charged to "aid industry-led United States joint research and development ventures...including those involving collaborative technology demonstration projects which develop and test prototype equipment and processes...." The Statute emphasized that the funding should go "to support projects which are high risk and which have the potential for eventual substantial widespread commercial application." It further directed that "the Program focuses on improving the competitive position of the United States and its businesses, gives preference to discoveries and to technologies that have great economic potential, and avoids providing undue advantage to specific companies." (1)
The ATP's Rule (2), developed to implement the program, and other program descriptive materials, further define the program. The Rule emphasizes that the ATP-funded technologies must be "enabling" in that they must offer a "wide breadth of potential application and form an important technical basis for future commercial applications;" and they must be "high value, because when applied, they offer significant benefits to the U.S. economy." Recent ATP program descriptions define "enabling technologies" in terms of their "potential to generate economic and technical opportunities that stimulate broad-based benefits for the nation, beyond those accruing directly to ATP awardees." Enabling technologies are further categorized as "pathbreaking, infrastructural, and/or multi-use." (3) Pathbreaking technology is described as inducing revolutionary change in existing fields or opening up new fields of activity. Infrastructural technology is described as supporting the R&D, production, and business of entire industries. Multi-use technology is described as having many distinct applications. (4) "High-risk technologies" are defined by the ATP as "technical challenges which display significant recognized uncertainty of success, where success will dramatically change the future direction of technology and its market impact." (5) Note that while the ATP is constrained to funding research that is deemed to be high-risk in the sense that the scientific and technical work is highly challenging and fought with difficulty, it prefers, all else being equal, low business-risk circumstances.
In summary, the ATP enters into partnerships with industry to create and commercialize rapidly high-risk, enabling technologies that have the potential to improve substantially the productivity and the competitiveness of U.S. businesses and to yield large economic benefits to the nation. Reflecting the goal of broad-based benefits from the program extending beyond the direct award recipients, the ATP emphasizes the importance of a strong potential for spillover benefits from the technologies it funds.
1.4 Overview of ATP Partnerships with Industry
Industry leads in the ATP-style of industry/government partnerships for advanced technology development. Industry conceives, proposes, and carries out the research projects, and undertakes subsequent commercialization activities. The ATP evaluates technology development proposals submitted to it for their technical and business\economic merits in rigorous competitions. (6) It makes awards to the top-scoring projects, and subsequently monitors technical and business progress throughout the project duration. If desired by the companies and to the extent feasible and appropriate, the ATP project monitor will arrange NIST laboratory support or use of special government laboratory facilities to overcome tough technical obstacles standing in the way of success; hence, ATP awards may entail scientific/technical assistance as well as dollars.
Funding assistance is provided in the form of government sharing of "allowable costs" -- research costs only, exclusive of product-development costs -- through the formation of government/industry cooperative agreements. These arrangements entail a more active role for government than a grant. They also differ from a contract in that the government is not a customer for project output. Contracts and grants are typically NOT used by the ATP to fund its technology development projects. The ATP funds the research on a cost-reimbursable basis, quarter-to-quarter. Either the company awardees or the ATP may terminate a project at any time.
Companies can propose projects in ATP's technology development competitions either as single-company applicants or as research joint ventures. There is a cap on single-company awards of $2 million and 3 years. Joint ventures are not subject to a funding cap and the projects may last up to 5 years. The single-company award recipients must cover all of their indirect costs and may cover some of their direct costs; the joint-venture participants must provide more than half the total project costs.
A joint venture at a minimum must have two for-profit companies performing research and contributing to the cost-share. Many joint ventures involve more than two members. Proposing companies can, at their discretion, include other companies, universities, non-profit research laboratories, associations, and others in their projects, either as formal partners in a joint venture or as informal partners or subcontractors in either a joint venture or a single-company project.
The ATP is NOT a funder of streams of basic research in the traditional mode of the National Science Foundation. It is NOT a procurer of mission-related applied research in the mode of the Defense Department and the Department of Energy. It funds research; not product development. It operates in the middle ground between basic research and product development.
The ATP's focus is on civilian technologies -- not defense technologies. The ATP is normally NOT a customer for the results of the technology development projects it funds. The ultimate commercial outcomes are the results of the marketplace interplay of demand and supply forces. The ATP looks closely in project proposals at proposers' arguments that they intend to pursue future commercialization of the technologies, and assesses the strength of their proposed pathways to market and their commitment to follow through with commercialization plans. It relies on the presence of expected private returns to induce companies to be willing to cost-share the research with the ATP and subsequently to carry out commercial development of the new technology with private capital. But it selects only those projects for awards for which it thinks the potential social rate of return (the return to the nation) far exceeds the potential private rate of return on investment, and for which it thinks the private sector will either not do the project at all, or not within the critical time, or in the scale/scope, necessary to realize the potential societal benefits.
In absence of intervention, these research "spillovers" (or externalities), that accrue to others than the party that undertakes the research, are generally ignored in private investment decisions, resulting in a type of "market failure" characterized by an underinvestment in R&D from a national perspective. Capital market imperfections may be an additional factor that results in the funding of a less than socially desirable level of research. The ATP is one of the national policy strategies for overcoming R&D market failure. A successful ATP will in the long-run result in net societal benefits greater that would have resulted without it.
1.5 A Three-Dimensional Model of the ATP (7)
Key characteristics of the ATP have been captured by J.-C. Spender in a three-dimensional model of the program (Spender, The Three Dimensional Model of the Advanced Technology Program, NIST report in draft, October, 1996). The model, depicted in figure 1, measures on the O-TK axis the success of ATP-funded technology development projects to increase the level of scientific and technical knowledge (by overcoming high-risk technical challenges). It measures on the O-PR axis the net private returns on investment captured by ATP project participants. And, thirdly, it measures on the O-PG axis the net spillover returns to others in the economy who benefit from the project either through consumer surplus benefits derived by purchasing improved or cheaper goods and services at prices that do not fully capture their increased value, or by utilizing the knowledge derived from the ATP project to generate additional economic benefits without fully compensating the knowledge generators. The vectors mapped by OTPQ in the figure below describe a fully successful ATP project that delivers a combination of high-risk technical accomplishments, positive net private benefits, and positive net spillover benefits. The ATP seeks to maximize net social benefits (defined here as benefits to the nation) from the program, subject to the constraints that the state of the art be pressed, the effort be led and carried out by U.S. industry, and the benefits be "broad-based." In the parlance of ATP, the program seeks with each project it funds a "journey to Q." By encouraging companies to increase their development of enabling technologies that will strengthen the nation economically, the ATP represents a new streamlined, efficient model for government, whereby government facilitates rather than directs; taps industry's resources and know-how rather than bureaucracy to get the job done.
Note: Based on charts presented by J.-C. Spender in a forthcoming report.
2. Overview of ATP's Evaluation Program
2.1 External Demands for Evaluation of ATP
Externally, the ATP faces intense demands for performance measures. Many requests are driven by policy issues. Specific requests for evaluation results come frequently from individual members of Congress and their staff, from Congressional subcommittees, the General Accounting Office, the Executive Office of the President, the Office of Management and Budget, the Office of Inspector General, the Press, think tanks, industry, and others.
The ATP also receives many external inquiries about its evaluation program that are focused more on evaluation tools and methodologies than on specific evaluation results. For example, inquiries come from counterpart programs in other countries, from other Federal agencies, State and regional government agencies, universities, businesses, and consultants who, like the ATP, are concerned about the ins and outs of performance metrics and measurement methodologies. It is the ATP's policy to share with others through publications, presentations, and discussions its evaluation plan, methods, tools, data (excepting proprietary data), and results.
2.2 Internal Demands for Evaluation of ATP
There also are strong internal demands for evaluation of the ATP. These demands come from within ATP itself, as well as from its parent departments: the National Institute of Standards and Technology (NIST), the Technology Administration, and the U.S. Department of Commerce. Internally, the ATP is interested in performance evaluation, first, as a management tool to make the program better meet its mission and operate more efficiently; second, to find out how we are doing; and, third, to meet the many external requirements and requests for ATP program results. Finding out as early as possible what is working best and what is producing results can inform the project-selection phase of the ATP, improve the management of the program, and likely increase the long-run success of the ATP in meeting its economic and technical goals. Determining if the ATP is in the long-run meeting its goals and realizing a large return for the U.S. taxpayer is, of course, essential to informing technology policy.
2.3 Mandated Requirements for Evaluation
Title II of the American Technology Preeminence Act of 1991 (P.L. 102-245, enacted in 1992) directed that a comprehensive report on the results of the ATP be submitted to each House of the Congress and the President not later than 1996. (This report, The Advanced Technology Program: A Progress Report on the Impacts of an Industry-Government Technology Partnership, was delivered in April 1996.)
In addition, the ATP, like other federal programs, is subject to the evaluation requirements of the 1993 Government Performance and Results Act (GPRA). The GPRA resulted from a bipartisan effort to improve accountability, productivity, and effectiveness of Federal programs through strategic planning, goal setting, and performance assessment. The ATP/NIST, like other Federal government agencies, is developing assessment plans and techniques, and carrying out evaluation studies in compliance with the GPRA.
2.4 What and When to Measure
The three dimensions of ATP represented in figure 1 -- funding projects that (1) add to the scientific and technical knowledge base, (2) result in and hasten technology-based commercialized products and processes, and (3) ultimately yield widespread economic benefits beyond the direct awardees -- condition and inform ATP's evaluation plan. The program's dimensions inform the decisions of what to evaluate, when the evaluation can occur, and how it can be done. Also important to designing ATP's evaluation plan, but not depicted in the static model of figure 1, are the process dynamics within each dimension and at the points of transition from one dimension to another.
The fact that the objective of the ATP is to maximize "Q," rather than any single one of the three program dimensions means that the metrics in each dimension cannot fairly be taken in isolation of the others. For example, the ATP is charged not with selecting projects to maximize contributions to the scientific and technical knowledge base, but rather projects that press the state of the art while satisfying the other two dimensions of the program. Thus, one target of program metrics is the extent to which the research funded by ATP adds to the nation's scientific and technical knowledge base, but the interpretation of results must be made within the context of the other dimensions: What is the likelihood that award recipients will pursue commercialization of the resulting technology after the research is completed, and what is the likelihood that there will be substantial spillover benefits to the nation not appropriated by the companies carrying out the project? If one, on the other hand, focuses on the extent to which, and speed with which, the awardees translate their new technical know-how into products, processes, and services with commercial significance, it is important to take into account the degree of innovation entailed and appropriability problems. Similarly, if the focus is spillover benefits, one needs also to consider private benefits. And one always needs to question what the level of benefits would likely be without the ATP. No single program dimension is alone the end-all of ATP or the focus of its program metrics. Measuring the level of "Q" is the quest of program metrics, but that is, of course, a complex and exceedingly difficult challenge that we are not likely to achieve except piecemeal, in stages and by degrees.
Not only do these three dimensions serve to constrain one another, but they are also linked dynamically in cause-and-effect, tension-bound, and time-dimensioned relationships. Projects that push further on the innovation envelope, for example, may present greater-than-average marketplace challenges to awardees in achieving their commercialization goals. Failure to achieve commercialization goals in a timely way may jeopardize the achievement of most of the targeted spillover benefits. The cast of characters needed to accomplish technical goals is likely different from those needed to achieve business and economic goals. Innovators working with certain technologies in industries with certain characteristics are more likely to achieve spillover benefits than others. Difficult-to-capture intellectual property in the technology may weaken the chances of commercial follow-through, but strengthen the chances of spillover benefits. In short, there is much to understand about project paths, transition points, diffusion patterns and rates, and underlying cause-and-effect relationships that combine to determine program outcomes.
Ideally, bottom-line metrics for long-term program impact will include measures of private rates of return on research investment, social rates of return, and "public rates of return," that is, the social-rate-of-return component attributable to the ATP. But, realistically, it is expected that proposed evaluation studies will tackle parts of the problem; not all of them at once. Acceptable projects may address only one of the three dimensions, or only certain aspects of project paths, transition points, diffusion patterns, and the like.
2.5 ATP's Evaluation Strategy
To square the often pressing demands in the short run for evaluation results with the reality that patience is required to realize and validate empirically long-run program outcomes, the ATP has adopted a time-dimensioned, multi-part evaluation strategy. This strategy has thus far entailed (1) the use of short-run indicators of program progress towards targeted long-term goals; (2) descriptive project progress updates; (3) case studies of projects and groups of projects; (4) projections of expected long-term impacts conducted ex ante of actual impacts (with acknowledgment of the limitation imposed by the large uncertainties entailed in such projections); (5) surveys of company plans, technical and business progress, and assessments of short-term business impacts; and (6) statistical, descriptive profiling of projects, participants, technologies, and intended applications of the technologies. At the same time, the ATP is working to improve the tools of longer term evaluation, to collect critical data that will be needed to implement this activity, to build feedback loops between project selection and project evaluation, and in other ways to enlarge and improve its capabilities both of achieving the long-run technological and macroeconomic goals of the program and of measuring specific performance.
2.6 Evaluation Techniques Currently Used by the ATP
Peer review of proposed projects against project selection criteria is applied up-front in ATP's rigorous awards competitions. The goal is to select projects that are likely to achieve the ATP's mission. It is sometimes overlooked that this step is a form of project evaluation, because the assessment occurs ex ante, with the emphasis on achieving the program's mission as opposed to measuring performance towards achieving the mission. At this initial awards stage of the ATP project cycle, "expert" evaluators are charged with using their resident knowledge and judgment to evaluate proposals against published selection criteria and to identify those that appear to have strong potential for achieving a high "Q" value as illustrated schematically in figure 1. Scientists, engineers, business people, and economists -- all of whom agree to abide by non-disclosure and avoidance of conflict-of-interest rules and sign statements to that effect -- serve as peer reviewers.
The opportunity exists to feed back, as appropriate, the results of performance evaluation to the peer-reviewers, with the objective of informing the initial project selection stage so that future success conditions are enhanced. A step in this direction would be a more explicit evaluation of the spillover potential of proposed projects by comparing project/proposer attributes against an identified set of factors that are found likely to increase or decrease spillover effects. (8)
Real-time project monitoring is used by ATP to determine the progress of funded projects currently underway against technical, business, and economic goals. This is also sometimes overlooked as entailing a component of project evaluation, perhaps in part because the project monitoring task also encompasses project management and administrative activities, and in part because it provides just pieces of the information needed for evaluation. Project monitoring, however, is important to evaluation, in that it brings ATP staff face-to-face with the performing entities and provides the opportunity for the staff to become intimately familiar with the projects, particularly the technical work. Thus informed, the project monitoring staff (9) often serve as informational sources for project evaluation case studies.
Data collection and analysis is used by the ATP to track project progress, to understand the overall project portfolio statistically , to assess results, and to contribute eventually to measuring long-run outcomes. Data come primarily from third-party surveys, ATP special studies, and ATP's internally administered "business reporting system."
Case studies of ATP-funded projects and groups of related projects have been performed at various stages of the project life cycle to capture progress to date, measure short-to-medium term impacts at the level of the firm, and, in some cases, provide the information needed to interface with macroeconomic models for projecting national impacts.
Surveys of ATP project participants have been used to capture and report statistically on short-run results -- particularly early business-related progress, as well as to gather feedback from program participants on their satisfaction with working with the ATP.
Econometrics and other statistical analysis techniques are being used by the ATP to shed light on underlying cause-and-effect relationships, such as spillover mechanisms, and to project impacts from the firm level across the entire economy using large-scale macroeconomic models.
Modeling has been used to provide frameworks for better understanding and assessing the program.
2.7 Overview of ATP's Evaluation Accomplishments through August 1996
The ATP's Economic Assessment Office has developed ATP's approach to economic evaluation in consultation with leading economists in the field. The Economic Assessment Office in conjunction with the National Bureau of Economic Research has held periodic workshops with leaders in evaluating technological impacts to obtain feedback on the ATP's evaluation approach and plans, and to solicit advice on future directions (Workshops chaired by Professor Zvi Griliches of Harvard University were held at NIST in December 1994 and September 1995). Input to the ATP from those workshops, as well as from other sources, is reflected in the outline of topics of interest given in section 3.
An early background planning study was performed for the ATP by Albert Link, consultant (A. Link, Measuring the Economic Impact of the Advanced Technology Program; A Planning Study, January 1992). ATP's evaluation plan has been described in internal documents and in a number of internal and external presentations (e.g., R. Ruegg, "R&D Evaluation: Methodological Issues," American Evaluation Association 1994 Annual Meeting), and summarized in three NIST special reports (Setting Priorities and Measuring Results at the National Institute of Standards and Technology, January 31, 1994; Delivering Results, June 1995; and The Advanced Technology Program: A Progress Report on the Impacts of an Industry-Government Technology Partnership, April 1996). The evaluation plan is updated, expanded, and presented in more detail in a forthcoming ATP report (R. Ruegg, The Advanced Technology Program's Economic Evaluation Plan and Progress in Implementation, draft report in preparation). The 3-D model of the ATP presented and discussed earlier has provided a useful framework for recent planning of program evaluation. (J.-C. Spender, The Three Dimensional Model of the Advanced Technology Program, September 1996 draft report).
2.7.2 Data Collection and Analysis
The Economic Assessment Office has established several databases in support of program evaluation. The "Awards Database" is used for statistical profiling of funded projects, technologies, and participants. It provides answers to numerous questions about what, where, and whom ATP is funding and, therefore, is useful as a management tool and as a resource for project evaluation. It is supplemented by an " Applicants Database " -- confidential because the ATP does not release the names of applicants, only awardees -- but potentially useful for evaluation. An integrated set of databases, called the " Business Reporting System " is an important component of ATP's data compilation in support of project evaluation. These data, compiled systematically by electronic survey of project participants, track the evolution of projects towards achieving their business and economic goals. The Business Reporting System consists of several parts: An initial report on planned application areas for the technology and planned strategies for commercialization; quarterly short reports that update participant status and call out major business developments during the period; and an annual report on progress towards implementing the commercialization strategies and on short-term economic impacts of the projects, including, but not limited to, early sales revenues, impacts on R&D cycles, collaboration effects, intellectual property creation, and company changes in jobs they attribute to their ATP project. Additional sections of the Business Reporting System now under development are a close-out report which completes the reporting of developments during the course of the research project, and updates future plans for commercialization; and a post-project report , to be administered three times (every other year) over the six-year period following project completion, to capture future commercialization efforts and data supporting assessment of technology diffusion and spillover effects. (In addition, quarterly technical reports are filed with the ATP for each project during its ATP-funding life, but these are unique for each project and the information does not lend itself to database format and statistical analysis.)
The Business Reporting System has been implemented for all projects funded by ATP since FY 93. At present, there are 450 organizations in the database, representing about 200 projects. (Approximately sixty projects funded prior to FY 93 are not in the Business Reporting System, but partial data have been captured for some of these projects.) A recent ATP staff paper analyzed some of the available data from the Business Reporting System and reported early results (J. Powell, The ATP's Business Reporting System: A Tool for Economic Evaluation, September 1996). Additional databases capture data resulting from non-routine, in-house studies by ATP staff, and intellectual property developed in the ATP-funded projects.
2.7.3 Broad Surveys of Project Participants
The ATP has sponsored two broad surveys of funded companies by third-party contractors using telephone interviews. The first survey included 26 organizations participating in the eleven projects funded by ATP in its first competition, one year into the projects (Solomon Associates, The Advanced Technology Program; An Assessment of Short-Term Impacts: First Competition Participants, February 1993). The second, larger survey included 125 companies and consortia who participated in the ATP from 1990 to 1992 (Silber and Associates, Survey of Advanced Technology Program 1990-1992 Awardees: Company Opinion about the ATP and Its Early Effects, January 30, 1996). At the time of the second survey, most of the participants covered were well into their research projects; a few had reached completion.
2.7.4 Background Studies on Evaluation
Edwin Mansfield prepared a background paper for the ATP on estimating social and private returns from innovations, preparatory to performing a set of project case studies (E. Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities, January 1996. Professor Adam Jaffe prepared a background report, now in ATP review, on spillover effects (A. Jaffe, Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program, October 1996). The report models spillover effects and identifies factors that influence whether a given project is more or less likely to lead to large spillovers. ATP's staff has also published general methodological pieces on economic evaluation (R. Ruegg, "Economic Evaluation Methods," Handbook on Energy Economics, 1996; and "Risk Assessment," The Engineering Handbook, 1996).
Several case studies of ATP projects, done in the early years of ATP by Professor Albert Link, focused on research efficiency in research joint ventures, and several additional studies that update those earlier cases are nearing completion (A. Link, Economic Analysis of the Printed Wiring Board Research Joint Venture, September 1996 draft report, updates his earlier case study of the same joint venture project published in April 1993).
The ATP is experimenting with combining microeconomic case study with the use of macroeconomic models to develop national economic impact projections for projects. Two studies taking two different approaches to using the REMI model for macroeconomic projections are nearing completion: (CONSAD Research Corporation, The Application of a Macroeconomic Interindustry Model to an ATP Joint Venture Project; A Case Study of the Development of Advanced Technologies and Systems for Controlling Dimensional Variation in Automobile Body Manufacturing (the 2 millimeter Project), July 1996 draft report; and E. Robles, Using the REMI Model to Estimate the National Economic Impacts of the 2 millimeter Project, October 1996 draft report).
2.7.6 Studies of ATP Counterpart Programs
The ATP collects information on foreign programs that are similar to ATP with two objectives: (1) it hopes to learn from the experience of those programs, and (2) it needs data from them to perform "determinations of eligibility" of foreign-owned U.S. subsidiaries to participate in the ATP. One factor in the eligibility is whether U.S. companies are treated the same as any other company in ATP-similar programs that operate in the country of origin of the ultimate parent company of the U.S. subsidiary that is applying to the ATP. An ATP report in progress compares the features of a number of counterpart programs with the ATP (C. Chang, A Comparison of the U.S. Advanced Technology Program with Similar Programs Abroad, October 1996 draft report).
A study based on participant interview data was recently conducted of ATP's impact on accelerating technology development by cutting the time for starting projects and by compressing research cycle time. The report is in publication (F. Laidlaw, "Acceleration of Technology Development by The Advanced Technology Program: The Experience of 28 Projects Funded in 1991, National Institute of Standards and Technology," October 1996 draft report). Cycle-time reduction is of keen interest to the ATP because ATP's authorizing legislation calls for it to accelerate R&D and the commercialization of resulting technologies.
3. Evaluation Research Topics of Interest to the ATP
The ATP wishes to improve and extend its evaluation capabilities, tools, and metrics. The areas listed below are of special interest. Researchers who wish to propose studies in these areas are reminded to relate the proposed research directly to the ATP and to consider that the ATP is interested in specific implications of the research for the program. Proposed research is expected to assist in the measure of ATP's impacts, and may also inform ATP funding decisions. The ATP considers it outside its scope to fund general studies in these areas, not of direct relevance to it.
3.1 Spillover Pathways
3.2 Research Collaborations
3.3 Financing Issues
3.4 Economic Modeling and Methodology
3.5 Project and Program Impact Assessment
3.6 Other Topics
The ATP does not consider the above listing of topics necessarily an exhaustive treatment. The listing is intended merely to convey to the evaluation community a general idea of the current evaluation interests of the ATP. It is expected that experts working in the area will likely suggest research topics not explicitly referenced here that nevertheless might be of great value to evaluating the ATP.
4. ATP's Plans for Reviewing and Funding Evaluation Proposals
4.1 Evaluation of Proposals by an ATP-Convened Board
Approximately at six-month intervals, the ATP plans to convene a Source Evaluation Board (SEB) to consider proposals submitted to the ATP by outside researchers on topics pertaining to the evaluation of the ATP's impacts on the economy. The SEB, comprised of government economists and technologists in and outside of the ATP, will review each candidate proposal. The SEB will evaluate the merits of the proposed study's objective and approach; the relevancy of the study to ATP's evaluation concerns as outlined in section 3; cost; qualifications of the performer; and potential obstacles to success. With the advice of the SEB, the ATP will decide whether or not it wishes to pursue the work as proposed, and what modifications to suggest. If it wishes to support the work, it will decide the most appropriate approach.
4.2 Role of the National Bureau of Economic Research (NBER)
Some studies proposed to the ATP and evaluated by the SEB will be funded as task orders under an "umbrella" contract with the NBER, which is serving as a general contractor to the ATP for the purpose of undertaking studies that press the state-of-the-art of economic evaluation. The NBER will be asked to review each draft task order submitted to it by the ATP, assess the feasibility of undertaking the study as described, decide which it will support, and recommend any necessary changes to those it agrees to undertake. For projects funded by the ATP through task orders with the NBER, the NBER will serve as administrator, tracking progress against project milestones and monitoring costs in conformance with the terms of its contractual Statement of Work with NIST. Sign-off approval of project work will be performed jointly by the President of the NBER, or the NBER's ATP-Project Director, and by the ATP.
As background, the NBER is a private, nonprofit, nonpartisan research organization committed to advancing and disseminating unbiased economic research among public policy makers, business professionals, and the academic community. The NBER coordinates research by more than 380 economics professionals at leading universities and business schools throughout the U.S., concentrated in four areas of research: developing new statistical measurements, estimating quantitative models of economic behavior, assessing the effects of public policies on the U.S. economy, and projecting the effects of alternative policy proposals. World renowned researchers in the field of the economics of science and technology, including experts in evaluation, are associate members of the NBER. The NBER is unique in that, unlike other economic research centers whose activities consist primarily of the research of a relatively small resident staff, the NBER has links to hundreds of leading professional economists with a wide range of interest and expertise, presenting a great range of resources with which to meet the ATP's evaluation needs.
5. Advice on Preparing Evaluation Proposals and Reports for the ATP
5.1 Desired Proposal Characteristics
It is expected that successful proposals of economic research to the ATP will exhibit the following characteristics:
5.2 Desired Proposal Contents
Proposers of economic research are asked at a minimum to include the following elements:
5.3 Criteria Against which Evaluation Proposals will be Judged
Proposals will be judged by the SEB against the following criteria:
5.4 Minimum Requirements for Contractors
Contractors of economic research for the ATP will at a minimum be subject to the following requirements:
5.5 Requirements for Evaluation Reports
Typically, researchers should plan on submitting several drafts of their reports in sequence, with a feedback loop for comments and revision, prior to the final report submittal.
Reports would generally be expected to include the following elements:
Researchers are usually asked to submit both a hard copy of their report and an electronic file copy. Figures and tables should either be integrated electronically in the text or camera-ready hard copy should be provided.
Questions, comments, or proposals on evaluation can be submitted to the ATP at any time at the following address:
Economic Assessment
Office, Advanced Technology Program
You may wish to contact Rosalie Ruegg, Office Director: 301-975-6135, e-mail rosalie.ruegg@nist.gov; Jeanne Powell, Group Leader: 301-975-4196, e-mail jeanne.powell@nist.gov; Richard Spivack, Economist: 301-975-5063, e-mail richard.spivack@nist.gov; or Connie Chang, Economist: 301-975-4318, e-mail connie.chang@nist.gov. The Office FAX is 301-921-6319.
In support of its technology development activities, the ATP maintains a toll-free "hotline" number 1-800-ATP-FUND. Application kits for applying to ATP's announced competitions for awards to develop advanced technologies (not evaluation research!) may be requested by calling the hotline number. The ATP also makes information available through its World Wide Web home page at https://webarchive.library.unt.edu/eot2008/20080916072554/http://www.atp.nist.gov, where you can find a cross link to the Economic Assessment Office's home page, and copies of some of the evaluation reports.
Your comments are welcomed.
Endnotes
Chang, Connie K. N. A
Comparison of the U.S. Advanced Technology Program with Similar Programs
Abroad, National Institute of Standards and Technology, October 1996,
draft report.
|
 |
ATP website comments: webmaster-atp@nist.gov / Technical ATP inquiries: InfoCoord.ATP@nist.gov NIST is an agency of the U.S. Commerce Department |
 |
