Abstract

Acknowledgements

Introduction

Business Progress--Some Early Results

• Technologies with Diverse Applications
• Business Goals
• When Are Revenues Expected?

Short-Term (R&D Phase) Impacts--Some Early Results

• Early Commercialization Impacts
• Stimulation of High-Risk R&D
• Collaboration Impacts
• Information Dissemination

Conclusions

Future Work

ABSTRACT

THE ATP'S BUSINESS REPORTING SYSTEM: A TOOL FOR ECONOMIC EVALUATION-- Use of an electronic survey instrument to gather information on an annual basis about business progress and economic impact of advanced research projects cost-shared by government and industry (Economic Assessment Office, Advanced Technology Program, USA)

Program evaluation has been an important part of the Advanced Technology Program (ATP)--a government-industry partnership--from its inception. The ATP's Economic Assessment Office (EAO) has implemented approaches for tracking short-term indicators of program results, while also developing new state-of-the-art evaluation tools for measuring long-term economic impact. An electronic survey for tracking evolution of projects towards achieving their business and economic goals is a core part of ATP's program evaluation framework. This comprehensive Business Reporting System consists of several parts. At the beginning of the project, project participants report on their planned application areas for the technology and strategies for commercialization. Annually they report on progress towards implementing their commercialization strategies and on short-term economic impacts of the projects, including early sales revenues, shortened R&D cycles, collaboration effects, intellectual property creation, and early job creation. Additional sections of the Business Reporting System now under development will focus on the post-ATP funding period, capturing technology commercialization and diffusion. Over the following six-year period, participants will report three times, increasing the emphasis on economic impacts of the ATP-funded technology to the nation.

The Business Reporting System consists of a series of largely closed-ended questions and responses in electronic, database form. This database system is part of an integrated set of databases that supports comprehensive analyses covering all participants in ATP-funded research projects across nearly any desired subgroup. For short-term analyses, this integrated system provides a flexible analysis tool for generating reports of business progress of projects and early economic impacts. It also helps the ATP identify promising candidates for in-depth case studies of early ATP economic impact. In the longer term, the Business Reporting System will support efforts by independent research economists to measure broad economic benefits of this Federal program to the nation.

For the reporting period ending December 31, 1995, there were nearly 400 organizations in the Business Reporting System. Although most of the participants/projects reporting to date are still in the early R&D phases, adequate data exists to 1) illustrate some of the types of analyses possible, 2) provide a snapshot of commercial opportunities that may be expected to result from the awards portfolio and an approximate time line, 3) give evidence that companies are taking necessary steps for successful future commercialization, 4) provide early indication that the non-proprietary information developed with ATP funding is contributing to a shared knowledge base, and 5) indicate patent filings attributable to the research projects. For projects which have received at least one year of funding, participants are reporting significant acceleration of R&D, stimulation of beneficial collaboration, and deepening of their R&D effort as a result of ATP funding.

The Advanced Technology Program, administered by the National Institute of Standards and Technology of the U.S. Department of Commerce, assists U.S. businesses on a cost-share basis to develop high risk, enabling technologies for the purpose of stimulating U.S. economic growth and competitiveness of U.S. businesses. Authorized by the U.S. Congress in 1988, the ATP started operation with a small appropriation of $10M under the Bush Administration in 1990 and grew rapidly to a budget of $341M in 1995 under the Clinton Administration. To date the ATP has received 2,210 proposals in 22 merit-based competitions, and has announced 280 awards (178 for single-company projects and 102 for joint venture projects) for nearly $2 billion of research, with $970 million provided by the ATP. Although the ATP funds only research, funding decisions take into account business and economic merit of proposals as well as scientific and technical merit.

ACKNOWLEDGEMENTS

The author wishes to thank all those who contributed to this report. Special appreciation is extended to Ms. Rosalie T. Ruegg for her thorough and detailed reviews over the course of preparation of the original abstract submitted to the Conference on Comparative Analysis of Enterprise Data '96 and the preparation of the conference presentation and final paper which followed. Many thanks are due to Dr. Richard N. Spivack and Dr. John P. Gudas of the Advanced Technology Program and to Dr. Stephen F. Weber of the Applied Economics Office, BFRL, for their insightful comments and suggestions which have been incorporated in this report. Thanks are due to Mr. Ray Mele for preparation of the graphical illustrations for the presentation in Helsinki and subsequent revisions for this paper. Finally, appreciation is extended to the more than 250 ATP project participants whose cooperation in preparing timely and thorough reports of business plans for their ATP-funded technologies and early impacts of ATP funding made this study, along with future ones, possible.

The ATP's Business Reporting System: A Tool for Economic Evaluation

Introduction

The Advanced Technology Program (ATP), administered by the National Institute of Standards and Technology, assists U.S. businesses, on a cost-share basis, in developing high risk and enabling technologies for the purpose of stimulating U.S. economic growth. The ATP is similar in some ways to European programs that support industrial R&D; for example, the LINK Program in the U.K. and Business-Oriented Technology Program (PBTS) in the Netherlands.

The ATP funds research projects proposed by U.S. companies through a highly competitive selection process. Funded projects are expected to advance the state-of-the-art in overcoming challenging technical barriers to solving problems or exploiting opportunities with substantial economic significance. One condition of ATP's funding is that U.S. industry both lead the research and show a promising path to timely commercialization of resulting technologies. A further condition is that the ultimate expected outcome generate spillover benefits beyond those accruing directly to funding recipients.

Program evaluation has been an important part of the ATP from its inception. The ATP's Economic Assessment Office (EAO) has implemented multiple approaches for tracking the short-term indicators of program results, while also developing new state-of-the-art evaluation tools for measuring long-term economic impact. (Other evaluation approaches employed by the ATP include peer review, case study, third-party survey, and econometrics and other statistical analyses.) An electronic survey for tracking evolution of projects towards achieving their business and economic goals is a core part of ATP's program evaluation framework. This comprehensive Business Reporting System (BRS) consists of several parts. At the beginning of the project, project participants report on their planned application areas for the technology and strategies for commercialization. They report annually on progress towards implementing their commercialization strategies and on short-term economic impacts of the projects, including early sales revenues, shortened R&D cycles, collaboration effects, intellectual property creation, and early job creation. Additional sections of the BRS now under development will focus on the post-ATP funding period, capturing technology commercialization and diffusion. Over the six-year period following project completion, participants will report three times, with increasing emphasis on economic impacts of the ATP-funded technology to the nation.

The BRS consists of a series of largely closed-ended questions and responses in electronic, database form. This database system is part of an integrated set of databases that supports comprehensive statistical analyses covering all participants in ATP-funded research projects. For short-term analyses, this integrated system provides a flexible analysis tool for generating reports of business progress and early results.

Data are collected at the project participant level (from each company, university, and not-for-profit organization participating in the project as a single-company award recipient or as a joint venture partner). This ensures maximum confidentiality and detail concerning the contributions of different types of organizations to the R&D effort and the multiple commercialization paths of ATP-funded technologies. When fully implemented, the BRS database will also enable researchers to look inside the "black box" of the firm at previously unstudied characteristics of firm behavior and their relationship to output and employment at the industry and national levels.

In combination, these features of the BRS enable it to meet its three objectives:

1. Monitor business progress against plans for achieving:
   • Commercialization

   • Broad-based economic benefits

2. Measure short-term (R&D phase) impacts

3. Build a database to support long-term (post ATP-project phase) evaluation of economic impact

The BRS has been implemented for all projects funded in FY93 and later, from their inception. As of December 31, 1995, there were nearly 400 organizations, including all but the first 60 projects funded and a few not yet actually started, of the 280 projects funded to date, in the data system. Following is a brief picture of some of the early results to date in meeting the first two objectives.

Business Progress--Some Early Results

The BRS tracks business progress for each application of the ATP-funded technology that each participating company plans to pursue--whether it plans to manufacture a product in-house, adopt a new or improved manufacturing process, perform a service, license its technology, or some combination of these, possibly in alliance with strategic partners. The following early business progress results are based on the set of reports filed December 31, 1995. The data set was restricted to reports judged relatively complete (covering 70% of organizations in the BRS at that date) and to companies expressing an intent to commercialize the technologies in these applications eventually (at least one company in each project).

Technologies with Diverse Applications. Consistent with the ATP mission of funding enabling technologies with potential for significant broad-based benefits, most ATP-funded technologies have multiple commercial applications. For example, projects in the materials technology area may have applications in the industrial equipment, motor vehicle, construction, petroleum drilling, and electronics industries, or some combination of these.

The ATP uses its own 5-digit technology code system in parallel with Standard Industrial Classification (SIC) Codes to establish technology-application/user industry linkages. Each company self-selects a 5-digit technology code and 4-digit SIC code to describe each application of that technology. Figure 1 summarizes the two coding distributions and provides a picture of the linkages between ATP-funded technologies and their application areas. Companies working in the materials technology area, for example, have designated eight different SIC industry categories (2-digit level) as applications targets.

Business Goals. At the beginning of a project, companies identify key attributes of the technology from a business perspective and establish corresponding baseline values and end-of-project goals. This information establishes a benchmarking capability and means of measuring the technical accomplishments of ATP projects in the context of their broader business objectives. In general, most cited attributes involve performance or quality improvements, cost reduction, or some combination of these. In addition, acceleration of the R&D phase, resulting in time-to-market reduction, has consistently been cited as an important goal in the BRS and other ATP surveys. The ATP's mission specifically includes accelerating technology development and commercialization.

ATP-funded companies have cited performance improvement as a goal more frequently than cost. (See Figure 2.) Performance-improvement goals ranging from 5% to more than 500% increases over current performance capabilities were reported for 60% of the applications. Cost reduction was cited as a major goal for 40% of the applications. Time-to-market reduction of a year or more was an additional goal for 86% of the applications.

Commercialization Strategies. Most ATP-funded companies plan to produce a product or service themselves, based on the technology platform they develop in the project, in their own existing or planned production facilities. (See Figure 3.) In-house production is the focus for 70% of applications. For 26% of applications, companies plan to implement a new process in-house as their primary strategy; for 44% of applications, companies indicate that licensing of the technology to other companies is a primary or secondary commercialization strategy.

Companies report different strategies for different applications; for example, in-house production in areas where they have current manufacturing capability or market awareness, licensing in others. Some companies appear to be planning multiple strategies to achieve maximum market penetration for a given application.

When Are Revenues Expected? Given the advanced nature of the research being performed in ATP projects, there are often significant R&D hurdles remaining when ATP funding ends. Confirming the long-term nature of ATP-funded R&D projects, and the need for patience before expecting to see wide-spread economic impacts from the program, companies report that 90% of their applications will not reach the marketplace at all until after ATP funding ends. (See Figure 4.) More than 30% are not expected to result in revenues until at least four years after ATP funding ends. The ATP does not fund product development, but there are sometimes product spin-offs during the course of research.

Short-Term (R&D Phase) Impacts--Some Early Results

As of September 1995, the first projects to enter the BRS had completed a full year and submitted their first annual (anniversary) report. The following analyses are based on the first anniversary reports received from 41 organizations (19 in single-company projects; 22 in joint venture projects) funded in FY93. These reports cover 89% of active participants in FY93 projects at their first anniversary dates.

Early Commercialization Impacts: Some companies have taken advantage of early spin-off product opportunities or implemented processes that stem from partial achievement of the goals of the ATP project. Table 1 summarizes the early commercialization and employment impacts reported by the FY93 projects.

A total of 162 new jobs have been created across the 18 projects and 25 companies reporting employment effects. Although most of these new jobs directly supported the R&D effort, the following comments seem to indicate that the ATP award has enabled a smoother hiring process and a more coordinated, longer-term planning approach to hiring and resource management than likely would have occurred without it.

Anecdotal Comments:

"The ATP funding allowed us to pull together a cross-department team.... This is usually not done and had the effect of broadening interests and expertise."

"The mix of employees has changed. We have hired people with manufacturing experience. We also have added a die-cutter, a manufacturing process manager, and some production people. This is creating a new sub-culture, because up till now, we only had a scientific culture."

"The ATP award has made it easier to attract highly skilled scientists..."

Stimulation of High-Risk R&D. ATP seeks to fund projects that would not be funded by the private sector alone in a timely way due to the relatively high risk and high ratio of social benefits to private benefits. Thus, in its evaluation process, ATP seeks to determine if government funding is displacing private capital. If it appears likely that the project would be funded privately, it becomes important to assess whether the timing, scale, or scope of the research will be different because of ATP funding, with resulting net benefits to the U.S. public. The BRS anniversary report section contains a number of questions that address the effects of ATP finding on the timing, level of private funding, and type of R&D performed.

The FY93 first-year anniversary reports indicate that many of the projects would not have occurred without ATP funding or that the R&D is significantly further along as a result of it. (See Figure 5.) This result is consistent with recent third-party surveys. Eighty-eight percent of the FY93 awardees indicated they were ahead in the R&D cycle after one year of funding as a result of the ATP award; 34% of these indicated there would be no project without ATP.

More than 75% of companies report that they have put in more funding (amounting to $22 million to date) as a result of the ATP award than they would have without it (See Figure 6), indicating that ATP awards have stimulated additional industry spending, not displaced it. Single-company projects and small businesses (in single-company projects and joint ventures) indicate greater leveraging effects from the ATP award on the level of industry R&D investment than do joint venture participants and medium-to-large businesses. Note that the effects on level of industry R&D investment (Figure 6) are reported separately from the effects on acceleration (Figure 5) and the effects on project size (Figure 8), but the series of effects need to be viewed as a whole for proper perspective.

Seventy-six percent indicated that the ATP award had changed the nature or effects of the R&D performed by the company in the ATP-funded area. (See Figure 7)

The respondents indicating a change in the R&D performed further identified the most significant areas of change. (See Figure 8.) Nearly 90% of these respondents reported that the ATP award enabled them to broaden the scope of their R&D; over 80% indicated that ATP has stimulated research with higher technical risk than the companies would have pursued alone. In essence, the ATP shares the technical risk along with the costs of the R&D projects, enabling companies to undertake more ambitious R&D projects, with more challenging goals, than they would take on alone. Nearly three-fourths of these same respondents indicated an increased interest in collaborations with other companies for performing R&D as a result of their ATP award.

Collaboration Impacts. To date, the ATP has funded over 100 industry consortia involving formal joint venture agreements. In addition, a large number of the single-company projects involve subcontracts and strategic alliances with other firms. The BRS contains a number of questions that help assess the benefits and costs of these relationships, and the potential for a structural change in the way industry R&D is conducted in the U.S.

To start with, participants are asked whether the project has benefitted from collaboration with other organizations. Nearly three-fourths of the companies gave a positive response (including many in single-company projects with informal collaborations); three-fourths of those giving a positive response further indicated that ATP was responsible for the collaboration "to a great extent." (See Figure 9.)

Anecdotal Comments (from Single-Company projects):

"... by working with collaborators rather than developing some of the capabilities in-house, we have maintained the flexibility to switch approaches (by switching collaborators) if a different approach proved to be better suited....Avoided the time and cost to develop human capital in areas where others have a strong expertise."

"ATP has stimulated ongoing collaboration with companies and individual subcontractors that will continue long after ATP funding has ended."

"Collaboration was primarily with a subcontractor doing usability studies for us. They gave us a unique perspective on the end-users' concern and how we should address these in our system design."

"Collaboration with research groups which has resulted from our ATP involvement has significantly expanded the list of possible applications for the ... technology and has accelerated the development of products for these applications."

Project participants that have previously indicated a positive experience with collaboration (30 companies) are then asked to evaluate a number of specific potential effects of their collaborations. Results have been grouped into two categories: effects significantly or moderately enabled by collaboration in the ATP project for 80% or more of the participants in Figure 10 and effects less enabled by collaboration in Figure 11.

The companies most often reported that their ATP collaboration enabled the stimulation of creative thinking (94% indicated a "significant or moderate" effect), and many of them reported planning for future collaborations as enabled by their ATP collaboration (90% indicated a "significant or moderate" effect). The other effects cited most frequently as enabled by their ATP collaboration were a) obtaining R&D expertise not available within the company and b) acceleration of commercialization/entry into the marketplace.

In addition, more than half the companies that indicated a positive experience with collaboration said it helped them to ensure a reliable/quality source of supply, plan for manufacturing concurrently with performance of the R&D, identify customer needs, and save labor costs.

To some extent, "transaction costs" and delays have occurred in the course of negotiating joint venture intellectual property agreements and ATP Terms and Conditions for starting projects. But for the same 30 companies, the negative effects of collaboration appear to be relatively minor in terms of delays. (See Figure 12.) Nearly half, however, note that project coordinationand management costs were "significant or moderate."

Information Dissemination. Sharing of knowledge is important to achieving widespread diffusion of the ATP-funded technologies and maximum knowledge spillover benefits. The BRS contains questions concerning strategies for dissemination of non-proprietary information and for protection of intellectual property, and for progress in implementation. Activity reported to date indicates that many of the ATP awardees appear to be quite active in communicating the results of their work to the scientific community. The number of formal journal articles and papers/presentations at conferences is growing rapidly. (See Table 2.) Patent applications are beginning to be filed also.

Conclusions

Although most of the participants/projects reporting to date are still in the early R&D phases, adequate data exist to 1) illustrate some of the types of analyses possible, 2) provide a snapshot of the diverse applications and commercial opportunities that may be expected to result from the current ATP awards portfolio and an approximate time line, 3) give evidence that companies are taking necessary steps for successful future commercialization, 4) provide early indication that the information developed with ATP funding--both the published non-proprietary information and the proprietary information revealed through patent disclosure--is contributing to a shared knowledge base, and 5) indicate patent filings attributable to the research projects. For projects which have received at least one year of funding, participants are reporting significant acceleration of R&D, stimulation of beneficial collaborations, and a change in the nature of R&D performed as a result of ATP funding.

Future Work

In addition to collection of additional data over time from ATP's portfolio of projects, considerable work is needed in a number of areas:

• Improved data quality--To achieve maximum data quality, more effort is needed to work with the companies to ensure they understand the meaning and intent of the survey questions and provide reasonably thorough reports.

• Review for potential bias--We have worked with consultants to design questions and response choices that minimize the chance of biased responses; however, more needs to be done to reduce the chances of bias in the data. In general, the BRS results for FY93- and-later ATP projects are consistent with surveys conducted by third-party contractors covering projects and participants funded prior to FY93.

• Expanded coverage, analysis, and variety of reports--The existing BRS can support much more in-depth analysis (for example, by organization types, joint venture types, industry sectors, technology types, and geographical location) than can be covered here; furthermore, the results presented here reflect data for only a small fraction of existing survey questions. Future reports can address different issues and cross-sections of responses. Additionally, the BRS data can be linked to other SIC-based industry data sources.

• Additional question development--The major impacts of the ATP will likely occur some years after the period of funding covered by the ATP award. Additional question development is planned which will focus on the post-project commercialization phase and address inter-industry and intra-industry diffusion and benefits to future producers and users of these technologies, as well as longer-term economic impacts on the companies funded.

Date created: June 1996
Last updated: June 17, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)