|
A
Toolkit for Evaluating Public R&D Investment
Models, Methods, and Findings from ATP's First Decade
Rosalie Ruegg
Irwin Feller
July 2003 |
Contents
Download in Adobe
PDF front section, Part
I, Part II and Part
III.
For paper copies, e-mail your request to atp-eao@nist.gov.
Note
from Project Manager [View
PDF]
About
the Authors [View PDF]
Acknowledgments [View
PDF]
Abstract [View
PDF]
Executive
Summary [View PDF]
- Evaluation Underpinnings
- Multi-Faceted Methodological
Approach
- An Emerging Body of
Findings
- Firm/Industry Effects
- Collaboration Effects
- Spillover Effects
- Interfaces with State
Programs and Comparison with Counterpart Programs Abroad
- Overall ATP Performance
- Recommendations for
Future Directions
Acronyms
and Abbreviations [View PDF]
Introduction [View
PDF]
- The Political Economy
of the Advanced Technology Program
- The Role of Evaluation
PART
I: EVALUATION FRAMEWORK [View
PDF]
Chapter
1. Evaluation Fundamentals
- Objectives of Evaluation
- Mapping Evaluation
to Mission and Stages of Implementation: A Generic Evaluation Logic
Model
- Evaluation Steps
- Best Practices
Chapter
2. Choosing Methods of Evaluation
- Analytical/Conceptual
Methods for Modeling and Informing Underlying Program Theory
- Survey Method
- Case Study: Descriptive
- Case Study: Economic
Estimation
- Sociometric/Social
Network Analysis
- Bibliometrics: Counting,
Citing, and Analyzing Content of Documents
- Historical Tracing
- Expert Judgment
- Suggested Readings
on Evaluation Methods
Chapter
3. ATPs Evaluation Program
- Background: Evaluation
Drivers
- ATPs Evaluation
Logic Model
- Conceptual Tests of
ATPs Success
- ATPs Approach
to Evaluation
PART
II: APPLYING EVALUATION METHODS TO ATP [View
PDF]
Chapter
4. Modeling and Informing Underlying Program Theory
- Concepts, Models,
Metrics, and Paths Connecting Program Activities to Intended Impacts
- Conditioning Expectations
through Studies of Private-Sector Behavior
- Conditioning Expectations
through Studies of Other Public Sector Programs
- Testing Dominant Paradigms
- Summary of Research
Informing Underlying Program Theory
Chapter
5. Survey Method
- Gaining Early and
Broad Perspective of a Programs Effects
- Extending and Deepening
Survey Data on Program Effects
- Establishing Routine
Project Reporting by Electronic Survey
- Soliciting Feedback
by Survey on Customer Satisfaction and Marketing Issues
- Using Survey for Case
Study and to Address Research Questions
- Summary of ATPs
Use of the Survey Method
Chapter
6. Case Study Method
- Economic Case Study
of Individual Projects
- Project and Portfolio
Assessment Using Multiple Cases Studies with Uniform Collection of
Key Indicator Data
- Explicating Program
Features and Exploring Program Dynamics
- Summary of ATPs
Use of the Case Study Method
Chapter
7. Econometric/Statistical Method
- Testing ATPs
Leveraging Effects on Advanced Technology Development
- Modeling Impacts of
Public-Private Partnerships on Firm Productivity
- Spillovers, Appropriability,
and Firm Productivity
- Analyzing the Role
of Universities in Public-Private Partnerships
- Modeling the Impact
of Publishing by Industry Scientists on the Quality of Innovative
Output
- Investigating Characteristics
and Impacts of Joint Ventures
- Summary of Econometric/Statistical
Methods
Chapter
8. Other Evaluation Methods Used by ATP
- Expert Judgment
- Bibliometrics
- Emerging Methods:
Using Social Network Analysis to Identify Knowledge Spillovers
- Emerging Method: Using
the Cost Index Method to Estimate Social Benefits
- Emerging Method: Using
a Composite Performance Rating System Constructed from Case Study-Derived Indicator Data
- Summary of Other Evaluation
Methods
PART
III: AN EMERGING BODY OF KNOWLEDGE [View
PDF]
Chapter 9. A Crosscutting Look
at Study Findings
- Direct Private Firm
Effects
- Collaboration Effects
- Spillover Effects
- State and International
Technology Programs
- Measures of Overall
ATP Performance
- Summary of Crosscutting
Findings
Chapter
10. Conclusions and Recommendations
- Program Evaluation:
An Essential Tool for ATP
- Opportunities and
Future Directions
- Closing Note
References
About the Advanced Technology
Program (inside back cover)
Figures
Tables
- Table
11. Why Evaluate?
- Table
12. Steps in Evaluation
- Table
21. Overview of Evaluation Methods
- Table
22. Steps in Survey-Based Statistical Studies
- Table
31. ATPs Mission, Mechanisms, and Features
- Table
32. Select ATP Studies Commissioned and Completed, 19911995
- Table
33. Select ATP Studies Commissioned and Completed, 19961999
- Table
34. Select ATP Studies Commissioned and Completed, or Delivered
in Draft, 2000
- Table
41. Twenty-Two Studies and Papers Modeling and Informing
Underlying Program Theory
- Table
42. Classification of Spillovers
- Table
43. Factors Increasing the Likelihood of Spillovers
- Table
44. Recommended Metrics for ATP and Timing of Data Collection
- Table
45. Interview Questions Investigating Determinants of Success
of Joint Ventures
- Table
46. Volume of Venture Capital Activity
- Table
47. Categorizing State Technology Assistance Program by
Type of Challenge and Stage of Technology Development and Commercialization
- Table
48. ATP Strategies Designed to Promote Technology Diffusion
- Table
51. Eight of Eleven Studies Using Survey Method Represented*
- Table
52. Program Effects Treated in the First Survey of ATP
Projects
- Table
53. Illustrative Survey Questions on Commercial Progress
- Table
54. Illustrative Survey Questions Relating to Knowledge
Spillovers
- Table
55. Illustrative Survey Questions about Collaborative Partners
- Table
56. Components of ATPs Business Reporting System
- Table
57. Examples of Variables for Comparing Small and Larger
Firms Using BRS Data
- Table
58. Competitive Position of Member Companies in the World
PWB Market
- Table
59. Survey Questions Probing the Counterfactual: With and
Without ATP
- Table
61. Ten of Sixteen Studies Featuring Case Study Represented
- Table
62. Summary of Survey Findings on Partial Early-Stage Economic
Impacts
- Table
63. Overview of ATP Projects Included in this Study
- Table
64. Impact of ATP Funding on the Development of Medical
Technologies for Seven Tissue Engineering Projects
- Table
65. Comparison of QALY Utility-Weights for Different Health
States
- Table
66. Social Return on Investment and Social Return on Public
Investment: ATP Projects in Tissue Engineering for a Single Preliminary
Application
- Table
67. Composite Private Returns: ATP Projects in Tissue Engineering
for a Single Preliminary Application
- Table
68. Annual Impact on U.S. Macroeconomy of Near-Term, Five-Year
Implementation Path: Year 2004
- Table
69. Secondary Market Opportunities for CCAR Technology
- Table
610. Base Case Cash Flows from Improved Quality, Yield,
and Production Rates and from Reduced Refrigeration Costs from
Application of the CCAR Technology for Food Processing
- Table
611. Base Case Net Present Value, Internal Rate of Return,
and Benefit Cost Ratio (Calculated from the Cash Flows in
Table 610) for the CCAR Technology
- Table
612. Additional Qualitative Benefits from Using the CCAR
Technology
- Table
613. ATPs Role
- Table
614. Status Report Data
- Table
71. Eight of Ten Studies Using Econometric/Statistical
Methods Represented*
- Table
81. Ten Studies Using Expert Judgment, Bibliometrics, and
Emerging Methods, including Cost Index, Social Network Analysis/Fuzzy
Logic, and Composite Performance Rating System
- Table
82. Data Sample Collected from Expert Physicians and Used
to Estimate the Bass Technology Diffusion Model
- Table
83. Dissemination of Non-Proprietary Information from ATP-Funded
Projects
- Table
84. Structured Interview Guide for Collecting Data Needed
to Exercise the Model
- Table
91. Studies Informing the Impacts of ATP on Private Firms
- Table
92. Illustrative Metrics Showing ATPs Impact on Firms Innovation
Goals
- Table
93. The Extent to Which Non-Winners Pursue the Proposed
R&D Project Without ATP Funding
- Table
94. Estimates of Economic Value of a One-Year Reduction
in Applied Research Cycle Time, in Order of Decreasing Value:
$5 Million to $6 Million Median Value
- Table
95. ATP Effects that Helped Interviewees to Reduce Cycle
Time
- Table
96. Carryover of Cycle-Time Improvements to Other Projects
- Table
97. Effect of ATP Funding on Completed Projects
- Table
98. A Comparison of Small and Larger Firms in ATP
- Table
99. Progress of Participating Companies in Commercializing
New Technologies
- Table
910. Examples of Products and Processes from the First
50 Completed ATP Projects
- Table
911. Impact on Industries of Near-Term, Five-Year Implementation
Path for Flow-Control Machining Technology
- Table
912. Studies Extending Knowledge about Collaboration Activities
- Table
913. Summary of Study Findings on Frequency of Collaboration
- Table
914. Propensity to Collaborate with Other Organizations
and Form New Partnerships
- Table
915. Collaborative Activity of the First 50 Completed Projects
- Table
916. Membership Changes in the Printed Wiring Board Research
Joint Venture, 19921996
- Table
917. Specific Benefits of Collaborations
- Table
918. Studies Informing Spillover Effects
- Table
919. Characteristics of R&D Networks Generating Pre-Competitive,
Enabling Technologies
- Table
920. Spillovers Imputed by Comparing Composite Social Returns,
Public Returns, and Composite Private Returns on Seven Tissue
Engineering Projects
- Table
921. Studies Comparing State and International Programs
- Table
922. Comparative Features of ATP and Its Analogues
- Table
923. Studies Providing Broader Assessments of Program Performance
- Table
924. Measures of Economic Benefit from Six Detailed Case
Studies
Date created: February
9, 2004
Last updated:
August 2, 2005
|