 |
A
Toolkit for Evaluating Public R&D Investment
Models, Methods, and Findings from ATP's First Decade
Rosalie Ruegg
Irwin Feller
July 2003 |
Contents
Download in Adobe
PDF front section, Part
I, Part II and Part
III.
For paper copies, e-mail your request to atp-eao@nist.gov.
Note
from Project Manager [View
PDF]
About
the Authors [View PDF]
Acknowledgments [View
PDF]
Abstract [View
PDF]
Executive
Summary [View PDF]
- Evaluation Underpinnings
- Multi-Faceted Methodological
Approach
- An Emerging Body of
Findings
- Firm/Industry Effects
- Collaboration Effects
- Spillover Effects
- Interfaces with State
Programs and Comparison with Counterpart Programs Abroad
- Overall ATP Performance
- Recommendations for
Future Directions
Acronyms
and Abbreviations [View PDF]
Introduction [View
PDF]
- The Political Economy
of the Advanced Technology Program
- The Role of Evaluation
PART
I: EVALUATION FRAMEWORK [View
PDF]
Chapter
1. Evaluation Fundamentals
- Objectives of Evaluation
- Mapping Evaluation
to Mission and Stages of Implementation: A Generic Evaluation Logic
Model
- Evaluation Steps
- Best Practices
Chapter
2. Choosing Methods of Evaluation
- Analytical/Conceptual
Methods for Modeling and Informing Underlying Program Theory
- Survey Method
- Case Study: Descriptive
- Case Study: Economic
Estimation
- Sociometric/Social
Network Analysis
- Bibliometrics: Counting,
Citing, and Analyzing Content of Documents
- Historical Tracing
- Expert Judgment
- Suggested Readings
on Evaluation Methods
Chapter
3. ATP’s Evaluation Program
- Background: Evaluation
Drivers
- ATP’s Evaluation
Logic Model
- Conceptual Tests of
ATP’s Success
- ATP’s Approach
to Evaluation
PART
II: APPLYING EVALUATION METHODS TO ATP [View
PDF]
Chapter
4. Modeling and Informing Underlying Program Theory
- Concepts, Models,
Metrics, and Paths Connecting Program Activities to Intended Impacts
- Conditioning Expectations
through Studies of Private-Sector Behavior
- Conditioning Expectations
through Studies of Other Public Sector Programs
- Testing Dominant Paradigms
- Summary of Research
Informing Underlying Program Theory
Chapter
5. Survey Method
- Gaining Early and
Broad Perspective of a Program’s Effects
- Extending and Deepening
Survey Data on Program Effects
- Establishing Routine
Project Reporting by Electronic Survey
- Soliciting Feedback
by Survey on Customer Satisfaction and Marketing Issues
- Using Survey for Case
Study and to Address Research Questions
- Summary of ATP’s
Use of the Survey Method
Chapter
6. Case Study Method
- Economic Case Study
of Individual Projects
- Project and Portfolio
Assessment Using Multiple Cases Studies with Uniform Collection of
Key Indicator Data
- Explicating Program
Features and Exploring Program Dynamics
- Summary of ATP’s
Use of the Case Study Method
Chapter
7. Econometric/Statistical Method
- Testing ATP’s
Leveraging Effects on Advanced Technology Development
- Modeling Impacts of
Public-Private Partnerships on Firm Productivity
- Spillovers, Appropriability,
and Firm Productivity
- Analyzing the Role
of Universities in Public-Private Partnerships
- Modeling the Impact
of Publishing by Industry Scientists on the Quality of Innovative
Output
- Investigating Characteristics
and Impacts of Joint Ventures
- Summary of Econometric/Statistical
Methods
Chapter
8. Other Evaluation Methods Used by ATP
- Expert Judgment
- Bibliometrics
- Emerging Methods:
Using Social Network Analysis to Identify Knowledge Spillovers
- Emerging Method: Using
the Cost Index Method to Estimate Social Benefits
- Emerging Method: Using
a Composite Performance Rating System Constructed from Case Study-Derived “Indicator” Data
- Summary of Other Evaluation
Methods
PART
III: AN EMERGING BODY OF KNOWLEDGE [View
PDF]
Chapter 9. A Crosscutting Look
at Study Findings
- Direct Private Firm
Effects
- Collaboration Effects
- Spillover Effects
- State and International
Technology Programs
- Measures of Overall
ATP Performance
- Summary of Crosscutting
Findings
Chapter
10. Conclusions and Recommendations
- Program Evaluation:
An Essential Tool for ATP
- Opportunities and
Future Directions
- Closing Note
References
About the Advanced Technology
Program (inside back cover)
Figures
Tables
- Table
1–1. Why Evaluate?
- Table
1–2. Steps in Evaluation
- Table
2–1. Overview of Evaluation Methods
- Table
2–2. Steps in Survey-Based Statistical Studies
- Table
3–1. ATP’s Mission, Mechanisms, and Features
- Table
3–2. Select ATP Studies Commissioned and Completed, 1991–1995
- Table
3–3. Select ATP Studies Commissioned and Completed, 1996–1999
- Table
3–4. Select ATP Studies Commissioned and Completed, or Delivered
in Draft, 2000
- Table
4–1. Twenty-Two Studies and Papers Modeling and Informing
Underlying Program Theory
- Table
4–2. Classification of Spillovers
- Table
4–3. Factors Increasing the Likelihood of Spillovers
- Table
4–4. Recommended Metrics for ATP and Timing of Data Collection
- Table
4–5. Interview Questions Investigating Determinants of Success
of Joint Ventures
- Table
4–6. Volume of Venture Capital Activity
- Table
4–7. Categorizing State Technology Assistance Program by
Type of Challenge and Stage of Technology Development and Commercialization
- Table
4–8. ATP Strategies Designed to Promote Technology Diffusion
- Table
5–1. Eight of Eleven Studies Using Survey Method Represented*
- Table
5–2. Program Effects Treated in the First Survey of ATP
Projects
- Table
5–3. Illustrative Survey Questions on Commercial Progress
- Table
5–4. Illustrative Survey Questions Relating to Knowledge
Spillovers
- Table
5–5. Illustrative Survey Questions about Collaborative Partners
- Table
5–6. Components of ATP’s Business Reporting System
- Table
5–7. Examples of Variables for Comparing Small and Larger
Firms Using BRS Data
- Table
5–8. Competitive Position of Member Companies in the World
PWB Market
- Table
5–9. Survey Questions Probing the Counterfactual: With and
Without ATP
- Table
6–1. Ten of Sixteen Studies Featuring Case Study Represented
- Table
6–2. Summary of Survey Findings on Partial Early-Stage Economic
Impacts
- Table
6–3. Overview of ATP Projects Included in this Study
- Table
6–4. Impact of ATP Funding on the Development of Medical
Technologies for Seven Tissue Engineering Projects
- Table
6–5. Comparison of QALY Utility-Weights for Different Health
States
- Table
6–6. Social Return on Investment and Social Return on Public
Investment: ATP Projects in Tissue Engineering for a Single Preliminary
Application
- Table
6–7. Composite Private Returns: ATP Projects in Tissue Engineering
for a Single Preliminary Application
- Table
6–8. Annual Impact on U.S. Macroeconomy of Near-Term, Five-Year
Implementation Path: Year 2004
- Table
6–9. Secondary Market Opportunities for CCAR Technology
- Table
6–10. Base Case Cash Flows from Improved Quality, Yield,
and Production Rates and from Reduced Refrigeration Costs from
Application of the CCAR Technology for Food Processing
- Table
6–11. Base Case Net Present Value, Internal Rate of Return,
and Benefit Cost Ratio (Calculated from the Cash Flows in
Table 6–10) for the CCAR Technology
- Table
6–12. Additional Qualitative Benefits from Using the CCAR
Technology
- Table
6–13. ATP’s Role
- Table
6–14. Status Report Data
- Table
7–1. Eight of Ten Studies Using Econometric/Statistical
Methods Represented*
- Table
8–1. Ten Studies Using Expert Judgment, Bibliometrics, and
Emerging Methods, including Cost Index, Social Network Analysis/Fuzzy
Logic, and Composite Performance Rating System
- Table
8–2. Data Sample Collected from Expert Physicians and Used
to Estimate the Bass Technology Diffusion Model
- Table
8–3. Dissemination of Non-Proprietary Information from ATP-Funded
Projects
- Table
8–4. Structured Interview Guide for Collecting Data Needed
to Exercise the Model
- Table
9–1. Studies Informing the Impacts of ATP on Private Firms
- Table
9–2. Illustrative Metrics Showing ATP’s Impact on Firms’ Innovation
Goals
- Table
9–3. The Extent to Which Non-Winners Pursue the Proposed
R&D Project Without ATP Funding
- Table
9–4. Estimates of Economic Value of a One-Year Reduction
in Applied Research Cycle Time, in Order of Decreasing Value:
$5 Million to $6 Million Median Value
- Table
9–5. ATP Effects that Helped Interviewees to Reduce Cycle
Time
- Table
9–6. Carryover of Cycle-Time Improvements to Other Projects
- Table
9–7. Effect of ATP Funding on Completed Projects
- Table
9–8. A Comparison of Small and Larger Firms in ATP
- Table
9–9. Progress of Participating Companies in Commercializing
New Technologies
- Table
9–10. Examples of Products and Processes from the First
50 Completed ATP Projects
- Table
9–11. Impact on Industries of Near-Term, Five-Year Implementation
Path for Flow-Control Machining Technology
- Table
9–12. Studies Extending Knowledge about Collaboration Activities
- Table
9–13. Summary of Study Findings on Frequency of Collaboration
- Table
9–14. Propensity to Collaborate with Other Organizations
and Form New Partnerships
- Table
9–15. Collaborative Activity of the First 50 Completed Projects
- Table
9–16. Membership Changes in the Printed Wiring Board Research
Joint Venture, 1992–1996
- Table
9–17. Specific Benefits of Collaborations
- Table
9–18. Studies Informing Spillover Effects
- Table
9–19. Characteristics of R&D Networks Generating Pre-Competitive,
Enabling Technologies
- Table
9–20. Spillovers Imputed by Comparing Composite Social Returns,
Public Returns, and Composite Private Returns on Seven Tissue
Engineering Projects
- Table
9–21. Studies Comparing State and International Programs
- Table
9–22. Comparative Features of ATP and Its Analogues
- Table
9–23. Studies Providing Broader Assessments of Program Performance
- Table
9–24. Measures of Economic Benefit from Six Detailed Case
Studies
Date created: February
9, 2004
Last updated:
August 2, 2005
|
