Bridging from Project Case Study to Portfolio Analysis in a Public R&D Program A Framework for Evaluation and Introduction Rosalie Ruegg GCR 06-891 April 2006 View Adobe PDF version of report.

Contents

Abstract

Acknowledgements

Executive Summary

Introduction

Part I. A Framework Linking Evaluation of Individual Projects to Assessment of a Portfolio of Projects

1.0 Developing the Project-to-Portfolio Approach

• 1.1 Multi-Tiered Analytical Capability
• 1.2 Steps Involved in Developing the Framework
• 1.3 Resulting Evaluative Products

2.0 Pros and Cons of Rooting the Approach in Case Study

• 2.1 Case Study Advantages
• 2.2 Case Study Disadvantages
• 2.3 Case Study, A Serious Research Tool?

3.0 ATP's Development and Use of the Project-to-Portfolio Approach

• 3.1 Mini Case Studies Using Uniform Data Collection for All Completed Projects
• 3.2 Publication of ATP's First Status Report
• 3.3 New Features in ATP's Second Status Report

Part II. Development and Application of the Composite Performance Rating System (CPRS)

4.0 Prior Practice Using Composite Ratings

4.1 Composite Scoring by U.S. Education Department to Assess if Participants Meet Regulatory Requirements

4.2 Composite Scoring Proposed to Improve Healthcare Performance in OECD Countries

• 4.2.1 Composite Scoring of Hospital Performance
• 4.2.2 Composite Scoring for Investor Stock Ratings
• 4.2.3 Composite Scoring of the Entire S&T Innovation Process
• 4.2.4 Conclusions From Reviewing Prior Practice

5.0 CPRS Development: First Steps

• 5.1 CPRS General Formulation
• 5.2 Defining Mission-Driven Goals for CPRS Development
• 5.3 Data Constraints in Constructing ATP's CPRS

6.0 Specifying Indicator Variables for Use in the CPRS Formulation

• 6.1 Variables to Indicate Progress Toward Knowledge Creation
• 6.2 Variables to Indicate Progress Toward Knowledge Dissemination
• 6.3 Variables to Indicate Progress Toward Commercialization

7.0 Applying Weighting Algorithms to the Selected Indicator Variables and Calculating Scores

• 7.1 Knowledge Creation Scoring
• 7.2 Knowledge Dissemination Scoring
• 7.3 Commercialization Progress Scoring
• 7.4 Composite Scoring and Star Rating

8.0 Critique of CPRS

• 8.1 Data Limitations
• 8.2 Methodological Issues
• 8.3 An Inexact Measure

9.0 Summary and Conclusions

About the Economic Assessment Office ................ Inside front cover

About the Advanced Technology Program ............. Inside back cover

References

Figures

Figure 1. Multi-Tiered Analytical Capability
Figure 2. Distribution of Completed ATP Projects by CPRS

Tables

Table 1. Eight-Step Process in Developing Project-to-Portfolio Evaluation Framework
Table 2. Six Products From Applying the Eight-Step Process
Table 3. Advantages and Disadvantages of Case Study Methodology
Table 4. ATP Goals at Different Stages of the Project Life Cycle
Table 5. Calculation of Raw Scores for Knowledge Creation
Table 6. Sensitivity of Knowledge Creation to Changes in Indicator Values
Table 7. Calculation of Raw Scores for Knowledge Dissemination
Table 8. Sensitivity of Knowledge Dissemination to Changes in Indicator Values
Table 9. Calculation of Raw Scores for Commercialization
Table 10. Sensitivity of Commercialization Scores to Changes in Indicator Values
Table 11. Sensitivity Testing of Composite Scores and Star Ratings to Variations in Values of Indicator Variables
Table 12. Four-Star Project (Engineering Animation, Inc.)
Table 13. Zero-Star Project (Hampshire Instruments, Inc.)

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)