 |
|
|
|
|
|
|
|
|
 |
 |
 |
 |
 |
 |
 |
 |
|
|
|
||||||||||||||||||
|
The scores for knowledge creation and dissemination are constructed from the following metrics: a) awards by third-party organizations recognizing the excellence of ATP-funded technology, b) publications and presentations, c) patents filed—granted and not yet granted, d) collaborative activity of awardees as a proxy for transmission of information outside the walls of the project through organizational interactions, and e) commercialization or near commercialization by award recipients of the technology as a proxy for whether knowledge dissemination is occurring through observation and reverse engineering. The scores for commercialization progress are constructed from the following metrics: a) attraction of additional investment capital by innovators in the post-ATP period, b) commercialization or near commercialization by award recipients of the technology, c) employment gains, d) awards by third-party organizations in recognition of small-company business performance, and e) analysts’ assessment of the outlook for continued progress by ATP award recipients and their close collaborators in applying the new technologies commercially. The ratings for overall
performance are constructed by combining the knowledge creation and dissemination
scores and the commercialization progress scores. The combined score is
converted to a zero-to-four star rating. Four stars is the highest performance
score, and zero stars is the lowest. Two stars is a medium performance
rating—signifying neither particularly weak nor strong performance. The table on the next page summarizes the results of this performance rating system applied to the first 50 projects.
The rating system presented here should be viewed as a prototype. Future status report studies may improve and extend the system. Similarly, the performance rating scores presented here should be taken merely as indicative of project performance as of the date the individual project assessments were made, i.e., as of late 1998 for the group of 38 projects published originally in the previous status report volume(2) and repeated here (data set A in table), and as of late 1999 and early 2000 for the group of 12 new projects (data set B in table) added here to the 38 to comprise the first 50 completed projects. The conditions of these projects may have improved, worsened, or remained essentially the same since the data were collected. As was explained in the text, the scores do not measure ultimate national economic benefit, although low scores would cast doubt on the likelihood that the project will attain the large benefits originally envisioned, and high scores would convey continued strong expectations that the project is on track to deliver large benefits. Projects with the same scores are not necessarily equal in their benefits potential or in the value of their achievements to date. Similar scores do, however, suggest roughly comparable levels of project outputs and outlooks for the future at the time the projects were assessed. Both the absolute values and the relative positions assigned to the individual projects in the table could change in the future. DETAILS OF SCORING
SYSTEM The
Knowledge Creation part of the score is calculated as follows: Technical Awards — The number of technical awards for the 50 projects ranged from 0 to 4. Because the outside recognition of technical excellence is considered a good indicator that significant new knowledge has been created, the count of such awards forms one part of this score and each award received is counted fully. Thus, the weight in the raw scores for this element among the 50 projects also ranged from 0 to 4. Patent Filings — The presence of a patent filing is taken as an indicator that new knowledge has been created. The number of patent filings among the 50 projects ranged from 0 to 26. Patent filings are set to add to the raw score at a sharply declining rate, calculated as 0.5 times the square root of the number of additional patents. The weight in the raw score for this element among the 50 projects ranged from 0 to 2.5. Publications and Presentations — The existence of a publication or presentation is taken as another indicator that new knowledge has been created. The number of publications and presentations ranged from 0 to 214 among the 50 projects. Publications and presentations add to the raw score at a rate calculated as 0.5 times the fourth root of the number of publications and presentations. The aim is to give only a small additional credit to the raw score for numbers in excess of one. The resulting weight in the raw score for this element among the 50 projects ranged from 0 to 1.2. Product Now or Expected Soon — New product or process is taken as another indicator that new knowledge has been created. The number of products, now or expected, ranged from 0 to 5. Having product now or expected is set to add half as much to the raw score as does a technical award, and the same as does a patent filing or a publication or patent. But no additional credit is assigned for having more than one. This decision was made because often multiple products reflect the same underlying new technical knowledge. Total Knowledge Creation Raw Score — The raw scores for the above elements are summed. For the group of 50, total raw scores ranged from 0 to 4.5. The
Knowledge Dissemination part of the score is calculated as follows:
Technical Awards — Technical awards are also included in the Knowledge Dissemination score because they raise awareness of the new technology and thereby may stimulate others to seek knowledge about it. However, the award does not itself convey much detailed knowledge. Therefore, this element is given only a small weight in the raw score. Calculated as 0.25 times the square root of the number of technical awards, the first award counts as 0.25 in the raw score. For the group of 50 projects, the maximum number of technical awards is 4. Thus this element ranges from a value of 0 to 0.5 in the raw score. Collaborations — The following forms of collaboration were taken as indicative of knowledge flows from the innovators to others via contact among scientific and technical researchers and managers: a) R&D collaboration with nonuniversity organizations, b) collaboration for commercialization, and c) close university ties. Projects were assigned a score from 0 to 3 points depending on how many of these forms of collaboration they had. Note that these forms of collaboration observed for the project participants are used here as a proxy for collaboration with others outside the project leading to knowledge flows. Patents — By disclosing information, patents serve to disseminate project knowledge. Patents are set to add to the score at a declining rate. The raw-score value of the first 10 patents is calculated as 1 times the square root of the number of patents, and patents in excess of 10 contribute to the score at the rate of 0.1 times the square root of the number of patents greater than 10. The weight of this element in the raw score among the 50 projects ranged from 0 to 3.6. The decision is to give very little weight to additional patents in excess of 10 in the scoring system, and perhaps further analysis of patents as disseminators of knowledge will support a different decision. Publications and Presentations — Publications and presentations are treated the same as patents in calculating the score for knowledge dissemination. Their weight in the knowledge dissemination raw score for the 50 projects ranged from 0 to 4.6. In both cases, the influence on the overall scores of extremely large numbers in just several of the projects is greatly moderated by the calculation procedure that is used. Products and Processes Now or Expected — Products and processes are included in the knowledge dissemination measure because they embody the new know-how, and technical knowledge can be extracted through inspection and reverse engineering of products. They are assigned less weight than publications and presentations and patents, however, because they convey the knowledge less explicitly. This element is calculated as 0.5 times the square root of the number of products. Knowledge Dissemination
Raw Score — The components of the knowledge dissemination score
are summed to calculate the aggregate raw score. The aggregate raw score
for the 50 projects ranged from 0.7 to 9.4. Products and Processes Now or Expected — Having a product or process is assigned a greater weight than having any one of the other elements comprising this total score. Having a single project or process, now or expected, is assigned a raw score of 4.25. Additional products add at a rate of 1.25 times the square root of the number of products and processes. Their weight in the raw score for the 50 projects ranged from 0 to 5.8. Capital Attraction — Attracting capital in any of several ways, such as from private investors, from other government sources, or through collaborative commercialization agreements with other investing companies, was assigned a value of 3 points in computing the raw score—less than having a product or process. Because of the lack of information about the relative amounts of capital attracted from various sources for the different projects and the terms of use, no attempt was made here to assign different weights for attracting different sources or amounts of capital. Moreover, the weights were not additive for attracting multiple types of capital. The weight for this element in the raw score for the 50 projects was either 0 or 3. Further refinement of how this metric is included in the scoring system may be possible. Employment Gains — Employment data were recorded at the project start and near project finish by project analysts, but only for small, single-applicant projects. The data were not provided for most other single-applicant projects and none for joint-venture projects. A value of 1.5 is assigned for this element to those types of projects for which data were not collected to reduce the bias against them in the scores. This again is a place for further possible refinement to the scoring system. In the case where a small-company leader of a single-applicant project had gone bankrupt, a negative value of –6 was assigned as the raw-score weight—rather than a zero employment gain—to signal that there is a serious impediment at this time to commercial progress. If employment for a small-company leader of a single-applicant project increased 50 percent or less, it is assigned a weight of 0 in the raw score. If the employment for this type of project increased more than 50 percent, a weight of 2.5 times the fourth root of the gain in excess of 50 percent was assigned. The objective of this method of scoring is to give a relatively strong weight to those projects that exhibited large employment growth among company leaders, but not to have this one factor dominate the other elements in the total scores (which otherwise would have happened in the face of growth rates ranging nearly as high as 2000 percent for several of the projects). Using this weighting system, the weights for employment gains in the raw scores for the 50 projects ranged from –6 to 5.2. Business Awards
— Business-related awards were included in computing commercialization
progress scores because they generally signal unusually strong business
strength or acumen on the part of the project leaders. For business-related awards, a raw score of 0 is assigned for no awards and 3.25 for one award, less than commercialized product or process based on the new technology, but slightly more than attracting capital. Additional awards are set to increase the raw score at a rate of 0.25 per additional award (i.e., only partial credit is given because additional awards largely signal the same factor of relative company strength). The number of business awards among the 50 projects ranged from 0 to 3. The weight for this element in the raw scores for the 50 projects ranged from 0 to 3.8. Outlook — The qualitative outlooks for the individual projects described by analysts in chapters two through six were translated into values from one to four by the developer of the prototype performance rating system. Hence, the values for this element are strongly analytical. If the analyst described the outlook as highly promising, excellent, or on track, the project outlook was assigned a value of 1 to indicate a strong outlook. If the analyst described the outlook as promising but with reservations or qualifications, or as indeterminate, the project outlook was assigned a value of 2 to indicate an outlook neither strong nor necessarily poor. If the analyst portrayed a pessimistic outlook, or if the leading company had gone bankrupt or was experiencing severe financial difficulties, the project outlook was assigned a value of 3 to indicate a poor outlook. To convert the outlook rating system to values that would be compatible with the performance scoring system, an outlook value of 1 was assigned a raw score of +4; an outlook value of 2 was assigned a raw score of 0; and an outlook value of 3 was assigned a raw score of –4. Commercialization Progress Raw Score — The components are summed for the aggregate raw score. For the 50 projects, the aggregate raw score ranged from –10 to 21.7. Overall Performance Rating — A combined raw score for knowledge creation and dissemination and commercialization progress of zero or less, resulted in the assignment of a final score of zero. Projects with combined raw scores greater than zero were divided into four groups, corresponding to the four-star rating system. Scores equal or greater than four are assigned four stars; scores equal or greater than three but less than four are assigned three stars; scores equal or greater than two but less than three are assigned two stars; and scores equal or greater than one but less than two are assigned one star. Again, the reader is reminded that the performance ratings shown in the table are based on the projects observed at a time in the past, and their conditions may have since improved, worsened, or remained essentially the same. Future updates on subsequent project developments may result in changes in their performance ratings. In addition, the reader is cautioned that the first-generation performance rating system presented and applied here may be refined in the future, and the absolute values and relative positions assigned to the individual projects changed. ____________________
Return to Table of Contents Date created: April
2002 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
ATP website comments: webmaster-atp@nist.gov / Technical ATP inquiries: InfoCoord.ATP@nist.gov NIST is an agency of the U.S. Commerce Department |
 |
