|
![](images/spacer.gif) |
|
![Taking Stock of the KDI - Science of Evaluation](images/eval_hdr.gif)
Research, like any other tax-supported activity, must be
accountable. This auditing function is not merely a matter of counting
publications or patents. Metrics have to be developed that match the real goals
of the research program. The KDI goals were to develop new ideas, innovative
tools, and a science and engineering workforce that could collaborate in
new, interdisciplinary areas. Each project also had its own specific goals,
such as modeling the earth in three dimensions to better predict oil reserves.
More generally, the National Science Foundation provides an interface between
basic, civilian non-medical sciences and the public. Hence, no single
metric would be sufficient to evaluate KDI, and any single metric would be
misleading.
NSF sponsored a systematic evaluation of the KDI, not just
to audit the program but also to serve as a diagnostic tool. What worked best,
and what should be managed differently? To begin, researchers from all 71 KDI
projects were invited to attend an NSF-sponsored workshop to discuss their
experiences. The variety of positive outcomes that workshop participants
described fell into four categories, as illustrated below.
New Ideas
New Tools
Student Training
Project Outreach
- Museum community improved access to software
- New collaborations with researchers at CONABIO Mexico
- Partnership with Sun Microsystems to obtain computer system
- Work with lawyers at the Courtroom 21 project
- Collaboration with IBM Watson Research Center
- Formation of alliance and development of transfer technology
- Meeting of researchers via seminar series
- Partnership from Pfizer as project spinoff
- Strengthening of relationships with government community
- Deployment of project software in the IT industry
- Supervision of talented high school students
- Formation of new community around project results
- Formation of close ties with IBM and Intel
As these lists suggest, the program as a whole met its
goals. What predicted success of the individual project?
Jonathon Cummings,
on the faculty at MIT and a former KDI postdoc, conducted a systematic survey of
grantees to find out. He found that dispersion rather than multi-disciplinarity
was the most problematic aspect of KDI projects. Projects with PIs in more
universities were significantly less well coordinated and reported fewer
positive outcomes. Coordination mechanisms that brought researchers together
(such as holding a project-related conference or workshop) appeared to reduce
the negative impact of dispersion.
One major difficulty still stands in the way of realistic
accountability. Much innovative research takes a long time to develop, and its
consequences may not be known for years. Evaluations need to be used with great
care to ensure that the scientists are busy doing long-range science and taking
risks, rather than working for short-term reporting goals.
Back to Top of Page |