KDI Home    Contact Us   

Research, like any other tax-supported activity, must be accountable. This auditing function is not merely a matter of counting publications or patents. Metrics have to be developed that match the real goals of the research program. The KDI goals were to develop new ideas, innovative tools, and a science and engineering workforce that could collaborate in new, interdisciplinary areas. Each project also had its own specific goals, such as modeling the earth in three dimensions to better predict oil reserves. More generally, the National Science Foundation provides an interface between basic, civilian non-medical sciences and the public. Hence, no single metric would be sufficient to evaluate KDI, and any single metric would be misleading.

NSF sponsored a systematic evaluation of the KDI, not just to audit the program but also to serve as a diagnostic tool. What worked best, and what should be managed differently? To begin, researchers from all 71 KDI projects were invited to attend an NSF-sponsored workshop to discuss their experiences. The variety of positive outcomes that workshop participants described fell into four categories, as illustrated below.

New Ideas

• Algorithm for large-scale predictive species distribution
• 3d optical and magnetometer signals for speech
• Combined game logic and algorithms
• Blood flow simulation around prosthetic heart valves
• Multi-electrode recording MEG and fMRI
• Structure-based vehicle detection/tracking
• Use of eye movements to study language production
• Digital language archives
• Brain machine interfaces
• Performance analysis of comparative gene finders
• Ecological informatics
• Digital technologies for archaeology
• Application of conservation laws to interface motion

New Tools

• System to support manual manipulation of virtual objects
• Web interface for analyzing and mapping species
• Speech data collection and analysis program
• Hardware technology for tracking facial expressions
• Web-based tutoring system for cognitive science
• Dynamic lead time production scheduling software
• Program to calculate fluid-structure interaction
• Software environment for polycrystal sample generation
• Code for analysis of multi-electrode data
• Language acquisition by autonomous robot program
• Software for relativistic astrophysics simulation
• Database of special functions of applied mathematics
• Open source resource for others doing surgical simulation

Student Training

• Grad finished thesis on learning in a flight training setting
• Undergrad thesis was published in top journal
• KDI research was incorporated in doctoral dissertation
• Student moved to aviation research unit of major company
• Two Master’s theses were completed
• Postdoc got assistant professor job
• Ph.D. thesis on topic was completed under the PI's supervision
• Ph.D. student defended thesis on modeling traffic dynamics
• Student got an academic job teaching linguistics at Northwestern
• Grad assistant received a prestigious postdoc at Yale
• Student got a job at Rand Corporation
• Staff got an academic appointment
• First ever Stanford BS honors thesis in MS&E

Project Outreach

• Museum community improved access to software
• New collaborations with researchers at CONABIO Mexico
• Partnership with Sun Microsystems to obtain computer system
• Work with lawyers at the Courtroom 21 project
• Collaboration with IBM Watson Research Center
• Formation of alliance and development of transfer technology
• Meeting of researchers via seminar series
• Partnership from Pfizer as project spinoff
• Strengthening of relationships with government community
• Deployment of project software in the IT industry
• Supervision of talented high school students
• Formation of new community around project results
• Formation of close ties with IBM and Intel

As these lists suggest, the program as a whole met its goals. What predicted success of the individual project? Jonathon Cummings, on the faculty at MIT and a former KDI postdoc, conducted a systematic survey of grantees to find out. He found that dispersion rather than multi-disciplinarity was the most problematic aspect of KDI projects. Projects with PIs in more universities were significantly less well coordinated and reported fewer positive outcomes. Coordination mechanisms that brought researchers together (such as holding a project-related conference or workshop) appeared to reduce the negative impact of dispersion.

One major difficulty still stands in the way of realistic accountability. Much innovative research takes a long time to develop, and its consequences may not be known for years. Evaluations need to be used with great care to ensure that the scientists are busy doing long-range science and taking risks, rather than working for short-term reporting goals.

Back to Top of Page