Sarewitz on NSF Broader Impacts

Dan Sarewitz has a column in Nature this week critical of NSF's well-meaning but ill-suited effort to modify its "broader impact criterion" -- the so-called "Criterion 2" --for the evaluation of specific project proposals.  Here is an excerpt (PDF):

Last month, the board published a revised criterion, and scientists had until this week to provide comments to the NSF before the final version is issued. But Criterion 2.1, as it might be called, is just as confusing and counterproductive as its predecessor.

At the heart of the new approach is “a broad set of important national goals”. Some address education, training and diversity; others highlight institutional factors (“partnerships between academia and industry”); yet others focus on the particular goals of “economic competitiveness” and “national security”. The new Criterion 2 would require that all proposals provide “a compelling description of how the project or the [principal investigator] will advance” one or more of the goals.

The nine goals seem at best arbitrary, and at worst an exercise in political triangulation. How else to explain the absence of such important aims as better energy technology, more effective environmental management, reinvigorated manufacturing, reduced vulnerability to natural and technological hazards, reversal of urban-infrastructure decay or improved performance of the research system? These are the sorts of goal that continue to justify public investments in fundamental research.

Yet, more troubling than the goals themselves is the problem of democratic legitimacy. In applying Criterion 2, peer-review panels will often need to choose between projects of equal intellectual merit that serve different national goals. Who gave such panels the authority to decide, for example, whether a claim to advance participation of minorities is more or less important than one to advance national security?

Sarewitz also makes this important point:

To convincingly assess how a particular research project might contribute to national goals could be more difficult than the proposed project itself.

Rather than asking PIs to do what is essentially impossible or at least far beyond their expertise, Sarewitz suggests that the locus of accountability needs to move higher up in the agency:

Motivating researchers to reflect on their role in society and their claim to public support is a worthy goal. But to do so in the brutal competition for grant money will yield not serious analysis, but hype, cynicism and hypocrisy. The NSF’s capacity to meet broad national goals is best pursued through strategic design and implementation of its programmes, and best assessed at the programme-performance level. Individual projects and scientists should be held accountable to specific programmatic goals, not vague national ones.

Ultimately, it is important to recognize that NSF is one of only three agencies with a legislated mandate to do "basic research" rather than agency-mission focused research (the two others being NASA and DOE's Office of Science). Asking NSF to become more mission oriented undercuts the overall purpose of the agency.

In my view (Sarewitz does not go this far) NSF should scrap the Criterion 2 at the project level and evaluate proposals on their scientific merit using peers with appropriate expertise. Setting the overall direction of NSF's research portfolios will inevitably be a political exercise, and that is a more appropriate location for efforts to better connect NSF with national needs.