Vague impact criteria are a blessing in disguise. Researchers who push against criteria that allow considerable autonomy are foolish and should learn from overseas contemporaries that a clearer definition of impact requirements is not dissimiliar from a tightening of the noose, write J. Britt Holbrook and Robert Frodeman.

Recent weeks have seen the continuation of a spirited debate about impact requirements. Ananyo Bhattacharya’s piece in The Guardian demonstrates a common reaction from scholars to impact criteria:

“The research councils continue to demand impact statements with their grant applications, a requirement that can only reward the most mundane research or those scientists most able to dissemble or exaggerate”.

Bhattacharya outlines the basic tenets of the anti-impact creed: that impact requirements mean mundane research, reward skillful exaggerators, and that requirements for impacts are akin to pacts with the devil. We believe that this shows a failure of imagination about what ‘impact’ could mean, and it also misconceives the nature of research more generally.

Opposition to impact criteria is wider than the UK

The Academy seems to be somewhat united in its opposition to impact requirements because these requirements come from outside the Academy, as noted in a previous post by Jon Adams. Adams diagnoses the problem as a “fundamental tension” between “the demands of disciplinary autonomy” on one hand and “the demands of centralised funding to generate metrics capable of arbitrating between achievements in widely disparate fields” on the other.

Peer review plays a central role in regulating this tension. We believe there is no greater guardian of academic autonomy than peer review. Yet in the last fifteen years requirements for societal impact have been introduced into the system of academic peer review. The ‘impact agenda’ in the UK is only one of many instantiations of this worldwide trend.

Since its implementation in 1997, the US National Science Foundation’s (NSF) Broader Impacts Criterion has been characterized by some scientists and engineers as a burden. Resistance to the criterion often took the form of one loud complaint: a lack of clarity.

Although few scientists would have been greatly upset if the broader impacts criterion had simply disappeared, Congress took a different view. In fact, a requirement for NSF to attend to broader impacts was written into legislation – into the America COMPETES Reauthorization Act of 2010. Moreover, in response to the criticism that the criterion was unclear, Congress provided a list of “national needs” that the criterion could be used to meet.

Making impact criteria more clear can suffocate autonomy

The National Science Board (NSB) proposed new criteria, released in June 2011. Instead of being asked to address how the research might benefit society, the proposed impact criterion contained the Congressional list of national needs, and proposers and reviewers were asked to answer which needs the proposal addressed.  Alan Leshner was quoted as saying that NSB had attempted to make the proposed criterion “frighteningly clear”. Although the NSB did receive some pushback, in taking away the vagueness of the “benefit to society” clause, the proposed criterion would limit the freedom of proposers and reviewers to suggest and judge novel and creative ideas not included on the list of “national needs.” Through ours and others’ criticisms, NSB concluded that a degree of vagueness is actually a good thing: it allows for maximum autonomy on the part of proposers and peer reviewers to provide their own answers to the demand for accountability.

Usefully vague criteria allows for academic creativity

On January 9 2012, NSB released its final report. NSB’s revisions more explicitly integrate broader impacts with intellectual merit, require a separate broader impacts section in grant proposals, and mandate a separate assessment of a project’s broader impacts in the grantee’s final report. Most importantly, the new criterion is now usefully vague: “The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.” What those desired societal outcomes might be is left to the proposers – that is, not Congress – to decide.

Researchers who resist impact criteria can learn a lesson from NSB. If the criteria you currently operate under are vague enough to allow for considerable autonomy in how you respond, then embrace them. Resistance to impact criteria leads to a tightening of the accountability noose. What’s important is not that there be no impact criteria that apply to you – a blue skies fantasy – but that the criteria allow you the freedom to use the same creativity that you use in your research.

The RCUK has the balance right

When we look at RCUK, we see an organization that understands this dialectic between autonomy and accountability. First, they recognize the demand for autonomy:

The Research Councils give their funding recipients considerable flexibility and autonomy in the delivery of their research, postgraduate training and knowledge transfer activities. This flexibility and autonomy encompasses project definition, management, collaboration, participation, promotion and the dissemination of research outputs; this approach enables excellence with impact.

Second, they expect those who apply for funding to recognize that autonomy must come with accountability. RCUK expect researchers to:

  • demonstrate an awareness of the wider environment and context in which their research takes place
  • demonstrate an awareness of the social and ethical implications of their research, beyond usual research conduct considerations, and take account of public attitudes towards those issues
  • engage actively with the public at both the local and national levels about their research and its broader implications

When we look at this list (which continues), we see a lot of room to maneuver. If proposers and reviewers will stop viewing impact as an obstacle and start viewing it is an opportunity, then there is less risk of ending up with more specific demands for impact – or worse, without a science funding body at all.



The authors gratefully acknowledge the support for this research provided by the US National Science Foundation (NSF) under grant No.0830387, while emphasizing that any opinions, conclusions, and recommendations expressed here are those of the authors and do not necessarily reflect the views of NSF, or any of their employees. 

Note:  This article gives the views of the author(s), and not the position of the Impact of Social Sciences blog, nor of the London School of Economics.

Print Friendly