Comparative Assessment of Peer Review: Project Outcomes Report
Download this report by clicking here.
NSF Award #0830387
Principal Investigator: Robert Frodeman
Co-PIs: James Britt Holbrook, William Moen, Warren Burggren, Carl Mitcham
Public funding agencies are required to demonstrate accountability to their government funders (e.g.,
Congress) as well as to the public. Some agencies – including the US National Science Foundation (NSF)
– have used broader societal impacts criteria as part of the review process of grant proposals in order to
connect scientific research to societal needs. But these agencies have often encountered questions from
scientists and engineers for how to integrate such demands for broader societal impacts into their
research proposals. In an effort to help clarify the idea of broader impacts, in 2010 NSF and Congress
proposed a list of national needs that NSF-funded research would be required to meet. But was this the
best solution to the problem?
Our research asked several questions: Why do scientists express confusion at – and sometimes resist –
the requirement to address the societal impacts of their research? Was it because the broader impacts
criterion was unclear? Was it because they felt like they lacked the expertise to judge anything beyond
the science for its own sake? Or was it because they felt that such requirements interfered with
scientific autonomy?
Our research suggested that it was less that scientists and engineers felt unqualified to propose or judge
broader impacts and more that they simply did not want to address the broader impacts of their
research. This suggested that a focus on clarifying broader impacts activities may be misdirected.
Researchers may complain that broader impacts criteria are unclear not because the criteria are in fact
unclear, but rather because they do not want to address the criteria in the first place.
Dividing questions of ‘broader impacts’ off from questions of ‘intellectual merit’ also creates the
impression that such impacts on society are secondary to impacts on science. But broader impacts are
an essential facet of research. Benefits to society, in fact, are what drive science funding. Instead of
providing a checklist – national needs might change rapidly and unpredictably, after all – we ought to
require scientists and engineers to think about how their research responds to pressing societal needs.
We need, in other words, to balance the value of scientific autonomy with the value of societal
accountability.
Our findings were incorporated into a recent review and revision of NSF's Merit Review Criteria, which
identified the need to:
• Retain freedom for researchers to posit their own broader impacts rather than relying on a
predetermined list of national needs
• Treat broader impacts more like intellectual merit, and
• Develop better ways to measure the broader impacts of research
The CAPR website, which includes a digital repository containing over 850 documents related to peer
review is accessible here: http://csid-capr.unt.edu/.
Download this report by clicking here.