Peer review and the ex ante assessment of societal impacts
J. Britt Holbrook and Robert Frodeman
Funding agencies and research councils around the world rely on peer review to assess the
potential impacts of proposed research. This article compares the procedures of two major
public science agencies - the US National Science Foundation and the European Commission's 7th
Framework Programme - for evaluating ex ante the potential societal impact of research
proposals. In this paper we survey the state of the art and discuss some of the conceptual
questions that arise in using ex ante peer review to assess the societal impact of scientific
research.
J. Britt Holbrook is Assistant Director of the Center for the Study of Interdisciplinarity and Research Assistant
Professor in the Department of Philosophy at the University of North Texas, 1155 Union Circle #310920, Denton,
Texas 76203-5017, USA; Email: britt.holbrook@unt.edu.
Robert Frodeman is Director of the Center for the Study of Interdisciplinarity and Professor in the Department of
Philosophy at the University of North Texas, USA; Email: frodeman@unt.edu.
Introduction
Peer review is the multi-dimensional evaluation tool of the academy. It is used for everything from
determining the quality of articles and books, to whether individual academics are worthy of tenure or
promotion, to the viability of entire academic programmes. Moreover, peer review is also used in quasi-
academic contexts, including the review of grant proposals and the appraisal of science to be used in
regulatory contexts (Jasanoff, 1990).
The peer review of grant proposals requires that we make judgments about research yet to be
performed. The difficulty of knowing that one is judging accurately ahead of time prompted Ziman
(1983) to characterize peer review as a "higher form of nonsense." In the view of some, the ex ante use
of peer review by funding agencies and research councils around the world requires reviewers to
become "unwilling futurologists" (Rip, 2000).
Despite the difficulties surrounding predicting the future, the ex ante character of peer review is not the
central conceptual quandary plaguing grant proposal peer review today. Rather, citing the increasing
incorporation of 'relevance' as a review criterion at research councils, Rip (2000: 468) claimed that grant
proposal peer review is moving toward "still higher forms of nonsense". Asking scientists to predict the
future outside their own areas of expertise - that is, to judge the potential societal impact of scientific
research - is seen as even more difficult than asking them to predict the future within their own areas of
expertise. Or as Bozeman and Boardman (2009: 189) put it:
In the case of peer review related to BIC [the US National Science Foundation's Broader Impacts
Criterion], the problem is that scientists (and all of us) have no special claim to knowledge of
'the social good'. Why is the scientist who does research on the genetics of grasses any more
qualified to judge social good than the person who mows the grass?
This article explores the incorporation of 'relevance', or as we will say, broader societal impact, within
the decision-making process of public science funding agencies.
Holbrook, J. Britt & Frodeman, Robert. Peer Review and the Ex Ante Assessment of Societal Impacts. [Guildford, United Kingdom]. UNT Digital Library. http://digital.library.unt.edu/ark:/67531/metadc38888/. Accessed September 12, 2014.
