Science Progress | Where science, technology, and progressive policy meet
SCIENCE PRIORITIES

Science: For Science’s or Society’s Sake?

Owning the National Science Foundation’s Broader Impacts Criterion

What return do taxpayers get for their investment in scientific and engineering research? This seems a natural enough question, unless you happen to be a scientist or engineer asked to give an account of the broader societal impacts of your work. Since 1997, however, those who apply to the National Science Foundation, or NSF, for grant funding have been asked to do just that. In addition to making a case for the “intellectual merit” of their research, research project proposers also had to discuss their research’s “broader impacts” on society. Researchers have resisted NSF’s Broader Impacts Merit Review criterion since its inception, arguing that it was irrelevant, impossible to answer, or, most commonly, just plain unclear.

This resistance stems in part from the definition of the sort of research NSF is supposed to fund—basic research. In 1945 Vannevar Bush, who was coming off a successful run as chairman of the Manhattan Project, spent his political capital arguing that the federal government ought to support basic research. Bush’s report, “Science – the Endless Frontier,” defined basic research as research done without regard for its practical consequences. The term was essentially a more practical way of saying pure science—a term that had been in circulation at least since the late 19th century. The point was to conduct research that was driven by the researcher’s own curiosity rather than by any sort of external need. This is why the term of art is often “blue skies” research, named after the question “Why is the sky blue?”[1]

Read the full series on the NSF’s changing criteria for federal science grant proposal evaluation:

This struggle between scientists looking to rebel against government efforts to restrict their freedom reminds us a little of the movie classic “Cool Hand Luke” in which prison warden Strother Martin engages in a running conflict with Paul Newman’s character, the inmate Luke. Luke has little interest in following prison rules, and Martin, in frustration, eventually lays down the law. Luke is told that he needs to “get his mind right” about how things are going to operate. Luke sees things differently.

Those of us who watch old movies will remember that Luke comes to a melancholy end. Of course, the relationship between Congress on the one hand and the NSF and individual scientists on the other is not the same as that of the warden and inmates. But one point does seem apt. Following a congressional mandate the updated merit review criteria issued by the National Science Board, or NSB, in December 2011 make it clear that the Broader Impacts criterion is not going away. Congress has, quite literally, laid down the law. Rather than continue to fight, the scientific and engineering community should look for ways to own the Broader Impacts criterion and make it work to their overall advantage.

The struggle to communicate how science benefits society

Since its implementation in 1997, NSF’s Broader Impacts criterion has often been viewed by scientists and engineers as an onerous burden. The view was that the real work of science was captured by outlining a project’s intellectual merit—that is, outlining its impact on science for an audience of other scientists. Many research project proposers and reviewers considered NSF’s requirement that scientists discuss the broader impacts of a project on society to be irrelevant.

Congress, however, viewed things differently. In fact, a requirement for NSF to attend to broader impacts was written into the America COMPETES Reauthorization Act of 2010. Moreover, in response to the repeated criticism that the Broader Impacts criterion was unclear, Congress provided a list of “national needs” that the criterion could be used to meet. Rather than going away, as some scientists had hoped it would, a new, more prescriptive Broader Impacts criterion emerged.

NSF then decided to reassess the entirety of its merit, or peer, review criteria. Proposed new criteria were released in June 2011, with a period for feedback.[2] NSB did receive some pushback—according to an entry on the Nature News Blog from December 13, 2011, the task force eventually settled on a “non-prescriptive, big-tent definition” of broader impacts:

Since 1997, the NSF has required all grant proposers to justify their requests not just on intellectual merit, but also on this notion of broader impacts. Yet researchers have found the requirements distressingly vague. Legislation passed by Congress in 2010 confirmed the importance of broader impacts, and also tried to be more specific, listing some of the activities that would count as having societal benefit. But when the task force’s May 2011 draft report dutifully repeated some of these examples, some critics worried that the NSF’s criteria would end up being too specific. [Task Force co-chair] Bruer’s team has since removed the list. “It raised problems about why some things were on the list and others not,” says Bruer.[3]

In NSB’s initial June 2011 revision, potential benefits to society appeared to have been limited to the list of national goals provided by Congress. But some feared that in taking away the vagueness of the “benefit to society” clause, the proposed criterion would limit the freedom of proposers and reviewers to suggest and judge novel and creative ideas not included on the list.[4] Although critics of the Broader Impacts criterion had consistently claimed that it was unclear or “distressingly vague,” NSB came to realize that a degree of vagueness is actually a good thing—it allows for maximum autonomy on the part of proposers and peer reviewers to provide their own answers to the demand for accountability.

A deeper emphasis on “broader impacts”

On January 9, 2012, NSB released its final report, “National Science Foundation’s Merit Review Criteria: Review and Revisions.” In the final revisions NSF’s Broader Impacts criterion does not merely survive—its status in the review process has grown. NSB’s revisions more explicitly integrate broader impacts with intellectual merit. They also require a separate broader impacts section in grant proposals and mandate a separate account and assessment of a project’s broader impacts in the grantee’s final report. In other words, the Broader Impacts criterion is approaching parity with the Intellectual Merit criterion.

Welcome to Broader Impacts 2.0, per the NSF:

Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.[5]

Note that not only is the new Broader Impacts criterion usefully vague, but it also explicitly requires that proposers (that is, not Congress) articulate a project’s potential benefits to society.[6]

The final revisions also ask proposers and reviewers to address the same set of questions for intellectual merit and broader impacts—in areas such as the soundness of the plan for the proposed activity, the qualifications of the proposer(s) to carry out the proposed activities, and the access to necessary resources. Indeed, even the question of the potential transformativity of the proposed activities now applies to both intellectual merit and broader impact.

NSB’s integration of intellectual merit and broader impact means seeing the connections between things formerly thought to be separable. NSB’s new criteria recognize that in the 21st century, our disciplinary peers are no longer our only audience. Moreover, science funding is increasingly tied to the notion that basic research is a driver of innovation, which in turn drives economic growth. This will require an adjustment in the way we think about broader impacts: Scientists and engineers will need to begin to see that even basic research must take place in the context of the needs of the users of that knowledge.

It is obviously too early to say whether scientists and engineers will actually make the adjustment. By removing the list of national needs in the June 2011 proposed revisions and enhancing the usefully vague “benefits to society” language of the final version of the Broader Impacts criterion, however, NSF is allowing members of the scientific and engineering community the freedom and creativity to give an account of their own broader impacts.

Whether scientists and engineers will embrace this freedom or continue to resist such calls for accountability remains to be seen. If we are to learn anything from the past, however, unless scientists get their minds right about broader impacts, Congress will be happy to step in to resolve this failure to communicate.

Robert Frodeman is professor of philosophy and director of the Center for the Study of Interdisciplinarity (CSID) at the University of North Texas. J. Britt Holbrook is research assistant professor of philosophy and assistant director at the Center for the Study of Interdisciplinarity at the University of North Texas. One of the central research projects at CSID is understanding the meaning of “broader impacts.”  The authors gratefully acknowledge the support for this research provided by the US National Science Foundation (NSF) under grant No.0830387, while emphasizing that any opinions, conclusions, and recommendations expressed here are those of the authors and do not necessarily reflect the views of NSF, or any of their employees.

References


[1] Researchers in the United Kingdom are also resisting attempts at what they perceive as the government imposing an “impact agenda” on blue skies research. This recent exchange between Bhattacharya and Khan and Curry in The Guardian gives the basic flavor the debate.

[2] Several relevant publications also followed the release of the proposed new criteria, including AAAS, 2011; Frodeman and Holbrook 2011; Holbrook and Frodeman, 2011; Mervis, 2011; and Sarewitz, 2011.

[5] In full, the final December 14, 2011, revision of the merit review criteria reads as follows:

Merit Review Criteria

When evaluating NSF proposals, reviewers should consider what the proposers want to do, why they want to do it, how they plan to do it, how they will know if they succeed, and what benefits would accrue if the project is successful. These issues apply both to the technical aspects of the proposal and the way in which the project may make broader contributions. To that end, reviewers are asked to evaluate all proposals against two criteria:

Intellectual Merit: The intellectual Merit criterion encompasses the potential to advance knowledge; and

Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.

The following elements should be considered in the review for both criteria:

1. What is the potential for the proposed activity to

a. advance knowledge and understanding within its own field or across

different fields (Intellectual Merit); and

b. benefit society or advance desired societal outcomes (Broader Impacts)?

2. To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?

3. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success?

4. How well qualified is the individual, team, or institution to conduct the proposed activities?

5. Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities?

Compare this final revision to the June 2011 proposed revision, which, after listing nine national goals, read as follows:

Broader impacts of the proposed activity

The purpose of this review criterion is to ensure the consideration of how the proposed project advances a national goal(s). Elements to consider in the review are:

  1. Which national goal (or goals) is (or are) addressed in this proposal? Has the PI presented a compelling description of how the project or the PI will advance that goal(s)?
  2. Is there a well-reasoned plan for the proposed activities, including, if appropriate, department-level or institutional engagement?
  3. Is the rationale for choosing the approach well-justified? Have any innovations been incorporated?
  4. How well qualified is the individual, team, or institution to carry out the proposed broader impacts activities?
  5. Are there adequate resources available to the PI or institution to carry out the proposed activities?

There was a separate June 2011 description of the Intellectual Merit Criterion.

[6] This complies with the America COMPETES Reauthorization Act of 2010, which only mandates that NSF shall employ “a Broader Impacts review criterion.” The law does not require that NSF adopt the list of national needs in the Act, as Holbrook (2010) discusses.

Comments on this article

By clicking and submitting a comment I acknowledge the Science Progress Privacy Policy and agree to the Science Progress Terms of Use. I understand that my comments are also being governed by Facebook's Terms of Use and Privacy Policy.