Well done, NSB!
Legislation passed by Congress in 2010 confirmed the importance of broader impacts, and also tried to be more specific, listing some of the activities that would count as having societal benefit. But when the task force’s May 2011 draft report dutifully repeated some of these examples, some critics worried that the NSF’s criteria would end up being too specific. [NSB Task Force on Merit Review co-Chair] Bruer’s team has since removed the list.
Count CSID among those critics who urged NSB to rethink their attempt to make the Broader Impacts Criterion “frighteningly clear,” as Alan Leshner said last October.
When the draft criteria came out last June, Bob Frodeman and I published two responses, one here and the other here, arguing precisely that the proposed criteria were too restrictive. In the letter to Science, we wrote:
The proposed changes in the merit review criteria move too far in the direction of accountability, at the cost of scientific creativity and autonomy. The set of principles
(in terms of national goals) also suffers from excessive detail at the cost of flexibility.
In the Science Progress piece, we wrote:
Reviewers who reflect on their judgments regarding intellectual merit will recognize that they do not set themselves the task of predicting the future, but rather focus on whether the proposer has thought through their proposed research. Reviewers ask questions—is it feasible, is it an important topic, and so forth—about the planning and thought that have gone into the proposal, about whether the research has a chance to succeed. No reviewer is a fortune teller. The same is the case for judging broader impacts. The point is not to predict the future but to ensure the proposer has thought through what the broader impacts might be, who might benefit, and how to encourage that to happen.
In other words, the notion of broader impacts is no less inherently clear and no more inherently vague than the notion of intellectual merit.
It sounds as though NSB was listening:
“We don’t dictate what type of activities are intellectual merit,” says Bruer, president of the James McDonnell Foundation in St. Louis. “By the same token, we shouldn’t be prescriptive about what constitutes broader impacts. We’re not being overly prescriptive for either of them.”
This move restores NSF’s course regarding broader impacts, which we outline in detail here.
Take all of the preceding as preamble to what I think is an interesting theoretical question. How does one know, much less show, that one’s research has had an impact? In our case, for instance, even if NSB heeded our advice, will we ever know?
There is a real problem with tracking impacts. First, the term is ambiguous: in keeping track of impacts, ought we to look at broader societal impacts, or impact on the academic community (so to speak narrower intellectual impacts)? If only the latter, we can certainly attempt to track things like citations. But broader societal impacts tend to leave less obvious traces, even if they change the world. Moreover, other people certainly offered their own feedback to NSB, some of which was similar to our own.
Unless someone from NSB cites our work, and this is not common practice in the way that citing other work in academic publishing is, we will be hard pressed to demonstrate our own broader impacts.
Yes, NSB seems to have gone in the direction we recommended. Yes, we tried very hard to connect our NSF-funded research on peer review with policy makers and funding agency officials. But unless someone comes out and says, ‘yes, we listened, and your work helped us,’ there will always be room for doubt.
The bottom line, then, is that just as we have metrics, however imperfect, for tracking the impact of research on research, we also need ways to track the broader impacts of our research.
Thanks to @SPRU for pointing out that the Nature News blog entry had vanished.
Here is what it said:
NSF takes broad look at broader impacts
A National Science Foundation (NSF) task force has finalized its recommendations for tweaking the agency’s two merit review criteria, ‘intellectual merit’ and ‘broader impacts’. And central to that effort was a non-prescriptive, big-tent definition of broader impacts, says task force co-chair John Bruer (pictured), who presented the report on Tuesday to the National Science Board in Washington, DC.
“We don’t dictate what type of activities are intellectual merit,” says Bruer, president of the James McDonnell Foundation in St. Louis. “By the same token, we shouldn’t be prescriptive about what constitutes broader impacts. We’re not being overly prescriptive for either of them.”
Since 1997, the NSF has required all grant proposers to justify their requests not just on intellectual merit, but also on this notion of broader impacts. Yet researchers have found the requirements distressingly vague. Legislation passed by Congress in 2010 confirmed the importance of broader impacts, and also tried to be more specific, listing some of the activities that would count as having societal benefit. But when the task force’s May 2011 draft report dutifully repeated some of these examples, some critics worried that the NSF’s criteria would end up being too specific. Bruer’s team has since removed the list. “It raised problems about why some things were on the list and others not,” says Bruer.
As a result, the task force stripped down its recommendations and kept the wording for the two criteria essentially the same as before. The two criteria now come along with three principles, one of which is for researchers and institutions to gather data, whenever possible, assessing the effectiveness of not only the activities relating to intellectual merit, but also the broader impacts.
The National Science Board is likely to approve the final report on Wednesday. At that point, the NSF will begin implementing its recommendations. One thing that remains to be done is finding the right balance in shouldering the responsibility of broader impacts between principal investigator and institution, says NSF director Subra Suresh. “The responsibility cannot be bounced between the organization and the individual.”