Well done, NSB!
Legislation passed by Congress in 2010 confirmed the importance of broader impacts, and also tried to be more specific, listing some of the activities that would count as having societal benefit. But when the task force’s May 2011 draft report dutifully repeated some of these examples, some critics worried that the NSF’s criteria would end up being too specific. [NSB Task Force on Merit Review co-Chair] Bruer’s team has since removed the list.
Count CSID among those critics who urged NSB to rethink their attempt to make the Broader Impacts Criterion “frighteningly clear,” as Alan Leshner said last October.
When the draft criteria came out last June, Bob Frodeman and I published two responses, one here and the other here, arguing precisely that the proposed criteria were too restrictive. In the letter to Science, we wrote:
The proposed changes in the merit review criteria move too far in the direction of accountability, at the cost of scientific creativity and autonomy. The set of principles
(in terms of national goals) also suffers from excessive detail at the cost of flexibility.
In the Science Progress piece, we wrote:
Reviewers who reflect on their judgments regarding intellectual merit will recognize that they do not set themselves the task of predicting the future, but rather focus on whether the proposer has thought through their proposed research. Reviewers ask questions—is it feasible, is it an important topic, and so forth—about the planning and thought that have gone into the proposal, about whether the research has a chance to succeed. No reviewer is a fortune teller. The same is the case for judging broader impacts. The point is not to predict the future but to ensure the proposer has thought through what the broader impacts might be, who might benefit, and how to encourage that to happen.
In other words, the notion of broader impacts is no less inherently clear and no more inherently vague than the notion of intellectual merit.
It sounds as though NSB was listening:
“We don’t dictate what type of activities are intellectual merit,” says Bruer, president of the James McDonnell Foundation in St. Louis. “By the same token, we shouldn’t be prescriptive about what constitutes broader impacts. We’re not being overly prescriptive for either of them.”
This move restores NSF’s course regarding broader impacts, which we outline in detail here.
Take all of the preceding as preamble to what I think is an interesting theoretical question. How does one know, much less show, that one’s research has had an impact? In our case, for instance, even if NSB heeded our advice, will we ever know?
There is a real problem with tracking impacts. First, the term is ambiguous: in keeping track of impacts, ought we to look at broader societal impacts, or impact on the academic community (so to speak narrower intellectual impacts)? If only the latter, we can certainly attempt to track things like citations. But broader societal impacts tend to leave less obvious traces, even if they change the world. Moreover, other people certainly offered their own feedback to NSB, some of which was similar to our own.
Unless someone from NSB cites our work, and this is not common practice in the way that citing other work in academic publishing is, we will be hard pressed to demonstrate our own broader impacts.
Yes, NSB seems to have gone in the direction we recommended. Yes, we tried very hard to connect our NSF-funded research on peer review with policy makers and funding agency officials. But unless someone comes out and says, ‘yes, we listened, and your work helped us,’ there will always be room for doubt.