Since 1997 NSF has required that all grant proposals be evaluated on two dimensions: Intellectual Merit and Broader Impacts.
Intellectual Merit has always been clear to grant writers and grant reviewers (though they often disagree about the merits of any particular proposal). “Broader Impacts” has always been rather murky with different grant writers and grant reviewers interpreting it in incompatible ways. It has also been very difficult to evaluate whether “Broader Impacts” have been achieved by a funded project.
To try to clarify things, NSF just approved a report titled National Science Foundation’s Merit Review Criteria: Review and Revisions December 14, 2011, subject to final edits. Unfortunately, they have not put the report on their own web site yet, so I had to get a copy from scienceinsider. After reading the report (the body, not the hundreds of pages of appendices), I’m at least as confused as I was before about what the h*** NSF expects for Broader Impacts.
It seems that the guidance they offer is pretty much limited to one page:
Merit Review Criteria
When evaluating NSF proposals, reviewers should consider what the proposers want to do, why they want to do it, how they plan to do it, how they will know if they succeed, and what benefits would accrue if the project is successful. These issues apply both to the technical aspects of the proposal and the way in which the project may make broader contributions. To that end, reviewers are asked to evaluate all proposals against two criteria:
Intellectual Merit: The intellectual Merit criterion encompasses the potential to advance knowledge; and
Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.The following elements should be considered in the review for both criteria:
- What is the potential for the proposed activity to
a. advance knowledge and understanding within its own field or across different fields (Intellectual Merit); and
b. benefit society or advance desired societal outcomes (Broader Impacts)?- To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?
- Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success?
- How well qualified is the individual, team, or institution to conduct the proposed activities?
- Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities?
They deliberately avoided further clarity:
Because of the great breadth and diversity of research and education activities that are supported by NSF, the Board has decided not to recommend a specific set of activities related to Broader Impacts, just as it would not recommend particular types of research–those decisions are best left to the PIs to describe and to the NSF to evaluate, for relevance to programmatic priorities and alignment with NSF’s core strategies for achieving its mission, as described in the NSF Strategic Plan for FY 2011- 2016 “Empowering the Nation Through Discovery and Innovation:”
- Be a leader in envisioning the future of science and engineering.
- Integrate research and education and build capacity.
- Broaden participation in the science and engineering research and education enterprises.
- Learn through assessment and evaluation of NSF programs, processes, and outcomes.
So grant writers and reviewers are left with nothing more specific than “benefit society or advance desired societal outcomes”. Different people will have very different ideas about what societal outcomes are “desired”, and Broader Impacts will continue to be used as a blunt instrument to reject intellectually worthy proposals based on the whims of reviewers. If there were a list of “desired societal outcomes” then it might be possible to compare proposals, but without such a list, politics, religion, and random taste prevail.
More likely, there is a secret list of what NSF really wants for broader impacts, which is supposed to be “understood” without ever being stated. So far as I can tell, NSF has just said “read our minds, we’re going to judge whether you address our goals without ever telling you what those goals are”. When this is combined with NSF’s new policy of funding bunches of stuff without external review, it becomes absolutely critical that there be clarity on what the goals are.
I hate guessing games like this.
Related articles
- NSF Tweaks Its Merit Review Rules (news.sciencemag.org)
- NSF takes broad look at broader impacts (blogs.nature.com)
- NSF Broader Impacts Broadened (writedit.wordpress.com)
- New NSF program without peer review (gasstationwithoutpumps.wordpress.com)
- New NSF Program Sidesteps External Peer Review (news.sciencemag.org)
Hi, there. I posted a response to your blog here: http://cas-csid.cas.unt.edu/?p=2842.
Let me know what you think!
Britt
Comment by jbrittholbrook — 2012 January 12 @ 07:46 |
I think that you have exaggerated my position in your response, but that is ok, since I exaggerated NSF position somewhat in my rant. Some of the straw-man arguments you put up are pretty far from my opinions though, so I wish you had not put it in the form of a dialog with my name associated with the straw man.
I think that there is a difference in the Intellectual Merit and Broader Impacts evaluations as done at NSF. The panelists and reviewers are chosen as experts in the science for the proposal that they are evaluating. They generally understand and can evaluate the science pretty well.
Neither the reviewers nor the grant writers are experts in “Broader Impacts”, and so the judgements made there are generally pretty random, based on prejudices and arbitrary political opinions. If NSF wants Broader Impacts to be seriously addressed, they have to come up with a less random way of evaluating them in the proposals. Vague statements about “societal goals” are not going to help, because there are very few goals that panelists with different political beliefs agree on, and those goals are usually not addressable in specific projects.
If you want to bring up “scientific autonomy”, I’m afraid that the Broader Impacts rules are not a good place to make your stand—they were imposed by politicians and scientists are still trying to figure out what the h*** the politicians were asking for. Most scientists would prefer to go back to a system where scientific research funding was judged on scientific merit alone, with small bonuses for obviously useful stuff.
Comment by gasstationwithoutpumps — 2012 January 12 @ 08:10 |
I agree that I exaggerated your position somewhat — I’ll add a note to indicate that the ‘scientist’ role in the dialogue is not meant to represent you or any real scientist, but an extreme version of someone who isn’t favorably disposed to Broader Impacts as part of NSF Merit Review. Thanks for not becoming too upset!
As for the point you make about the lack of Broader Impacts expertise, I think there is some merit to what you say. But I disagree that expertise in the science relevant to the proposal renders reviewers immune to prejudice and arbitrary opinions. I also think vague Broader Impacts criteria do help. They allow proposers the sort of freedom they have with Intellectual Merit.
I realize that most scientists would prefer to go back to a system where scientific research funding was judged on scientific merit alone (though the Rothenberg article attached to the report as the last appendix casts doubt on whether this was ever the case at NSF). I simply think that such an attitude is self-defeating. Yes, Broader Impacts were in some sense “imposed by politicians” — but isn’t that a fair exchange for public funding of science? That it should have some societal benefit? If scientists continue to wait for the politicians to tell them what Broader Impacts they’d like to see, the politicians will likely be willing to oblige. THAT would interfere with scientific autonomy to a much greater degree than a vague criterion that the science paid for by society ought to benefit society in some way.
This is an opportunity for scientists to take control of Broader Impacts so that Broader Impacts don’t control them.
Comment by jbrittholbrook — 2012 January 12 @ 08:29 |
There are a lot of NSF programs for which Broader Impacts become very important – REU for example, and pretty much all of the educational programs like CE21 and TUES.
Comment by Bonnie — 2012 January 13 @ 05:30 |
Of course, for the educational programs (fellowships, REU, and so forth), the Broader Impacts are the whole point of the program, and the goals are pretty clear (generally increasing the number and diversity of the STEM PhDs, with the main diasgreements being over which aspects of diversity are most important—race, gender, geography, field of study, age, …).
I have no problem with Broader Impacts being important and the correct thing to consider for programs that are intended to address them. For that matter I think that NSF should be putting more of their funding into the fellowships, rather than paying grad students indirectly through research grants.
NSF as a whole should definitely be concerned both with original research and with meeting societal goals. That doesn’t mean that every grant they give should address everything. Sometimes it is more effective to specialize, and to have some grants addressing pure research questions and others addressing societal goals, perhaps with little research component, just implementing things already known to work.
Comment by gasstationwithoutpumps — 2012 January 13 @ 09:12 |
[...] NSF “clarifies” Broader Impacts [...]
Pingback by 2012 in review « Gas station without pumps — 2012 December 31 @ 11:18 |
[…] NSF “clarifies” Broader Impacts […]
Pingback by Blogoversary 3 | Gas station without pumps — 2013 June 1 @ 20:01 |