Attack on science, or a wakeup call? FY 2013 NSF Political Science Research Funding Eliminated by House

FY 2013 NSF Political Science Research Funding Eliminated by House.

How should political scientists respond to such an apparent attack from Congress? Let me present two options: (a) the typical appeal to the fact that these grants all went through peer review by experts in the field (that is, an appeal to the intellectual merit of the grants); or (b) an appeal to the broader societal impacts of funded projects in political science.

I admit that this is a false dilemma — the correct answer is both. But if they were forced to choose, I would advise them to pick broader impacts every time. Why? Because it’s broader impacts on society — some demonstration of the worthiness of political science research to receive public funding — that Congress needs.

NSF-funded political scientists should have a quiver full of arrows to target claims that political science is a waste of taxpayer money. Every funded project is supposed to have given an account of its broader impacts as part of the grant proposal. Have they done so? Well, we can look: this account is also supposed to appear as a separate paragraph of the one-page abstract, which is publicly available.

I’m only going to provide a cursory examination here. The broader impacts of funded projects should be easy to identify (preferably even labeled ‘broader impacts’), otherwise PIs are missing an opportunity.

There are 215 active awards in political science. That should mean 215 broader impacts statements clearly labeled as such. I know already before looking, however, that I will not find 215 such statements — and not because some of the awards are for collaborative research, such that they share the same abstract. The political science community would really be an anomaly if they had 100% participation here — this despite NSF’s claims to have the problem of broader impacts compliance fairly well under control (Figure 23 here shows the latest data — since 2003, fewer than 1% of proposals have been returned without review for failure to address both criteria, and there’s a general trend toward lowering the percentage).

The fact is, many funded projects do not include separate broader impacts statements in the one-page abstract. I am not singling out political science in this regard. But since the House is singling out political science, let’s take a look. I’ll just focus on some of the big dollar awards (say, at least $700,000).

Here is one. Award Abstract #1067949 Increasing Power & Decreasing Costs: A New Method for Drawing High-Quality National Probability Samples of U.S. Citizens. Unfortunately, there is no separately labeled broader impacts statement. There is one paragraph — if one sentence counts — at the end of the abstract:

If this project is successful, it could yield significant cost savings for face to face surveys while at the same time providing more powerful statistical results.

Saving costs sounds good, though it’s clear the costs targeted are those of researchers:

[The project] is designed to enable the survey research community to improve data collection, increase statistical power, yield high response rates, and reduce costs per completed interview.

This may be of real value to the political science research community (in terms of intellectual merit), or even in terms of allowing them to spend a larger portion of their grant funding on things other than surveys; but what is needed is attention to the broader impacts of the research on society. Someone may want to suggest that my criticism is misplaced. After all, these folks obviously know what they’re doing — their project went through peer review, and they were awarded over $800,000 to perform the proposed research. But this misses the larger point — that researchers in political science ought to demand clear statements of the broader societal impacts of the proposed research before they argue a project should be funded. In fact, in this particular case, according to the letter of NSF’s law, this proposal should have been returned without review.

Here‘s one that’s a bit better: Award Abstract #0648205 EITM Summer Training Institute, a grant for $846,166 to close the gap between theoretical and empirical methods. This sounds, again, like something that may be of value only to researchers; but, we are told in the final paragraph:

The value of EITM-based training extends beyond scholarly development. Political, policy, and social decisions depend just as much , perhaps even more, on sound reasoning and empirical inferences. A new generation of social scientists well trained in considering not just the academic merit but the social value of logically conistent, emperically grounded conclusions, acting , that is, on what Donald Stokes called “Pascal’s Quadrant,” [sic] can do no less than improve implementation of socially desirable outcomes.

There are still problems with this as a broader impacts statement. It is not labeled as such, and the proposers simply assert that acting on “Pascal’s Quadrant” (I assume they meant Pasteur’s Quadrant) will inevitably “improve implementation of socially desirable outcomes.” But at least they pay the notion of broader societal impacts lip service.

Here’s another: Award Abstract #0937727 Collaborative Research: American National Election Studies (ANES) 2009-2013 to answer the question, “Why did America vote as it did on Election Day?” I can imagine that the answer to this question might be of interest, especially to members of Congress. The (again unlabeled) broader impacts statement says:

By generating large multifaceted datasets of high quality, the ANES will equip researchers to learn new and important lessons about the world of politics. These data will be distributed widely and quickly to serve thousands of scholars and to be used in classrooms around the world to enrich research and education. The ANES will help to inform the nation about itself, exploring the causes and consequences of voting behavior andelectoral [sic] outcomes. With such knowledge, the policy [sic] will be better equipped to nurture and refine its system of government.

The goal of informing the nation about itself is laudable, though it is unclear exactly how the ANES will accomplish this goal. How will we be informed? How will the ANES connect its research to the potential users of its research — those in the nation outside of the field of political science? Perhaps this question is answered in the proposal itself, but it is notably absent from the award abstract. And it really shouldn’t be, given that the ANES has received over $5 million in funding to date.

Here’s another award of $700,000: Award Abstract #0962258 A Multi-level, Agent-based Model for Identifying the Factors that Enable or Constrain International Climate Change Negotiations. Broader impacts?

This research will help inform stakeholders — including citizens, interest groups, businesses, governments, and international organizations — so that they better understand the opportunities in a globally connected network of decision makers.

Again, we have an account of who might benefit — and the research is aimed at stakeholders outside of the Academy. That’s good. We also have an account of what the benefit might be. That’s also good. But missing is an account of how the researchers will connect their research with those stakeholders who might benefit from it.

There are several other funded projects of over $700,000 that do not address broader impacts at all in the award abstract. Again, these should not only not have received funding, but rather should have been returned without review.

There is one project in the over $7oo,ooo category, however, that does a pretty good job. This award of over $13 million, addresses the issue of infrastructure for research and education: Award Abstract #0824618 A Proposal to Continue ‘A National Data Program for the Social Sciences’:

Broader Impacts

The NDPSS has already had enormous impact beyond the boundaries of the survey itself. The GSS is held as the gold standard by which many other survey data collection activities are measured, and the ISSP program has led to innovations and developments in cross-cultural and cross-national research. The GSS thus serves as a model that is being emulated elsewhere, such as in the recently created European Social Survey program. This renewal project will pioneer the development of a state-of-the-art Interview-to-Internet archiving and dissemination system that will speed access to the computer-assisted interview data and allow users to access all information associated with each variable in a database via extensive hypertext links. The contributions of the GSS to the teaching of quantitative social science analysis are unprecedented. Its accessibility and ease of use has added value far beyond the original data collection efforts, and researchers in the social sciences continue to mine the data both old and new to advance knowledge and test theories. The GSS and ISSP program is part of the core infrastructure of social science research in the U.S.

Although it would be ideal to show connections between this project and non-academic stakeholders, it is by no means required under the current Broader Impacts Criterion, which includes impacts on research and education infrastructure. This project certainly does that, and the broader impacts statement is (mercifully) labeled as such.

The fact is, however, as a whole, neither proposers to nor reviewers for the Political Science Program at NSF have been doing their jobs when it comes to NSF’s Broader Impacts Criterion. The latest Committee of Visitors Report for Political Science (FY2010) notes:

We notice that some individual reviewers continue to have difficulty addressing broader impacts. Sometimes, for example, the reviewer simply repeats his/her statement about intellectual merit. (p. 149)

Yet this failure on the part of individual reviewers might be — at least partly — attributable to the fact that the proposals themselves do not do a good job of addressing broader impacts! On one hand, the reviewers shouldn’t be criticized for not articulating broader impacts that are not proposed; on the other, they perhaps deserve blame for not penalizing proposers who do a bad job with broader impacts.

We at CSID have argued before that if researchers in general don’t do a better job articulating the societal benefits of their research, someone else will do it for them. With political science, the situation seems to be more dire: if political scientists don’t articulate the benefits to society of political science research, society may simply refuse to fund it. This post from Matt Wood earlier today on the LSE Impact Blog raises similar issues facing political scientists in the UK, though there the situation seems less acute. But it really illustrates the same bottom line: political scientists need to do a better job articulating the relevance of their research. They should seize the opportunity presented by NSF’s Broader Impacts Criterion before it’s too late.