NSF and Public Accountability

New, More Prescriptive ‘Merit Criteria’ May Hinder Science Progress

SOURCE: Science Progress

Last week the National Science Board, or NSB, issued new draft criteria for the peer review of proposals submitted to the National Science Foundation, or NSF. The two “merit review” criteria of the last 14 years, intellectual merit and broader impact, remain, but they are now framed by a preamble that offers a set of principles that relate the criteria to a set of national goals. The proposed changes are significant and should draw the attention of both the scientific and policy communities.

Public science agencies worldwide find themselves pulled in two directions: Scientists require freedom to pursue research wherever it leads, while funders demand that research meets public needs.

Balancing the competing demands of autonomy and accountability is a constant challenge. Scientific curiosity that focuses too much on “blue skies” research risks losing the support of funders. But funding criteria that are too prescriptive can end up with bad science. Changing circumstances—scientific breakthroughs, success or failure in wars, budget surpluses or deficits, and cultural and political trends—mean that the relationship between science and society must regularly be renegotiated.

Public science agencies worldwide find themselves pulled in two directions: Scientists require freedom to pursue research wherever it leads, while funders demand that research meets public needs.

A brief history of NSF’s merit criteria

NSF is at the frontlines of these pressures. NSF is tasked with funding basic (i.e., nonmission) science. But Congress is interested in funding basic science for the practical results that accrue. Heir to the narrative outlined by Vannevar Bush in Science – the Endless Frontier (1945), NSF has been a model agency for strong autonomy advocates who argue that science progresses best via a paradox: Scientific research will be most societally relevant when scientists are left to pursue their own creative impulses with a minimum of outside direction.

This worked in the aftermath of World War II. But by the mid-1990s, political culture had begun to shift toward the pole of accountability. One sign of this was the 1993 Government Performance and Results Act, or GPRA, an accountability initiative across the federal government led by Vice President Gore. A second sign was the refashioning of the peer review criteria for the review of grant proposals at NSF in 1997.

Previously, NSF’s “merit” criteria had focused mainly on the intrinsic scientific merit of the research. Beginning in 1997, however, NSF changed the review criteria to two: intellectual merit and broader impacts. The first criterion called for reviewing proposals by disciplinary definitions of excellence. But this was now counterpoised by a second criterion that made transdisciplinary demands upon scientists, asking them to align their research with societal needs.

These new criteria stirred controversy across the scientific community. Biologists knew how to judge the biological aspects of proposals, but what particular expertise do they have in judging broader impacts? Over the last decade, however, the “broader impacts” criterion has come to be integrated within the culture of science. With some grumbling, scientists have become more comfortable with thinking of their research in terms of having both scientific merit and societal impact.

Nevertheless, controversy remained about what properly counted as a broader impact, and about how one balanced the broader impacts with the intellectual merit of proposed research. And as scientists were adjusting to the broader impacts criterion, our national focus on accountability continued to grow. Growing distrust of government, the most serious economic crisis since the Depression, and the passage of time made it clear that a review of the merit review criteria was called for. And so in May 2010, the National Science Foundation announced a top-to-bottom review of both criteria.

Changes to the merit criteria

Last week NSB announced draft criteria for the review of proposals submitted to the Foundation. And there are significant changes to the two criteria. Addressing the broader impacts criterion now means showing “which national goal (or goals) is (or are) addressed in the proposal?” The complete list of goals is:

• Increased economic competitiveness of the United States
• Development of a globally competitive STEM workforce
• Increased participation of women, persons with disabilities, and underrepresented minorities in STEM
• Increased partnerships between academia and industry
• Improved pre-K-12 STEM education and teacher development
• Improved undergraduate STEM education
• Increased public scientific literacy and public engagement with science and technology
• Increased national security
• Enhanced infrastructure for research and education, including facilities, instrumentation, networks, and partnerships

Those familiar with the old version of the broader impacts criterion will see much holdover in the new. But the crucial differences can be summarized in terms of three points:

A focus on economics and national security

Four of the nine national goals are concerned with economics; a fifth with national security. (The remaining four focus on education and outreach.) The list of national goals does not include other values such as environmental sustainability or alleviation of social inequality (health issues, of course, are the province of the National Institutes of Health). Besides ignoring these other issues, there is also the danger that the focus on economics could become counterproductive, insofar as it discourages more idealistic students from entering science and engineering. There is evidence, for instance, that women and minorities especially are more attracted to science and engineering for meeting human needs rather than economic development.

Clarity at the expense of flexibility

The new broader impacts criterion shifts the judgment of what counts as a broader impact away from proposers and reviewers. Previously, the individual PI had flexibility to define broader impacts as he or she saw fit—as long as the proposer could answer the question:

What may be the benefits of the proposed activity to society?

Under the proposed new criteria, proposers and reviewers are limited to the list of national needs. Easier? Perhaps. But unless the list is made representative and nonexhaustive, proposers will be restricted to addressing only those national goals that have appeared on the list. This also restricts the list to current national needs, tying our hands to respond to new and future challenges.

Weighting the criteria

The new principles and criteria also contain an interesting ambiguity. NSF’s portfolio of funded projects is supposed to advance the nine national goals “collectively.” Whereas all proposals had to address broader impacts under the previous criteria, under the new principles and criteria, “Broader impacts may be achieved through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by the project but ancillary to the research.” Does this mean that there is now no mandate that each individual proposal address broader impacts directly?

There are positive aspects to the new criteria. The idea of a set of principles is helpful. And there is a new emphasis on how the proposed broader impacts are to be achieved. Once one chooses the national goal advanced through the research, the new guidelines focus on questions of implementation, such as:

• Is there a well-reasoned plan for the proposed activities, including, if appropriate, department-level or institutional engagement?
• Are there adequate resources available to the PI or institution to carry out the proposed activities?

These sorts of procedural questions should help proposers and reviewers. They also strengthen the parallel between the two criteria. After asking about the role the proposed research will play in advancing knowledge, the new intellectual merit criterion also focuses on logistics:

• How well conceived and organized is the proposed activity?
• How well qualified is the individual or team to conduct the proposed research?
• Is there sufficient access to resources?

This parallelism is useful, for it helps dispel the illusion that “predicting” impacts is somehow more difficult than “predicting” intellectual success. Reviewers who reflect on their judgments regarding intellectual merit will recognize that they do not set themselves the task of predicting the future, but rather focus on whether the proposer has thought through their proposed research. Reviewers ask questions—is it feasible, is it an important topic, and so forth—about the planning and thought that have gone into the proposal, about whether the research has a chance to succeed. No reviewer is a fortune teller. The same is the case for judging broader impacts. The point is not to predict the future but to ensure the proposer has thought through what the broader impacts might be, who might benefit, and how to encourage that to happen.

As suggested above, we believe the proposed changes in the review criteria are too narrow and too restrictive. But we also know this is an ongoing process that will continue to be renegotiated as cultural attitudes shift.

Doubtless the proposed criteria and principles will be revised. NSF solicits comments until July 14, and the announcement notes that “It is expected that NSF will develop specific guidance for PIs, reviewers, and NSF staff on the use of these criteria after the drafts are finalized.” There is thus still time for the scientific community, and the science policy community, to weigh in.

In closing, it is worth noting that students of science policy have seen this movie before. Over the last 15 years, the European Commission has struggled with integrating societal impacts into the peer review process. Across Framework Programmes 5, 6, and 7 (running from 1998 to 2013), the EC has swung back and forth in terms of how to interpret broader impacts, and to what degree individual PIs should be given discretion in fulfilling the mandate. The EC is currently designing Framework Programme 8 and is again thinking through the tension between autonomy and accountability. It is a perennial question that each generation will struggle to strike the right balance on.

Robert Frodeman is professor of philosophy and director of the Center for the Study of Interdisciplinarity at the University of North Texas. J. Britt Holbrook is research assistant professor of philosophy and assistant director at the Center for the Study of Interdisciplinarity at the University of North Texas. The authors gratefully acknowledge the support for this research provided by the US National Science Foundation (NSF) under grant No.0830387, while emphasizing that any opinions, conclusions, and recommendations expressed here are those of the authors and do not necessarily reflect the views of NSF, or any of its employees.

Comments on this article

By clicking and submitting a comment I acknowledge the Science Progress Privacy Policy and agree to the Science Progress Terms of Use. I understand that my comments are also being governed by Facebook's Terms of Use and Privacy Policy.