Accountable Science

The COMPETES Act Needs to Demonstrate an Accountability Attitude

SOURCE: NSF The National Science Foundation headquarters in Arlington, Virginia, where thousands of government grants are written in support of American science and innovation.

Today, to invoke one of the great School House Rock! short films, the America COMPETES Reauthorization Act of 2010 is still just a bill, sitting there on Capitol Hill. If it becomes a law, however, it is likely to have broader ramifications than many interested in its enactment have considered—and that’s a good thing.

More than 750 organizations around the country, including especially science and technology-related businesses, universities, and their associated associations, support the House version of the bill, H.R. 5116, the purpose of which is “to invest in innovation through research and development and to improve the competitiveness of the United States.” H.R. 5116 authorizes more than $84 billion for basic research and innovation over the next five years, putting the United States on the road to doubling the budget of several federal agencies, including the National Science Foundation, over the next 10 years. The Senate bill, S. 3605, which is somewhat different from the House version, has passed through committee and has been sent to the full Senate for debate.

Unfortunately, almost everyone concerned with the America COMPETES Reauthorization Act of 2010 seems focused on the final amount of the authorization. Everyone seems caught up in wrangling over whether increasing the country’s investment in science and innovation research is a good economic move or not. Science interest groups, of course, are already convinced that whatever the final amount authorized, it won’t have been enough.

But besides the amount of money to be authorized, there is another important aspect of the House COMPETES reauthorization bill, H.R. 5116, that has received little attention. Title II, Section 214 of the bill revisits NSF’s long-standing Broader Impacts Criterion policy, or, BIC, which has quite an interesting history that has begun to be detailed in the scholarly literature.

Since fiscal year 1998, NSF has had only two merit review criteria for grant proposals, asking:

• “What is the intellectual merit of the proposed activity?”
• “What are the broader impacts of the proposed activity?”

For more than 10 years, peer reviewers of NSF proposals have been asked to consider both whether the proposed activity constitutes good science and whether it would have an impact (presumably positive) on the larger society outside of science, as well as within the scientific community. NSF instituted its more streamlined approach to merit review (prior to FY 1998, there were four criteria) in an effort both to simplify the process and to raise the profile of the societal benefits of public funding for basic research.

But despite its goals of simplifying the peer review process and ensuring that the taxpayers’ dollars are used on research that promotes overall social well-being, some members of the scientific community initially resisted BIC—and many continue to resist. Some have even noticed the attention to BIC in Sec. 214—and they don’t seem to like it.

Sec. 214, all of two pages long, is divided into two subsections titled “Goals” and “Policy.” The first outlines eight specific national needs that NSF’s Broader Impacts Criterion is well-suited to address, including increasing the economic competitiveness of the United States, developing a globally competitive workforce, increasing partnerships between academia and industry, increasing the participation of underrepresented groups in science and engineering, increasing national security, improving science education, and enhancing scientific literacy.

These goals are lifted directly from NSF’s report to Congress on BIC—a report requested in the original America COMPETES Act of 2007. So, as the House report (111-478) language makes clear, these eight national goals are not some sort of imposition of the will of Congress on NSF or those members of the scientific community who apply for NSF funding or review its grant proposals. Nor does the list of goals represent any intention of Congress to delimit new areas that scientists and engineers applying for NSF grants must address. As the report language states:

“the Committee does intend to leave room for innovation within the broader impacts portion of a proposal.”

Nevertheless, this first subsection is an attempt by Congress to help clarify BIC for those scientists and engineers who claim to be confused by it. What count as broader impacts activities? Congress is telling us the sorts of things it thinks should count, without trying to tie anyone’s hands in responding to BIC.

The subsection on “Policy” is actually much more directive than the first. In this subsection, Congress is telling NSF and the scientific community that they need to get serious about increasing the quality of their approaches to satisfying and reviewing BIC. As the report language puts the point:

“the [House] Committee [on Science and Technology] is concerned that this criterion has been in place for more than 10 years now with little effort put toward evaluation of its impact or toward holding anyone, including NSF funded investigators, accountable for their efforts to satisfy the criterion.”

Indeed, the committee even raises the rather complicated issue of broader impacts expertise:

“The Committee believes that if a broader impacts review criterion is to be applied at all, it should be treated with the same rigor as the scientific merit review criterion.”

In response, in the “Policy” subsection of Sec. 214 Congress directs the NSF director to:

• Develop more concrete metrics to apply to BIC review process
• Develop institutional support for researchers to help them address BIC in their research and grantmaking
• Have NSF require investigators to provide evidence of having received such institutional support, either from their own or another university or from someone with BIC-relevant expertise

The most recent similar requirement came three years ago, when the America COMPETES Act of 2007 required universities to certify that they have a plan to instruct all students who receive NSF support in the Responsible and Ethical Conduct of Research. The research ethics requirement is still reverberating around campuses across the country, and it will be interesting to see the ramifications of a new BIC requirement, should the language of the House version of the bill make it into law.

Part of the intrigue stems from the fact that many scientists claim that BIC is unfair, burdensome, or confusing. Why they make these claims is still an open question; but the explanation surely has something to do with expertise. Scientists trained in good science are thereby trained to address the first criterion, intellectual merit. But typically, they receive no training in the broader impacts of their science. So, many resist the requirement to give an account of those impacts and claim an inability to judge them as expert peers.

But this resistance to the Broader Impacts Criterion is hardly universal. Indeed, there are several groups that are trying to help those scientists and engineers confused by BIC to address broader impacts in their proposals and in their research. Cases in point:

• The team at COSEE-NOW, one of NSF’s Centers for Ocean Sciences Education Excellence
• The folks at NISE Network, experts in informal education in nanotech;
• The leaders of the CIRTL Network, who have seen the opportunity to leverage BIC to help integrate research and education
• The Scientific Freedom, Responsibility, and Law Program at the American Association for the Advancement of Science , which is well-suited to addressing the issues associated with BIC

There are also individual scientists who have embraced BIC:

• Diandra Leslie-Pelecky, whose approach to the physics of NASCAR provides one example of how scientists might make their research more accessible to the rest of society
• Bruce J. MacFadden, a paleontologist who has taught a graduate course on broader impacts and recently served a stint as a rotator at NSF
• and Sharon Franks, a Ph.D. scientist who has tried to spark a cultural change in favor of broader impacts among scientists as Director of the Research Proposal Development Service at the University of California-San Diego

There have also been champions of broader impacts at NSF, such as Luis Echegoyen, whose April 2008 Dear Colleague Letter written as the director of the Division of Chemistry remains one of the best examples of NSF’s commitment to increasing the quality of attention to broader impacts among scientists and engineers.

These champions of BIC share, with other scientists and engineers and no doubt with some members of Congress, the belief that investments in science and engineering lead to societal benefits. A belief they seem not to share with those scientists and engineers resistant to BIC, however, is that if science and engineering do lead to societal benefits, we ought to be able to demonstrate that fact. Congress, too, is prone to link research and development expenditures to demonstrable results—they want to see some evidence of a return on their investment of taxpayer dollars. It is just this sort of accountability attitude that Sec. 214 of the House version of the COMPETES bill demonstrates.

In emphasizing accountability, Congress is not rewriting NSF’s merit review criteria. The Broader Impacts Criterion may or may not be rewritten one day, but if and when that happens it will be the task of the National Science Board, NSF’s policy branch. NSB has recently formed a new Task Force on Merit Review, which has been charged to examine BIC to see whether any changes are warranted. So stay tuned! But Congress isn’t on the board. Nevertheless, when Congress speaks about their merit review system, NSF is likely to take notice. And in the case of BIC, that’s a good thing.

There is one more aspect of the report language that is a potential bombshell, and it is actually contained in the introductory sentences of the report on Sec. 214: “The Committee understands that the purpose of the broader impacts review criterion, first applied by NSF in the mid-1990′s, is to increase the impact of NSF supported research on individual and societal well being. The Committee applauds the National Science Board for having recommended a broader impacts review criterion, and believes it should be applied across more agencies than just NSF” (emphasis added).

The idea that public science and technology funding agencies need to demonstrate the results of the public’s investment in the research they sponsor is hardly new. All federal agencies write annual reports detailing their efforts to comply with the Government Performance and Results Act of 1993. But using peer review to demonstrate accountability by employing a societal impacts criterion is a practice in which NSF leads the way in the United States. This fact has not gone unnoticed, at least by members of Congress. Indeed, to my knowledge, the idea that other federal agencies should consider adopting something like NSF’s broader impacts criterion was first put forward by then-Sen. Barack Obama in October 2008.

Since it is “hidden” in the section on NSF’s Broader Impacts Criterion, one wonders how long it will take other federal science funding agencies to pick up on this idea. It would serve these agencies well to consider joining this ongoing discussion, if only because it speaks to issues with which they are already concerned—even if they do not currently have a societal impacts peer review criterion. Moreover, the conversation already extends far beyond concerns with NSF.

At a conference in Tokyo that I recently attended, I heard a talk given by Helga Nowotny, an academic expert on science and society and president of the European Research Council, in which she suggested that we need “new institutions” that recognize the co-evolution of science and society. NSF’s Broader Impacts Criterion marks them, for now, as one of the institutional leaders of what I think is a still growing trend in the co-mingling of scientific and societal concerns. Given the issues NSF has faced since the introduction of BIC, the fact that anyone would suggest that other agencies adopt a similar criterion is striking evidence of this trend.

There is another reason for us all to consider taking part in this conversation about science and society. Although there will be inevitable disagreements, there are some important things about which many of us will agree. I suggested earlier that the debate surrounding the current COMPETES bill had become hung up on wrangling about the final amount of the authorization. If we can get away from the argument over how much to spend on science and instead find things to discuss that we all agree are important—such as how to ensure the science we do fund creates the greatest benefit for society—then maybe we can find some common ground on which to move forward. Rather than haggling over each and every dollar of science funding, we should instead focus the discussion on maximizing the positive impacts—and minimizing the negative impacts—that science and technology have on society and human flourishing.

J. Britt Holbrook is the Assistant Director of the Center for the Study of Interdisciplinarity at the University of North Texas. Holbrook was Principal Investigator on an NSF grant for a workshop addressing issues with Broader Impacts Criterion policy, and he is currently co-Principal Investigator on another NSF grant investigating peer review at 6 funding agencies around the world. The author gratefully acknowledges the support for this research provided by the US National Science Foundation (NSF) under grant No.0830387, while emphasizing that any opinions, conclusions, and recommendations expressed here are those of the author and do not necessarily reflect the views of NSF, or any of its employees.

Comments on this article

By clicking and submitting a comment I acknowledge the Science Progress Privacy Policy and agree to the Science Progress Terms of Use. I understand that my comments are also being governed by Facebook's Terms of Use and Privacy Policy.