Research-funding agencies are forever trying to balance two opposing forces: scientists' desire to be left alone to do their research, and society's demand to see a return on its investment.
The European Commission, for example, has tried to strike that balance over the past decade by considering social effects when reviewing proposals under its various Framework programmes for research. And the Higher Education Funding Council for England announced last year that, starting in 2013, research will be assessed partly on its demonstrable benefits to the economy, society or culture.
But no agency has gone as far as the US National Science Foundation (NSF), which will not even consider a proposal unless it explicitly includes activities to demonstrate the project's 'broader impacts' on science or society at large. "The criterion was established to get scientists out of their ivory towers and connect them to society," explains Arden Bement, director of the NSF in Arlington, Virginia.
“The criterion was established to get scientists out of their ivory towers and connect them to society.”
Unfortunately, good intentions are not enough to guarantee success, says Diandra Leslie-Pelecky, a physicist at the University of Texas in Dallas who is active in popular science writing and other forms of outreach.
Leslie-Pelecky remembers a pilot project she carried out in 2001, when she was at the University of Nebraska in Lincoln. In many ways, it was typical of the kinds of things that NSF-funded researchers do to fulfil their broader-impacts requirement. She took three female graduate students on weekly visits to local classrooms, where they spent 45-minutes leading nine- and ten-year-old children in practical activities designed to teach them about electricity and circuits. The visitors also talked about their lab work and careers. In addition, Leslie-Pelecky did something less typical of broader-impacts efforts: she brought along education researchers to study the effect of this interaction on the children's perception of scientists.
Those assessments were startling, she says. After three months, most of the students said that they still weren't sure who these young 'teachers' were – except that they couldn't possibly be scientists. In their minds, scientists were unfriendly, grey-haired old men in white lab coats1.
"And that's what I worry about with broader impacts," says Leslie-Pelecky. "There are a lot of people putting time and effort into [these sorts of activities] and they have no idea if they're making any difference or not."
Many NSF-funded researchers find the foundation's definition of broader impacts to be, perhaps unsurprisingly, broad, and frustratingly vague. Among the examples of activities listed in the foundation's proposal guide are: developing educational materials for elementary, high-school and undergraduate students; involving these students in the research where appropriate; creating mentoring programmes; maintaining and operating shared research infrastructure; presenting research results to non-scientific audiences such as policy-makers; establishing international, industrial or government collaborations; developing exhibits in partnership with museums; forming start-up companies; and giving presentations to the public.
Because it lacks conceptual clarity, the broader-impacts requirement often leaves researchers unsure about what to include in their proposals, and leads to inconsistencies in how reviewers evaluate applications. "Broader impacts were designed to be open, but openness confuses a lot of people," says Luis Echegoyen, the division director for NSF chemistry.
To make matters worse, the NSF has made little attempt to systematically track how its broader-impacts requirements are being met, or how much grant money is being spent in the process. Nor does it have a system in place to evaluate the effectiveness of the various projects.
These problems with the broader-impacts requirement have been confirmed over the past decade in studies from the National Academy of Public Administration and elsewhere. In March, the NSF's oversight body, the National Science Board, launched a task force to examine how broader impacts can be improved. Chaired by Alan Leshner, chief executive of the American Association for the Advancement of Science in Washington DC, the task force is not expected to make its recommendations until 2011. In the meantime, a small number of academic institutions are already exploring ways to make broader-impacts efforts work better.
After all, says Ralph Nuzzo, a chemist and materials scientist at the University of Illinois in Urbana–Champaign, most US scientists have come to accept — even if grudgingly — that it is probably a good idea to demonstrate the wider implications of their work. "People want to do the right thing," says Nuzzo. "It's just hard to know what that is."
Scientists get creative
The film Molecules to the Max was created by an NSF-funded nanoscience centre.RENSSELAER POLYTECH. INST.Nonetheless, there have been some successes. At Rensselaer Polytechnic Institute in Troy, New York, for example, the NSF's Nanoscale Science and Engineering Center for Directed Assembly of Nanostructures sponsors the Molecularium project, which has produced teachers' materials on nanoscience and an animated three-dimensional IMAX film called Molecules to the Max. At the University of Wisconsin–Madison, microbial biochemist Douglas Weibel and his group have prepared a child-friendly, interactive display about microscopy that they exhibit every year at the university's one-day public science exposition. At Stanford University in California, chemical engineer Andrew Spakowitz spends two to three hours a week working with graduate and undergraduate students to provide workshops for patients at Stanford's Lucile Packard Children's Hospital, most of whom are unable to attend school. Spakowitz and his group create the workshops that cover topics such as pH and gravity, and lead the hands-on activities at the hospital.
“It makes scientists think more explicitly about how their work is connected to enhancing benefits to society.”
Some say that the broader-impacts criterion has helped to catalyse a change in the research-focused culture of academic science. "It makes scientists think more explicitly about how their work is connected to enhancing benefits to society," says Robert Mathieu, chair of the astronomy department at the University of Wisconsin–Madison and director of the university's NSF-funded Center for the Integration of Research, Teaching, and Learning (CIRTL). As an example, Mathieu points to the NSF's prestigious CAREER award for junior faculty members, which requires that applicants propose educational activities, such as designing courses and carrying out public-outreach activities, that are integrated with the proposed research. He has sat on several review panels, and says that the education section of proposals has grown in length and sophistication over the years.
Stanford chemical engineer Andrew Spakowitz (right) teaches children at the Lucile Packard Children's Hospital as part of his NSF award.K. HO, LUCILE PACKARD CHILDREN'S HOSPITALBut despite the NSF's efforts to educate scientists about broader impacts through websites, workshops and conference sessions, most still approach the criterion with confusion and dread. Researchers often end up repackaging what they're already doing. "Overwhelmingly," says Echegoyen, "the number one broader impact that most people in the chemistry division are using is 'training graduate students and postdocs.'"
One problem is that the kind of support network that researchers take for granted — working with collaborators, sharing ideas and advice, learning from published results, attending conferences — is still rudimentary when it comes to broader impacts. A useful model could be the network of technology-transfer offices that are found on many US campuses, which have been instrumental in helping researchers to maximize the commercial effect of their research.
A preliminary network for broader impacts already exists. Stanford, for example, has an Office of Science Outreach, which helped Spakowitz to make the initial contacts to get his project started at the hospital. And Mathieu's centre at the University of Wisconsin–Madison is part of a network of six CIRTLs located at research campuses such as Vanderbilt University in Nashville, Tennessee, and Texas A&M University in College Station. The Wisconsin centre runs workshops and conducts individual consultations with faculty members needing assistance with integrating broader-impacts activities into their grant proposals. The other CIRTLs are moving towards similar sorts of programmes. Mathieu and his group are putting together plans to expand this network to 20–25 universities. Their ultimate goal is for any US research university that wants its own CIRTL to have one, creating a community that shares best practices among its researchers and other professionals, and develops the expertise to effectively broaden impacts. Mathieu estimates that establishing CIRTLs at the nation's top research universities would cost roughly US$100 million over five years.
Yet such ideas lead to a more fundamental question. Is having every principal investigator working individually on broader impacts — for which many are inexperienced and untrained — the most efficient way of achieving the maximum effect?
Some scholars say no. In a paper published last year, Warren Burggren, a biologist and dean of the College of Arts and Sciences at the University of North Texas in Denton, writes that the job of implementing broader impacts should fall to the researcher's institution, not to the researcher him or herself2. The institution, be it college, department or centre, would pool a portion of the NSF grants obtained by its members and hire the professionals needed to broaden impacts effectively. Scientists should still be involved, but the coordination would happen at the institutional level. "I think it will be more efficient, because you've got people doing what they're trained for," says Burggren.
Another idea, suggested by Barry Bozeman, a science-policy expert at the University of Georgia in Athens, is for the NSF to create specific research programmes with strong broader-impact goals around areas in which the effects are important and obvious, such as climate change3. Bozeman says that the NSF is already following this strategy with awards that, for example, promote the recruitment and retention of women in academic science.
The NSF's broader-impacts requirement took its current form in 1997, when the foundation simplified the criteria used by reviewers to evaluate proposals. Two of the four criteria — the intrinsic scientific merit of the project, and the soundness of the team's approach — were merged into one, known as 'intellectual merit'. And the other two — the utility or relevance of the project, and its effect on the infrastructure of science and engineering — were collapsed into 'broader impacts'.
For the first few years, many proposers and reviewers ignored this second criterion, treating it with the same disregard that they had previously expressed towards 'utility or relevance'. It was only in 2002 that the NSF cracked down, announcing that any proposal that didn't separately address both the intellectual-merit and the broader-impacts criteria would be returned without review.
The right track
Of the few small-scale efforts to track and assess the broader-impacts requirement, none has been conclusive. In 2008, for example, the NSF sent Congress a report on broader impacts, as mandated in the 2007 America COMPETES Act — but included little more than anecdotal descriptions of specific research projects. The chemistry division recently contracted with an outside company to assess the broader-impacts activities of a sample of its grantees, but the project fell into limbo when the company dissolved. And the geosciences directorate carried out informal surveys of broader-impacts activities in the ocean and Earth sciences, which yielded some results, but also showed that the research and the broader-impacts work were often so interwoven that it was difficult to tease them apart.
Mostly, evaluation happens as a by-product of other NSF activities — routine reviews of grantees' annual reports, for example, or the regular review of programmes at each division carried out by a committee of external scientists.
“By not tracking broader-impacts activities, the NSF undervalues its true contribution to society.”
"By not tracking broader-impacts activities, the NSF undervalues its true contribution to society," says Melanie Roberts, an assistant director at the Colorado Initiative in Molecular Biotechnology at the University of Colorado in Boulder who has analysed the broader-impacts statements from recent grant abstracts. "It is missing an opportunity to create a knowledge base of how to carry out broader-impacts activities effectively and reward those who do a good job."
The confusion that persists despite the NSF's repeated attempts to clarify broader impacts perhaps reflects more fundamental issues about the relationship between science and society, says Britt Holbrook, a philosopher of science at the University of North Texas. Is the NSF 'passing the buck' by asking scientists to meet what is essentially a political goal: demonstrating the benefits of science?
ADVERTISEMENT
"My hypothesis is that the NSF has passed some of that burden to the people getting funded," says Holbrook, who has a grant from the foundation to study how different funding agencies incorporate societal impacts into their review process. "But when you do that, you get push back from the scientific and engineering community because it goes against the traditional idea of peer review," which is designed to assess work at a technical or scientific level.
And how does the NSF show impact, given that the agency's specialty, basic research often doesn't have an immediate pay-off, or else has a pay-off that is difficult to quantify? It's a delicate balancing act, says Neal Lane, a physicist at Rice University in Houston, Texas, who was the NSF director from 1993 to 1998 when the broader-impacts criterion was first implemented. It's important to get scientists to think about how their work affects society, he says. "But one has to be careful not to push it too far. If the NSF moves too far in the direction of doing things that have short-term benefits, then I think that is not consistent with the NSF's mission, and that would not be good for American science, engineering and technology."
