Featured Photo: 

Robert Frodeman is Professor of Philosophy at UNT and the Director of the Center for the Study of Interdisciplinarity (CSID).  He brings expertise in continental philosophy, environmental ethics and philosophy, philosophy of science policy, science and technology studies, and the theory and practice of interdisciplinarity. Britt Holbrook is a Research Assistant Professor of Philosophy and the Assistant Director of CSID.  His current research focuses on interdisciplinarity, peer review, and the relationship between science, technology, and society.   Together Frodeman and Holbrook conduct research into the theory and practice of inter- and transdisciplinary approaches to knowledge, and explore the incorporation of societal impacts considerations into the peer review process of publicly supported funding agencies, such as the National Science Foundation’s "Broader Impacts" Merit Review Criterion.

What’s the value of knowledge? Given the current economic climate – which poses difficult challenges for the university, in particular – we might assume that this is an especially pressing and timely matter. In fact, however, the question of the value of knowledge goes back thousands of years, and philosophers have always risen to the challenge of trying to answer it. Thales of Miletus, who lived during the sixth and fifth centuries BCE, was the first Western philosopher. There is a story that Thales once fell into a well while conducting astronomical observations, prompting a milkmaid to mock him for his desire to know the things in the sky while ignoring what was right in front of him. His most famous theory: “All is water.” This, as we now know, is false. So why not simply conclude that Thales – like all philosophers – is all wet?

Thales himself provides a reason not to throw the philosopher out with the bath water. One day he was discussing the value of money with some friends. (This was obviously back before money was the way we valued everything.) In any case, Thales claimed that everyone could make money if they really thought about it, and his friends challenged him to demonstrate. From his observations of the olive crop and his predictions about the weather, Thales knew that the following season would be a good one for olives. He went out and bought every olive press in the region; and he got them at good prices, since the last few harvests had been very meager. When the bumper crop Thales predicted came in, Thales had cornered the market on all the presses, and he was able to sell them at a substantial profit. Afterwards, he told his friends that his purpose was not to make money, but rather to demonstrate that philosophers could do so, if they wanted. According to Thales, however, philosophers are interested in things more valuable than money.

At the Center for the Study of Interdisciplinarity (CSID), philosophers continue to examine the value of knowledge. From the perspective of the mid- to late 20th century, societal pressures revealed gaps and inadequacies in the disciplinary structure of the academy: connections not being made, and topics not being examined. Interdisciplinary programs were developed in areas such as women's studies, area studies, and environmental studies to address these needs. This period also saw the development of a scholarly literature on interdisciplinarity (first codified by Klein, 1990) and the creation of professional societies (in the US, the Association for Integrated Studies, AIS, in 1979) devoted to exploring these issues.

However, interdisciplinarity may also be seen as the most recent expression of a set of perennial questions concerning the pertinence of knowledge for the goal of living well. Such questions span the entirety of Western culture, and reassert themselves with particular force during times of cultural change (see, for instance, Rousseau's Discourse on the Moral Effects of the Arts and Sciences (1750), Goethe's Faust (1808), or Nietzsche's On the Use and Abuse of History for Life (1873)). Today this question once again takes on renewed importance as we face a new set of challenges tied to the development of the internet and other new information technologies.

CSID places its work within this second, larger compass, seeing "interdisciplinarity" as a placeholder for more general concerns with the rapidly changing place of the academy and knowledge generally within 21st century society.

From the local and state to the federal level, society today demands greater accountability from researchers. Whether in the sciences or the humanities, knowledge production today must simultaneously be:

• Theoretically rigorous
• Policy relevant
• Culturally significant
• Economically sustainable

CSID research aims to achieve all of these goals, as well as helping others to achieve these goals through their own research.

One example is CSID’s CAPR project, pronounced ‘caper’. The Comparative Assessment of Peer Review (CAPR) is a four-year project (2008-2012) studying the changing nature of the peer review processes across six US and foreign public science agencies. CAPR is funded by the National Science Foundation's (NSF) Science of Science and Innovation Policy (SciSIP) program.

Peer review is the governing mechanism of the academy – the means to determine hiring and promotion, vetting of publications, and the distribution of research funds. As such, peer review has been a disciplinary concept, and peers have most often been defined in disciplinary terms. In other words, biologists judge biological research in terms of biological criteria, while philosophers judge philosophical research in terms of philosophical criteria. Today, however, increased calls for accountability have led to the introduction of questions of societal impact into the peer review process, essentially 'interdisciplining' peer review.

The CAPR research team examined how a select group of public science agencies around the world incorporate consideration of societal impacts¹ into the grant proposal peer review process. In addition to creating a digital repository²of more than 850 documents related to peer review, CAPR research revealed different agency approaches to questions such as:

• What should count as a broader societal impact?
• What is the proper balance between the values of intellectual merit and broader impact?
• What do questions of broader societal impact imply about what it means to be a peer?
• How is the notion of ‘peer’ changing in response to new social conditions?
• Is the peer review of proposals the best place to incorporate considerations of societal relevance into funding decision-making?

Six agencies were examined by the CAPR research team: in the US, NSF, the National Institutes of Health (NIH), and the National Oceanographic and Atmospheric Administration (NOAA); internationally, the Natural Sciences and Engineering Research Council of Canada (NSERC), the Dutch Technology Foundation (STW), and the European Commission’s (EC) 7th Framework Program. Across the period of this grant the research team held two user-engagement research workshops (one in Washington, DC, and a second in Brussels) with a wide range of public science agencies, offered midterm reports on the results of this research, and received guidance on future research needs. The first CAPR workshop was held in April 2010 in Washington, DC. A follow-up workshop, held in December 2010 in Brussels, brought the results of the previous workshop and our subsequent research to bear on the EC’s formulation of their next Framework Programme, to be known as Horizon 2020. In May 2012, a final CAPR workshop will be held at the Dalian University of Technology in Dalian, China.³

In addition to these user-engagement workshops, the team also conducted a survey to gain insight into the underlying assumptions of program officers, reviewers, and proposers concerning questions of broader impacts (Holbrook and Hrotic, under review). Publications resulting from the CAPR grant include Frodeman and Parker, 2009; Holbrook, 2010a, b and c; Frodeman and Holbrook, 2011a and b; Holbrook and Frodeman, 2011; Frodeman and Briggle, 2012; and Holbrook and Hrotic, under review. Finally, a book on Peer Review, Research Integrity, and the Governance of Science will be translated into Chinese and published by Peoples’ Publishing House, Renmin University.

The following statements summarize the findings of the CAPR team on the state of the art regarding the use of peer review to render ex ante judgments of the potential societal impact of proposed research (Holbrook and Frodeman, 2011):

• Science agencies around the world are placing increasing emphasis on funding research that has clearly identified potential benefits to society.
• Many agencies use the peer review process to assess the potential societal impacts of the research they fund.
• Agencies often encounter resistance from both proposers and reviewers to the incorporation of societal impacts considerations into the peer review process.
• Despite the well-documented resistance on the part of the scientific community to including impacts criteria in peer review, there is little evidence to suggest that peer review is in principle any less effective at ex ante assessments of societal impact than it is at ex ante assessments of scientific, technical, or intellectual merit.
• Agencies continue to experiment to find better ways to include societal impacts considerations in ex ante research evaluation.

These findings are not merely theoretical, however. CAPR research was brought to bear on the recent reconsideration of NSF’s Merit Review Process by the National Science Board, NSF’s governing branch.

Frodeman and Holbrook (2011a and 2011b) argued that NSB should maintain enough vagueness in the notion of ‘broader impacts’ to allow researchers to use the same creativity and exhibit the same autonomy that they do in terms of the ‘Intellectual Merit’ activities they propose. NSB had been considering providing a list of ‘national goals’ that the Broader Impacts Criterion was meant to help achieve. When NSB released its final ‘Review and Revisions’ in December 2011, however, the list of national goals had been removed. 

NSB came to realize that a degree of vagueness – especially when it is used intentionally – is actually a good thing: the final revisions allow proposers and peer reviewers to provide their own answers to the demand for accountability by addressing the potential transformativity of the proposed activities for both intellectual merit and broader impacts.

NSB's integration of intellectual merit and broader impact means seeing the connections between things formerly thought to be separable. NSB's new criteria recognize that in the 21^st century, our disciplinary peers are no longer our only audience. This will require an adjustment in the way we think about broader impacts: scientists and engineers will need to begin to see that even basic research must take place in the context of the needs of the users of that knowledge.

The CAPR project clearly achieves three of the four goals CSID has identified for research projects: it is theoretically rigorous (as the list of publications attests), culturally significant (as the book publication in China and the Dalian workshop demonstrate), and policy relevant (as NSB’s decisions regarding changes to NSF’s merit review process show). The only remaining question is whether CAPR – or CSID itself – is economically sustainable. CAPR did garner almost $400,000 in support from NSF. And CSID has received $600,000 over the last 4 years from NSF and NASA – enough to help propel the Department of Philosophy and Religion Studies into the top ranks of US doctoral programs in philosophy in terms of external research funding.4 This makes UNT Philosophy special. CSID, however, is unique. Despite the fact that there are thousands of interdisciplinary centers around the world, there is no other center for the study of interdisciplinarity. What is the real value of that for UNT?

[1] Societal (or ‘broader’) impacts are defined in different ways at various agencies and countries. But all seek to trace the connections between knowledge production and its use by non-academic actors.

[2] The CAPR Digital Repository can be accessed here: < http://csid-capr.unt.edu/repository>.

[3] Further information of these workshops is available here: < http://csid-capr.unt.edu/research>.

[4] In fact, starting from the 2010 Rankings of Doctoral Programs in America published by the National Research Council (NRC), UNT Philosophy would rank second behind only Carnegie Melon and ahead of Stanford and Duke in terms of external funding. If CSID alone were a department, it would rank third, just behind Stanford.

References

Frodeman, R.; Briggle, A. (2012). “The dedisciplining of peer review,” in Minerva. DOI 10.1007/s11024-012-9192-8, Published online February 11, 2012.

Frodeman, R.; Holbrook, J.B. (2007) Science's social effects, Issues in Science and Technology, p. 28, vol. 23: < http://www.issues.org/23.3/p_frodeman.html>, last accessed February 17, 2012.

Frodeman, R.; Holbrook, J.B. (2011a) NSF and public accountability: New, more prescriptive merit criteria may hinder science progress, Science Progress, June 27, 2011: http://www.scienceprogress.org/2011/06/nsf-and-public-accountability/.

Frodeman, R.; Holbrook, J.B. (2011b) NSF's struggle to articulate relevance, Science, p. 157, vol. 333, (2011). Published, 10.1126/science.333.6039.157.

Frodeman, R.; Parker, J. (2009) Intellectual merit and broader impact: The national science foundation's broader impacts criterion and the question of peer review", Social Epistemology, p. 337, vol. 23, (2009). Published, 10.1080/02691720903438144.

Holbrook, J B. (2010a) Accountable science: The COMPETES act needs to demonstrate an accountability attitude", Science Progress, September 16, 2010: http://www.scienceprogress.org/2010/09/accountable-science/.

Holbrook, J.B. (2010b) Peer review, in Eds., R. Frodeman, J.T. Klein, and C. Mitcham, Oxford Handbook of Interdisciplinarity (Oxford University Press, 2010), pp. 321-32.

Holbrook, J.B. (2010c). The use of societal impacts considerations in grant proposal peer review: A comparison of five models. Technology and Innovation, Vol. 12, 2010, pp. 213–224. DOI: 10.3727/194982410X12895770314078. Available via open access at <http://www.ingentaconnect.com/content/cog/ti/2010/00000012/00000003/art00005>, last accessed January 26, 2012.

Holbrook, J.B. and Frodeman, R. (2005) “Policy Dimensions of NSF’s Criterion 2,” in Ogmius, the Newsletter of the Center for Science and Technology Policy Research at the University of Colorado, Boulder, Number 13, Fall 2005.

Holbrook, J.B. and Frodeman, R. (2007) “Answering NSF’s Question: What Are the Broader Impacts of the Proposed Activity?” Professional Ethics Report XX (3), Summer 2007. http://www.aaas.org/spp/sfrl/per/newper50.shtml#NSF, last accessed January 26, 2011.

Holbrook, J.B. and Frodeman, R. 2011. “Peer review and the ex ante assessment of societal impacts.” Research Evaluation, 20(3), September 2011, pp. 239–246. DOI: 10.3152/095820211X12941371876788. <http://rev.oxfordjournals.org/content/20/3/239.full.pdf+html>, last accessed January 26, 2012.

Holbrook, J.B.; Hrotic, S. (submitted, under review) Blue skies, impacts, and peer review: Results of a survey, Minerva.