Contents

---

Executive Summary

Purpose and Scope

Findings

Conclusion

Appendix A. Universities Visited

Appendix B. Method

Appendix C. Interview Guides

Suggested Citation, Acknowledgments

Purpose and Scope

This report summarizes the views of experienced observers and practitioners in research universities about how the worlds of academic science and engineering (S&E) research and publication changed during the 15-year period between 1988 and 2003. It is part of a larger study by the National Science Foundation (NSF), Division of Science Resources Statistics (SRS) on changing patterns and trends in U.S. production of scientific articles since the late 1980s. Scientists and engineers ordinarily publish their research results in peer-reviewed journal articles, and the number of these articles is an indicator of research output, although an admittedly imperfect one. In recent years, international use of this and related indicators has become widespread as countries have sought to assess their relative performance in S&E research.

SRS's study was prompted by evidence that the growth in the number of U.S. articles in the Institute for Scientific Information (ISI) standard comprehensive database of scientific articles, which had continued for more than two decades, began to slow in the 1990s. Since then, the volume of U.S. articles has remained virtually unchanged. This flattening of U.S. output occurred even though research and development funds, research personnel, and similar research inputs continued to grow. At the same time, the ISI database indicated that growth in article counts from leading research-producing countries in Europe and Asia continued unabated. As a result, the U.S. share of worldwide S&E publications has been declining. Whether and how the sheer number of U.S. publications (including publications not included in the ISI database) or the number of publications per researcher changed during this period is more difficult to establish.

SRS plans to publish results from this study in a series of reports focused on different aspects of the issue. The core of the study is a quantitative examination of patterns and trends in article production in the 15 years from 1988 to 2003. SRS report Changing U.S. Output of Scientific Articles: 1988–2003 (NSF 07-320) describes these patterns and trends, and a forthcoming report will use quantitative analyses to explore possible explanations for these patterns and trends.

The present report, however, is entirely qualitative. It describes how knowledgeable and experienced participants in academic research and publication perceive the changing contexts in which their work occurs. The purpose is essentially exploratory—to report observations that may point to further lines of inquiry or suggest possible explanations of observed patterns and trends in scientific publication that may be examined in further research. Exploratory work such as this helps put quantitative findings into context, as well as yielding insights into possible ways to analyze existing quantitative data and possible new data that could be collected.

For the most part, this report presents a descriptive summary of what the researchers said and refrains from discussing the implications of the findings. The report's conclusion, though, notes some possible implications of these findings for efforts to make sense of the quantitative publication trend that prompted the study.

Data for this report were gathered during visits to nine research universities, where the study team met with researchers and research administrators across the spectrum of S&E disciplines. Members of the study team visited universities in four states: California, Illinois, Massachusetts, and North Carolina. The team tried to select a mix of universities—large and small, public and private, urban and rural, with and without medical schools, and with varied trends in their output of journal articles. Each of the universities visited can be considered part of the upper tier of the American academic research universe. As a group, they are not a representative sample of that tier, but their range of characteristics is in keeping with the range found among America's leading research universities. However, one reviewer who read a draft of this report opined that, judging by his visits to several of the universities studied, this group of universities appeared to be "more favorably disposed to expert judgment, interdisciplinarity, and qualitative measures than other large universities, especially public institutions." Appendix A lists the universities visited.

In almost all cases, the study team asked to talk with people who had been involved in research for 15 years or more and would therefore have a sense of how publication practices had been changing; the team also met with less experienced researchers to find out whether their perceptions of the current research environment were substantially different from those of their more experienced counterparts. The team sought people with different roles in the research enterprise—faculty members who ran individual laboratories, department chairs, heads of interdisciplinary centers, and administrators with a large measure of overall responsibility for administering either the research or education activities on campus. In most cases, interviews and meetings lasted 1.5 to 2 hours. Because the university visits averaged about 2 days, the team met with between 10 and 30 people per campus, depending on the balance between individual and group sessions and the size of the groups. Each visit included one or two interviews with high-level administrators who provided an overall perspective on changes in their university's research, education, and publication environment. Appendix B provides more detailed information about how the data were collected.

By visiting universities and talking with informants knowledgeable about academic research, SRS sought to better understand the changing circumstances that might affect the publication patterns observed in the quantitative data. In keeping with the study's exploratory purpose, interviews and meetings ranged widely, probing changes and continuities in several domains, including how research is done, how the publication process works, and what activities universities foster and value. Perceived changes in research and publication outside U.S. universities, both internationally and in the U.S. nonacademic sector, were also explored.

One issue central to the study is the changing role of peer-reviewed articles as a vehicle for disseminating scientific output. Analyses of article production do not capture other kinds of output that have long been significant in some fields—notably, books in many social sciences and psychology and products and processes in engineering. More importantly, article and citation counts do not register new forms of output associated with advances in information technology, such as databases, software programs, and contributions to electronic archives. In visiting the universities, SRS sought insight into whether the validity of article and citation counts as output indicators was changing, how article and citation data should be interpreted, and what other kinds of output might be used as indicators in the future.

Although scientists and engineers in various institutional sectors—universities, industry, government, and nonprofit organizations—produce research articles, this report focuses on the academic sector. This sector is by far the largest source of articles and tends to produce the articles that are most influential and that appear in the most influential journals. Moreover, within the academic sector, the study team chose to visit institutions at or near the forefront of the academic research system, which define research as a central component of their mission and produce large numbers of articles. The state of the research enterprise in institutions such as these is especially relevant to U.S. leadership in S&E worldwide, and developments in such institutions have ramifications in other institutions that produce and use research.

Because the goal of SRS's larger study is to understand trends in research output, the major focus of this report is change. However, the report inevitably gives some attention to enduring features of the academic landscape that help set the context for the changes observed.