Contents

---

SRS Publication Trends Study

Appendix A. Workshop Agenda and Participants

Suggested Citation

SRS Publication Trends Study

Introduction

Several years ago, the Division of Science Resources Statistics (SRS) of the National Science Foundation (NSF) initiated a study of U.S. science and engineering (S&E) publication trends in an international context. The study was prompted by evidence that the numbers of U.S. S&E articles, after increasing for decades, had leveled off in the early 1990s while funding and the number of researchers continued to increase. The study focused on developments between 1988 and 2003 and resulted in three research reports. One report delineated trends in publication and coauthorship,[1] another analyzed the covariates of publication trends in the U.S. academic sector,[2] and the third recorded the perspectives of academic scientists and administrators on the changing research and publication environment in U.S. universities.[3]

The results of the study were presented at a workshop, hosted by SRI International on behalf of NSF titled the "SRS Publication Trends Study Workshop," at which panels of researchers discussed the study findings. Sessions were devoted to the international context for the U.S. trends, the study's implications for publication indicators and bibliometric research, and the study's possible implications for science and technology (S&T) policy.

The main observations and themes that emerged at the workshop are summarized in this document. Comments are grouped by topic, using the topical organization of the workshop agenda: comments on the SRS study, international context, implications for research and indicators, and possible policy implications. Participants commented on each of these topics throughout the workshop. Where comments on different topics were closely intertwined, as often occurred for observations about implications for either research or policy, they are placed under the most appropriate topic to minimize repetition.

 Back to Top

Comments on Study Data and Analysis

A number of points were made during the workshop discussion about the data and analysis in the SRS reports.

Diana Hicks argued that as the quality of foreign S&E improves, one could expect that high-quality foreign articles will force U.S. articles of somewhat lower quality out of the Thomson ISI database and into lower-tier journals, thus reducing both the share and the absolute number of U.S. articles in the database.

Ben Martin observed that the journals in the Thomson ISI database make up a subset of the world's publications. The database has historically had a substantial U.S. and English language bias of unknown size. This bias is now decreasing because non-U.S. and non-English-speaking scientists are improving and behaving more like scientists from the United States. As a result, they are publishing more articles in journals in the database and therefore represent increasing competition for U.S. scientists.

Irwin Feller commented that the SRS reports were carefully constructed and presented, but noted some problems in the presentation of selected data. He pointed out that percentage changes were not normalized to the base, resulting in misleadingly high changes for smaller institutions because an absolute change on top of a smaller base always appears larger than the same change on top of a larger base. He also had reservations about findings derived from comparisons of heterogeneous universities (e.g., North Dakota State University and the University of Michigan), which he regarded as too different in size, orientation to research, infrastructure, and other characteristics to permit meaningful comparisons. He further argued that (1) the narrative sections overaggregate scientific fields; (2) the report uses peak year totals as one end point in computing growth rates, a choice that accentuates decline; and (3) discussion of the relationship between a university's rate of growth of research and development (R&D) funding and publications confounds absolute and relative values. He also commented on a sample bias problem in the qualitative report Changing Environment, arguing that elite universities like those covered in the report have different norms, values, and cultures than do other universities. He said the reports would have benefited from linkage to the relevant research literatures, which contain findings that complement and support those in the SRS reports.

During the discussion, workshop participants emphasized the importance of the lack of good data on the number of S&E research faculty at universities, a limitation noted in the Academic Publishing report. Also mentioned at the workshop were a possible problem in the counting of academic R&D expenditures (i.e., collaborative grant funds are not split up between principal investigators or institutions), and a possible effect of production backlogs on the most recent Thomson ISI publication and citation counts.

Other participants observed that the U.S. research system is intended to produce outputs in addition to research—especially people. One argued that publications and education should be treated as joint products of the academic research system production function. The import of this approach is that it permits the possibility of substitution of educational outputs for research ("publication") outputs within the aggregate output of universities and their faculties. This would clearly have implications for understanding trends in publication. The concern was raised that U.S. universities may differ widely in their combination of publication and education, making them difficult to compare.

 Back to Top

International Context

The SRS study findings were placed in international context by workshop participants. Prof. Martin stated that other developed countries are now beginning to go through a period of declining shares in Thomson ISI database articles similar to that of the United States. The sources of growing S&E journal articles are the emerging economies such as China. Cong Cao presented data showing accelerating increases in the number and share of Chinese articles in the Thomson ISI database. China now produces about 18% as many publications in the database as the United States. Dr. Cao attributed this growth to the dramatic growth of R&D expenditures, increasing enrollment of doctoral students, the practice of ranking academic institutions by the number of articles in the Thomson ISI database, and incentives provided by rewarding the authors of such articles.

There is also evidence that some developed countries are experiencing a disparity between research funding and publication outputs that is similar to the U.S. pattern. Ron Freedman presented data from a Canadian study that showed low growth in the number of Canadian Thomson ISI database articles while research funding grew strongly.

It was also noted that other countries are concerned about the quality of their science as measured by citations; for example, Europe is still significantly below the United States in citations and a smaller proportion of its articles are classified as highly cited. International scholars were emphatic in their view that the citation data indicate the continuing quality and strength of U.S. S&E. They suggested that Europeans see their relative standing in the citation data as an important indicator of relative weakness. Dr. Cao also referred to China's weak showing on citations as a problem; although China ranked 5th in 2005 in terms of the number of Thomson ISI database articles, it ranked 124th between 1994 and 2004 in the number of citations received per article.

Evidence was presented that international S&T collaboration is beneficial to the scientific capability of emerging economies, as measured by the number of articles in highly cited journals. Dr. Cao, for example, noted that China has become an important player in international S&T collaboration and that it benefits from international collaboration in this way. U.S.�China collaborations are an important part of China's international collaboration. Another participant observed that the rate of collaboration with the United States is decreasing in China and other developing countries as these countries develop collaborative links to more countries.

Several participants commented on the need for internationally comparable data on research funding. Prof. Martin stated that it would be interesting to know whether U.S. funding has been growing faster or slower than overseas funding, but those data are not available. He speculated that in fields where U.S. publications are doing better, U.S. funding growth has probably been similar to that in other countries. Another participant commented that statistics show that research funding is increasing around the world, but because scientific infrastructure is not included, the statistics are actually undercounting the resources being invested, especially in developing countries that are building up their infrastructure.

Dr. Hicks presented Organisation for Economic Co-operation and Development (OECD) data on combined R&D expenditures in higher education for the United States, the European Union (EU), China, Korea, Singapore, Japan, and Israel. The data show that the percentage of R&D expenditures attributed to the United States declined from 1988 to 1997. Data on the combined numbers of researchers (full-time equivalents) in higher education in the United States, the EU, China, Korea, Singapore, and Japan likewise showed the U.S. share of the group total declining between 1993 and 1999. Dr. Hicks also reported a long decline in U.S. S&E doctoral degrees as a percentage of the combined number of degrees awarded by the United Kingdom, Germany, China, India, Japan, South Korea, and Taiwan. Dr. Hicks concluded that the growth in investment in all other countries in aggregate has been outpacing U.S. growth. Because some of those countries started from a very low base and are essentially establishing an S&T system, this is not surprising. But it does mean that the efforts of these countries have shifted the international S&T landscape and that challenges to U.S. preeminence are emerging.

There was broad agreement at the workshop that the decline in the U.S. share of Thomson ISI database articles will continue based on broad trends in global S&T development and economic development. In contrast, participants said that the decline in absolute numbers of U.S. database articles may continue, depending on its causes and the U.S. response. Dr. Hicks projected that the number of U.S. publications in the Thomson ISI database will peak in the year 2021, due to the relatively inelastic database size and the pressure from more rapid growth of publication activity in other countries. Different scenarios in her analysis yielded earlier or later peaking for the United States.

 Back to Top

Implications for Research and Indicators

The workshop participants spent considerable time discussing implications of the SRS study for research. In addition to data needs (discussed earlier), research topics included alternative models and conceptual frameworks, their implications and adequacy, and research needs and suggestions.

There seemed to be agreement that it would be useful to clarify what different frameworks exist for making sense of the data and to recognize that these frameworks have different implications for research and policy. Dr. Feller offered three alternative interpretations of the data on flattening U.S. publication output:

Dr. Feller did not suggest that one interpretation is correct or more important than the others, but instead said that each is a plausible explanation for the observed data and that sifting through these alternative explanations is important because they imply considerably different policy actions. The first framework implies that global catch-up is a juggernaut that cannot be overcome; the second implies that we could spend more to get more; and the third implies that even if we spend more, we will not get a proportional return measured in publication numbers.

Researchers at the workshop said better models are needed to understand relationships within the research, publication, and innovation system. Susan Cozzens, for example, said that current models for linking publication to economic performance are inadequate to answer the question of whether the publication trends pose a problem for the U.S. economy. Without better models, policymakers cannot take the next step to determine specific policy actions because they risk making wasteful or counterproductive choices. For example, Ron Kostoff suggested that the federal government give funded researchers strong incentives to publish existing research findings that the researchers might not otherwise have chosen to publish. However, in the absence of a model relating publications to economic and other outcomes, it cannot be determined whether or how these additional publications would be used productively by other researchers or practitioners. Dr. Kostoff plausibly argued that American researchers underpublish their findings, but the opposite argument could also be plausibly made; at this point, the models and data do not exist to resolve this issue. As one participant stated, researchers have an idea of how the science system works but not a model or good empirical tests.

Another research topic at the workshop was the value of more disaggregated studies for addressing more fully elaborated models, and the weakness of the aggregated national data for doing so. Dr. Feller argued that future work in this area would benefit from more explicit models and from data on intervening output/cost function relationships. He mentioned several trends and variables that a model might incorporate: trends in federal support as a percentage of academic R&D,[4] declining success rates for proposals submitted to funding agencies, researchers' growing need for additional "preliminary results" when seeking funding, and the impact of start-up packages on costs.

Dr. Kostoff spoke about text mining techniques that focus on specific technologies and countries. He expressed his belief, based on his studies of nanotechnology and energetic materials, that aggregated studies lose much information and may be misleading. When one looks at these critical technologies, China is second only to the United States, which is much higher than the aggregate statistics suggest.

Dr. Cozzens addressed the desirability of more detailed data by pointing out that the NSF data do not allow researchers to focus on the specific areas involved in national problem solving such as energy. She noted that national laboratory publications are dropping off even more than the average, and researchers have not mapped their contributions well enough to determine whether this is a problem.

Tony van Raan's presentation emphasized the need to normalize across different fields in citation analysis because the average number of citations per article differs greatly among them. He argued that it is also important to distinguish between top performing and lower performing research groups. Making these distinctions, his research suggested that although some groups maximize their citations by publishing in the top journals or by maximizing international collaboration, this is not true for all groups.

Suggestions for research were made in the course of the discussion. Some suggestions were for research to help develop and interpret publication and citation indicators and some were for research to understand and explain the underlying phenomena and their impacts. Key suggestions included:

Many possible causes of the observed trends were discussed. Further research is needed to assess what trends in research and publication exist and confirm or eliminate them as causes or correlates for the resulting output trend. Possible causes or correlates include:

 Back to Top

Possible Policy Implications

The question of possible policy implications was discussed from a number of perspectives, including those of science policymakers and research program managers as well as those of science policy researchers and analysts.

Dr. Cozzens listed the following questions that she thought Dr. Marburger, the President's science advisor, would ask about the leveling off of U.S. publication:

Dr. Cozzens concluded that Dr. Marburger would not view the leveling off of U.S. publications as a serious policy problem because these questions could not be addressed definitively.

Bill Valdez listed questions that would be asked by the Director of the Office of Science at the Department of Energy (DOE):

Presentations by Dr. Kostoff and Dr. Valdez made it clear that, for many agency research planning and evaluation purposes, it is necessary to look at specific fields and technologies, and that national-level publication trend data are not useful. Dr. Kostoff stressed that research costs varied across fields and that movement of funds toward more expensive but more critical fields could lead to fewer publications per dollar.

Speakers and other workshop participants expressed concern that the uncertainty about the causes and consequences of flattening publication output makes it difficult to know what policy steps to take, if any; to argue for increasing funding for research over other competing priorities; and to know whether increased research funding would help. Policymakers need better data and models to make policy decisions. The hope was expressed that recent support for a Science of Science and Innovation Policy initiative will bear fruit in stimulating more research in this area.

Participants addressed the question of what, if any, policy implications can be extracted from the SRS study. They expressed concern that study findings might be used out of context or be misinterpreted. They urged caution in drawing implications from the study results. Participants raised concern that the disparity between U.S. research inputs and publication outputs could possibly be used as a rationale for cutting funding for research. A presumption behind such cuts might be that declining research productivity (as measured by publication output) indicates that marginal, relatively unproductive projects are being funded and that these could be eliminated without significant loss. In reality, however, this line of reasoning goes well beyond the data.

Nonetheless, some points of general agreement arose in the discussion. It was generally agreed that the share trend in publications is not cause for concern. The declining U.S. share of publications is due primarily to the improvement of S&E capabilities in other countries, not to a decline in U.S. capabilities.

There seemed to be sentiment that the leveling off of U.S. publications is not entirely a function of how the Thomson ISI database is assembled, although Dr. Hicks stressed the importance of this element. There are many possible causes of the leveling off of the absolute number of U.S. publications, but lack of data prevents researchers from assessing them. There are also multiple possible explanations for where the research resource inputs have gone, some of which were listed earlier, and data are insufficient to distinguish among them.

Is there reason for concern? Participants argued that flattening of absolute numbers of U.S. Thomson ISI database articles does not mean that the U.S. science system is weak. In fact, the U.S. science system remains quite strong in both quantity and quality, as measured by bibliometrics and as perceived by U.S. and foreign scientists and engineers, and it is the envy of other countries. Is the leveling off of U.S. publication numbers a threat to the U.S. economy? Is it affecting U.S. ability to address national challenges such as energy? Neither of these important questions can be answered based on today's data and models.

Participants generally agreed that the United States faces a changing world. The nation can no longer be number one in all scientific fields and needs to adjust its "psychology" to that fact. A global environment where good S&T is being done in many places can benefit the United States if the nation focuses on getting access to the best S&T around the world, bringing it into the United States, and linking it to markets and national needs. This would mean emphasizing U.S. absorptive capacity, that is, its ability to acquire and internalize knowledge developed elsewhere and exploit it economically—an issue that workshop participants stated has not been adequately addressed in S&T indicators or policy.

As summarized by one participant, the SRS study is an important part of the "policy stew," along with other recent reports, such as the National Academy of Sciences' "Rising Above the Gathering Storm" report, and other meetings, that is increasing awareness of trends in U.S. S&T capability and getting the United States to the point where it can evaluate the situation accurately and take effective action.

 Back to Top

---