altmetrics.org

altmetrics12
ACM Web Science Conference 2012 Workshop
Evanston, IL, 21 June 2012

Judit Bar-Ilan
Judit.Bar-Ilan@biu.ac.il
Department of Information Science, Bar-Ilan University, Ramat Gan, Israel

Abstract

The first volume of the Journal of the American Society for Information Science and Technology (JASIST) was published in 2001. JASIST continues JASIS (the Journal of the American Society of Information Science (1970-2000) and the American Documentation (1950-1969). JASIST is one of the leading information and library science journals. In this work we considered all articles published by JASIST between 2001 and 2011. We collected citation data from three sources: Web of Science, Scopus and Google Scholar and compared them with readership data from Mendeley.

Data were collected in April 2012, thus the partial data retrieved for 2012 publications were excluded from the analysis. Data were collected from the Web of Science (WOS), where articles, reviews and proceedings papers were selected (editorials, book reviews, letters, biographical items and bibliographies were excluded), and a total of 1637 items were retrieved. A similar process at Scopus resulted in the retrieval of 1628 items. An extensive search and data cleansing in Google Scholar, using Publish or Perish, resulted in five additional items. Data were also collected from Microsoft Academic Search, but are not included in this report. This process resulted in the identifications of 1651 items categorized as articles and/or reviews published by JASIST between 2001 and 2011. Items labeled as “proceedings papers or conference papers” are articles published in JASIST that are based on earlier published papers in proceedings, and thus can be considered as articles as well (31 such items were identified in WOS and 23 in Scopus).

All 1651 articles were manually searched on Mendeley. There were several reasons why we chose not utilize the Mendeley API (see also Bar-Ilan, Haustein, Peters, Priem, Shema & Terliesner, 2012). In case of multiple records for the given article, only one of them is retrieved by the API, and this record is not necessarily the record with the largest number of readers (in any case we were interested in comprehensive readership counts). Title searches are “tricky”, because the Mendeley search engine does not handle well special characters like hyphens, question marks, colons, and non-standard ASCII characters like ø or é. Title searches could not be replaced by DOI searches because not all the records in Mendeley have DOIs and even if they do, sometimes the DOI is not correct. And finally there were mistakes introduced by the readers, e.g. partial or slightly incorrect titles or incorrect identification of the source (JASIST). Such “suspected” items were carefully checked and were only included when we were quite confident, that the record refers to the specific item published by JASIST (e.g. correct DOI, exact abstract or the correct preview page). In order to be as comprehensive as possible we used title searches, as phrases. In most cases the subtitles were excluded from the search phrase and special characters were avoided. When necessary the name of one of the authors (usually the first author) was added to the search in order to reduce the number of retrieved records. The search results were scanned, duplicates (called “matching titles”) were expanded and the number of readers in all matching results were summed and recorded. Although probably some misspelled or misidentified items were missed in this process, we took reasonable steps to collect comprehensive readership results from Mendeley.

All, but 46 items were located in Mendeley, i.e. Mendeley covers 97.2% of the JASIST articles published between 2001 and 2011. Bar-Ilan et al. (2012) report 82% coverage of Mendeley out of more than 1,100 articles published by scientometric researchers, and Li, Thelwall and Giustini (2012) found that in 2010, Mendeley’s coverage for articles published in 2007 in Nature and Nature was 94% and 93% respectively. It should be noted that all articles published in Nature and Science were cited at least twice according to WOS, and the most cited items were cited more than 1,000 times by then. In comparison, the most highly cited JASIST article in our dataset was cited 289 times in WOS and 398 times in Scopus, and 17.2% , 12.4% and 7.6% of the items in the dataset were reported as uncited by WOS, Scopus and GS respectively. The coverage of Mendeley is extremely impressive, especially since the records are not created through systematic indexing as in the other databases, but by the users.

Table 1 displays the ten most-read articles. The Spearman correlations between Mendeley readership counts and WOS, Scopus and Google Scholar (.458, .502 and .519 respectively) are significant and of medium strength. The significance strength is similar to what was reported in previous studies, and seems to indicate that readership counts measure something different from citations. Priem, Piwowar and Hemminger (2012) found that there is higher correlation between Mendeley readership counts and downloads of PLoS articles than between these readership counts and citation counts on WOS and Scopus. Although this is expected it may also indicate that readership counts have more to do with reading than with citation, which is an active process. This is similar to visiting a Web site vs. linking to it, as reflected in the differences between PageRank and visiting popularity counts.

We conclude that Mendeley is a major altmetric data source, however further studies are needed to understand the meaning of Mendeley readership counts.

References

Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars’ visibility on the social Web. To be presented at STI2012.

Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461-471.

Priem, J., Piwowar, H., & Hemminger, B. H., (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv:1203.4745v1

This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

•