Open Review of Scientific Literature
by Thomas Arildsen
I recently became interested in open review as an ingredient in open science. There has been a lot of talk about open access in recent years. That, in itself is a very important ingredient for example for the sake of fairness in the sense that the outcome of research that is often funded by taxpayers’ money should also be open to the public. It is also important for advancing science in general, because open access helps ensure that more scientists have access to more of the existing knowledge that they can build upon to bring our collective knowledge forward. My interest in this area was in part spurred by this very inspiring discussion initiated by Pierre Vandergheynst.
Open access and open review are both parts of an ongoing movement that I believe is going to disrupt the traditional publishing model, but more about that later. Here, I want to focus on open review.
Traditionally, we have been used to reviews of papers submitted for publication in a journal being closed and typically blind or even double-blind. The review process being closed means that for a given published paper, readers simply have to trust that reviews were done thoroughly and by enough competent reviewers to ensure that we can actually trust the contents of the paper. Luckily, I believe, we can usually trust this, but for example the recent Reinhart & Rogoff episode shows that mistakes do slip through. For reasons like this and for the sake of open science per se, I believe we need more transparency in the review process (we also need published open data but that is another story). One way to do this is to make reviews open so that we as readers can see what comments reviewers made on the paper.
Reviewing a paper and then publishing it if the reviews assessed the paper as good enough is pre-publication peer review. Publishing the reviews after publication improves transparency and and as far as I can see, PeerJ (IMO an admirable open access publisher, unfortunately not in my field) is currently practising this. You can also take the somewhat bolder step of publishing papers immediately and then conducting the review in the open afterwards (post-publication peer review). As far as I can see, F1000 Research is doing this. In my opinion, this is an even better approach as it allows public insight into papers and their reviews also for papers that are not traditionally published in the end, i.e. approved by the reviewers.
There is also the question of whether reviews should be kept blind (or double-blind) or the reviewers’ identities should be open as well. I believe there are several arguments for and against this. One argument for doing fully non-blind reviews could be that a reviewer should be able to stand by what she or he says and not “cowardly” hide behind anonymity. On the other hand, especially junior reviewers may be reluctant to disclose their honest opinion about a paper out of fear that they will end in “bad standing” with the authors. Then again, openly linking reviews to reviewers can also facilitate building reputation by conducting reviews of good quality – read more about this in Pierre’s discussion. Another contribution to this debate was made by the founders of PubPeer (more about PubPeer at the end…)
Finally, and again this fits into the bigger picture of the ongoing disruption in the scientific publishing area, the open review approach can also (and should ultimately, IMO) be taken out of the area of traditional publishers. Authors can choose to upload their papers for example to open pre-print archives (such as arXiv), their institutional repositories, or even own homepages. Reviews can then be conducted based on these papers. Ultimately, publication could then turn into a system where “publishers” collect such papers based on their reviews and “publish” the ones they find most attractive, but that is a longer story I will get back to some other day. This approach to open peer review was really what I wanted to get to today. The thing is, several places are starting to pop up who offer platforms for open review. The ones I know of so far are:
- PubPeer
- Publons
- SelectedPapers
- PaperCritic
I will try to tell you what I know so far about these:
PubPeer (pubpeer.com)
This is an open review platform where you can comment on any published paper with a DOI. It allows works with PubMed IDs or arXiv IDs. Thus, in principle it covers both already (traditionally) published and pre-print papers (at least if they are on arXiv). I guess this also means research output that is not necessarily a paper, such as data, slideshows or posters, from for example figshare which assigns DOIs to uploaded content.
PubPeer received some attention recently when several flaws in a published stem cell paper were pointed out in a comment on PubPeer.
PubPeer has been criticised for being anonymous; both its founders and reviewers commenting on the platform are kept anonymous.
You can only sign up as a user on the platform if you are a first or last author of a paper they can find. So far, I have not been able to do this, as none of my papers seem to be in the areas they focus on. Furthermore, searching for yourself as an author in their database takes forever (seems like it is not that stable), so I have not actually gotten to the bottom of whether I can actually find my own name there.
[Update: I managed to sign up now, using a DOI of one of my papers. PubPeer also made me aware that anyone can comment on the platform without actually signing up as an author.]
Publons (publons.com)
This is an open review platform somewhat similar to PubPeer. It supports papers from Nature, Science and a number of physics journals as well as arXiv. Unlike PubPeer, Publons is not shrouded in anonymity. I guess it is a matter of opinion whether you support one or the other. At least Publons does not seem to have the same problems of signing up as PubPeer has.
SelectedPapers (selectedpapers.net)
This is a somewhat different platform from the two above. While it is an open review platform like PubPeer and Publons, it goes even further in the sense that it explicitly tries not to become a “walled garden” type of service and publishes review comments outside the platform itself, ideally on the reviewers’ platform of choice. SelectedPapers is still in its early stages of development and is currently in alpha release. So far, it only supports Google+ as the platform for publishing the reviews, but they promise that more are to come. I am watching this initiative closely as I find their concept extremely interesting. A lot of interesting reading about the philosophy behind the SelectedPapers network can be found here:
- http://thinking.bioinformatics.ucla.edu/2011/07/02/open-peer-review-by-a-selected-papers-network/
- http://johncarlosbaez.wordpress.com/2013/06/07/the-selected-papers-network-part-1/
- http://johncarlosbaez.wordpress.com/2013/06/14/the-selected-papers-network-part-2/
- http://johncarlosbaez.wordpress.com/2013/07/12/the-selected-papers-network-part-3/
- http://johncarlosbaez.wordpress.com/2013/07/29/the-selected-papers-network-part-4/
PaperCritic (papercritic.com)
I have not investigated this platform in detail. It is based on Mendeley and the whole platform seems to revolve around you being a Mendeley user. That is too much “walled garden” for me, so I am not very interested in this one. Furthermore, with its strict dependence on Mendeley it seems a likely candidate to be swallowed by Elsevier if it becomes successful (like Mendeley itself).
I hope this was a useful introduction to open review and some of the tools that currently exist to facilitate this. I hope some of you out there have experience with some of these or other platforms. Which one do you prefer? Feel free to submit the poll below and elaborate in the comments section if you like.
To complement the list of journal-independent peer review platforms I would like to also mention:
http://www.libreapp.org
http://www.peerageofscience.org
http://www.peerevaluation.org/
http://www.rubriq.com
http://www.axiosreview.org
http://science-open-reviewed.com
It is clear that independent peer review is becoming a trend in scholarly communication and one that is promising to seriously challenge the business model of commercial academic publishers. Available platforms come in different flavours. What scholars, in my opinion, should be careful is not to create new dependencies on closed, non-transparent, profit-driven models. Several of the existing platforms seem quite promising in that respect.
Our own initiative LIBRE | liberating research (www.libreapp.org) is developed by an open, not-for-profit organisation that any academic can join on equal terms to help us build an independent instrument that will assist our community free scientific research from commercial interests unrelated to the true pursuit of knowledge. We call the review model we propose “author-guided” open peer review as it enables authors themselves to invite any expert colleague to formally evaluate their work.
http://www.peerevaluation.org is another personal favourite that is already functional and seems to move along the same lines.
Thanks for adding to the list. I will want to check these out as well. I actually discovered Peer Evaluation myself a couple of days ago and signed up to check it out. I like the fact that it seems quite idealistic and noncommercial and I am glad to hear the same about Libre. You are not quite operational yet, are you?
Looking at these peer review sites, I actually think they fall in at least two different categories; the (probably mostly post-publication review oriented) that seem reviewer-driven and the (probably mostly pre-publication review oriented) that seem author-driven. PubPeer and Publons for example seem reviewer-driven, focusing on reviewers seeking out papers they would like to review. Peer Evaluation and Libre (?) for example seem author-driven, focusing on authors asking for reviews of their uploaded papers. Publons may be a bit of both since, judging by their front page, they seem to also want to offer an open review service to publishers.
By the way, regarding closed and non-transparent a you mentioned, I think Selectedpapers.net really makes an effort trying to be open and acting as an overlay to (ideally) whatever platform reviewers wish to publish their comments through.
Yes, Peer Evaluation is, for the same reasons, my best choise as well. It was a nice surprise to discover how ideologically close we are with this project. LIBRE (we expect to launch this October) will offer some new features and will hopefully make things even more user-friendly and author-driven.
I like your distinction, but the borders are not always clear. In LIBRE for example we enable and encourage both pre and post-publication reviews. We we would also like to see journals using our services in parallel to their own internal peer review process to increase the number of reports based on which they will make their final decision. So, we say that peer reviews can be submitted before, during (in agreement with journal editors)and after publication. And we are indeed “author-driven” (we actually call it author-guided — see our NSAP paper for more), but scholars can also request to review papers.
I also like the idea behind selectedpapers.net. It feels we are heading towards a “revolution” in the way scientific research is evaluated and hopefully many more platforms and services will emerge offering the academic community ways to regain control over the process. Hopefully most of them will adopt a noncomercial (otherwise it is hard to avoid conflicts of interest) and transparent review model. Exciting times for our community!
Natural Selection of Academic Papers (NSAP). Perakakis, P., Taylor, M., Mazza, M., Trachana, V. (2010). Scientometrics 85. url: http://www.academia.edu/1222917/Natural_selection_of_academic_papers
[…] of the platforms I mentioned previously are in the third and fourth quadrants above and they are all review platforms decoupled from the […]
[…] me to another thought: the reviews should be openly accessible as well (an issue I have discussed before on this blog). This way, there is more information available to readers for judging papers by their […]
Thank you Thomas for good critics. I agree that walled gardens is not optimal. They are not going to make breaktroughs. Google has made systematic use of openess and actually Amazon, Apple and facebook etc. Science has to develop like an fully open facebook or google and make user information open as the main feature.
[…] have previously mentioned some platforms for open / post-publication peer review in Open Review of Scientific Literature and discussed the roles of such platforms in Third-party review platforms. I just wanted to […]