Open Review of Scientific Literature

by Thomas Arildsen

I recently became interested in open review as an ingredient in open science. There has been a lot of talk about open access in recent years. That, in itself is a very important ingredient for example for the sake of fairness in the sense that the outcome of research that is often funded by taxpayers’ money should also be open to the public. It is also important for advancing science in general, because open access helps ensure that more scientists have access to more of the existing knowledge that they can build upon to bring our collective knowledge forward. My interest in this area was in part spurred by this very inspiring discussion initiated by Pierre Vandergheynst.

Open access and open review are both parts of an ongoing movement that I believe is going to disrupt the traditional publishing model, but more about that later. Here, I want to focus on open review.

Traditionally, we have been used to reviews of papers submitted for publication in a journal being closed and typically blind or even double-blind. The review process being closed means that for a given published paper, readers simply have to trust that reviews were done thoroughly and by enough competent reviewers to ensure that we can actually trust the contents of the paper. Luckily, I believe, we can usually trust this, but for example the recent Reinhart & Rogoff episode shows that mistakes do slip through. For reasons like this and for the sake of open science per se, I believe we need more transparency in the review process (we also need published open data but that is another story). One way to do this is to make reviews open so that we as readers can see what comments reviewers made on the paper.

Reviewing a paper and then publishing it if the reviews assessed the paper as good enough is pre-publication peer review. Publishing the reviews after publication improves transparency and  and as far as I can see, PeerJ (IMO an admirable open access publisher, unfortunately not in my field) is currently practising this. You can also take the somewhat bolder step of publishing papers immediately and then conducting the review in the open afterwards (post-publication peer review). As far as I can see, F1000 Research is doing this. In my opinion, this is an even better approach as it allows public insight into papers and their reviews also for papers that are not traditionally published in the end, i.e. approved by the reviewers.

There is also the question of whether reviews should be kept blind (or double-blind) or the reviewers’ identities should be open as well. I believe there are several arguments for and against this. One argument for doing fully non-blind reviews could be that a reviewer should be able to stand by what she or he says and not “cowardly” hide behind anonymity. On the other hand, especially junior reviewers may be reluctant to disclose their honest opinion about a paper out of fear that they will end in “bad standing” with the authors. Then again, openly linking reviews to reviewers can also facilitate building reputation by conducting reviews of good quality – read more about this in Pierre’s discussion. Another contribution to this debate was made by the founders of PubPeer (more about PubPeer at the end…)

Finally, and again this fits into the bigger picture of the ongoing disruption in the scientific publishing area, the open review approach can also (and should ultimately, IMO) be taken out of the area of traditional publishers. Authors can choose to upload their papers for example to open pre-print archives (such as arXiv), their institutional repositories, or even own homepages. Reviews can then be conducted based on these papers. Ultimately, publication could then turn into a system where “publishers” collect such papers based on their reviews and “publish” the ones they find most attractive, but that is a longer story I will get back to some other day. This approach to open peer review was really what I wanted to get to today. The thing is, several places are starting to pop up who offer platforms for open review. The ones I know of so far are:

  • PubPeer
  • Publons
  • SelectedPapers
  • PaperCritic

I will try to tell you what I know so far about these:

PubPeer (pubpeer.com)
This is an open review platform where you can comment on any published paper with a DOI. It allows works with PubMed IDs or arXiv IDs. Thus, in principle it covers both already (traditionally) published and pre-print papers (at least if they are on arXiv). I guess this also means research output that is not necessarily a paper, such as data, slideshows or posters, from for example figshare which assigns DOIs to uploaded content.
PubPeer received some attention recently when several flaws in a published stem cell paper were pointed out in a comment on PubPeer.
PubPeer has been criticised for being anonymous; both its founders and reviewers commenting on the platform are kept anonymous.
You can only sign up as a user on the platform if you are a first or last author of a paper they can find. So far, I have not been able to do this, as none of my papers seem to be in the areas they focus on. Furthermore, searching for yourself as an author in their database takes forever (seems like it is not that stable), so I have not actually gotten to the bottom of whether I can actually find my own name there.
[Update: I managed to sign up now, using a DOI of one of my papers. PubPeer also made me aware that anyone can comment on the platform without actually signing up as an author.]

Publons (publons.com)
This is an open review platform somewhat similar to PubPeer. It supports papers from Nature, Science and a number of physics journals as well as arXiv. Unlike PubPeer, Publons is not shrouded in anonymity. I guess it is a matter of opinion whether you support one or the other. At least Publons does not seem to have the same problems of signing up as PubPeer has.

SelectedPapers (selectedpapers.net)
This is a somewhat different platform from the two above. While it is an open review platform like PubPeer and Publons, it goes even further in the sense that it explicitly tries not to become a “walled garden” type of service and publishes review comments outside the platform itself, ideally on the reviewers’ platform of choice. SelectedPapers is still in its early stages of development and is currently in alpha release. So far, it only supports Google+ as the platform for publishing the reviews, but they promise that more are to come. I am watching this initiative closely as I find their concept extremely interesting. A lot of interesting reading about the philosophy behind the SelectedPapers network can be found here:

  1. http://thinking.bioinformatics.ucla.edu/2011/07/02/open-peer-review-by-a-selected-papers-network/
  2. http://johncarlosbaez.wordpress.com/2013/06/07/the-selected-papers-network-part-1/
  3. http://johncarlosbaez.wordpress.com/2013/06/14/the-selected-papers-network-part-2/
  4. http://johncarlosbaez.wordpress.com/2013/07/12/the-selected-papers-network-part-3/
  5. http://johncarlosbaez.wordpress.com/2013/07/29/the-selected-papers-network-part-4/

PaperCritic (papercritic.com)
I have not investigated this platform in detail. It is based on Mendeley and the whole platform seems to revolve around you being a Mendeley user. That is too much “walled garden” for me, so I am not very interested in this one. Furthermore, with its strict dependence on Mendeley it seems a likely candidate to be swallowed by Elsevier if it becomes successful (like Mendeley itself).

I hope this was a useful introduction to open review and some of the tools that currently exist to facilitate this. I hope some of you out there have experience with some of these or other platforms. Which one do you prefer? Feel free to submit the poll below and elaborate in the comments section if you like.