Skip to content

Instantly share code, notes, and snippets.


arfon/ Secret

Last active Aug 29, 2015
What would you like to do?
Open Journal

#Peer Review as a Service: It's not about the journal

This post sets out what we think is the future of peer review and publishing.

There are lots of posts like it on the internet. You've probably read lots of them. Many of them talk about the desire for greater openness, riffing on what might happen when the gates of academic publishing are thrown open to the masses. Many talk about the flaws and foibles inherent in the most common models of peer review, in which the stamp of Scientific Approval is conveyed via a mysterious process which involves careful or not-so-careful scrutiny by other experts in the field. Still more propose new models (perhaps peer review should be public), new tweaks (maybe we should stop pretending journals are mostly consumed on paper) or new ideas (what if code was attached to a paper and made open at the time of publication?).

We've read those blog posts, been to conferences where these things are discussed and argued about them in pubs on at least two continents. Eventually, we decided to try and do something - hence this post. It needs to cover a lot of ground - what we think we're trying to do, how far we've got and what comes next, as well as a bit of context and history - but stick with us.

Start with trying to solve the simplest possible important problem. When we worked together at the Zooniverse 'Do the simplest thing that works' became something of a mantra. For scholarly publishing, the secret sauce - the essential thing - is a mechanism for review. Even open archives like have review in the sense of only letting people who are endorsed by an existing community post, but here we'll assume that we're doing something more like traditional academic publishing - reviewing something called a paper (it could actually be code, or a figure, or a paragraph but let's stick to papers as that's easier to think about). Peer review at present is something that belongs to a journal; it's a set of rules and procedures, written and enforced by an editorial board and supported by a lot of email and some fairly wonky software. (There's a reason that I end up emailing co-authors with files called Draft17_review_comments_CJL_final.tex) If you want to set up a journal, you'll spend a lot of time arguing about how to write these rules, and building (or, more likely, buying from an existing publisher) the infrastructure to support them.

What's needed is to make this process as simple and as easy as possible. This will reduce the start up cost for new and experimental publishing formats, and divorce the process of peer review from the practice of sharing, publishing and, heck, even printing the results. Along with a few friends, we've written code to support peer review as a service. It assumes that some group of people (think editors or academic community) want to make comments on a thing (think paper). All you need is a URL for the thing - we think that GitHub repositories offer lots of nice advantages but it could be a dropbox link, or an arXiv paper or anything. Comments are just a list of issues on the paper or on specific parts of it; they are opened by a reviewer and then resolved by the reviewer after dialogue with the author, authors or editors. Once (enough) issues are resolved, your paper is 'accepted' and the service will proclaim the news to anything listening.1

That's all you actually need to start your own publishing revolution. To give you an idea of how this might work, imagine building a new journal. Let's call it the Open Journal of Astrophysics.

Around the time we were first seriously thinking about this stuff, Peter Coles - astronomer, blogger and agent provocateur of the UK academic scene - blogged about his desire to do away with traditional publishing2 and produce a lightweight journal. Peter's argument - which we have a lot of sympathy for - boils down to the fact that in the astronomical world paper sharing happens via the arXiv3 which contains essentially all professional astronomers have produced in the last twenty years or so. In large swathes of the subject - cosmology, in particular - it's common practice to post on arXiv even before submitting to a journal. What if, Peter mused, we could add peer review to the process of putting a paper on arXiv? What would be lost?

Copyediting, certainly. Traditional journals place great stress on the quality control they provide beyond peer review - academics in a hurry to get to the next paper or to generally avoid the nitty gritty of publishing rely on journal editorial staff to edit copy, correct references and so on, it's true. But decent copyediting costs a fraction of the fees most journals charge (either directly to author or in the form of expensive library subscriptions), and in any case a large fraction of what's on arXiv and happily being shared are proofs from before that process happens, simply because that's what the author has to hand.

We thought Peter's notion was worth a go. Commendably, he'd offered to put his money where his mouth was and pay for the setting up of an Open Journal of Astrophysics. We offered to build it for free and got on with building a suitable interface so that the OJA (or OJoA, I suppose) could exist. The idea is that the author would submit to arXiv, then declare to the OJA that they wished their paper reviewed. OJA ingests the paper, creating along the way a GitHub repository4 and alerts editors who could assign a reviewer the task of looking at the paper. It's important to note this is traditional peer review - the goal for OJA was to dramatically reduce the cost of running a journal, not to experiment with new models. The reviewer makes comments via a web interface which are relayed to the author, who can respond and update the paper, perhaps by submitting a new version to arXiv which would automatically be read by the OJA. Once everything is resolved, the editor can review the discussion and accept or reject the paper. At this point, we'd register the paper via a DOI5 and it could be included in all the normal places you expect peer reviewed articles to appear, such as Web of Science or NASA's ADS service.

<iframe src="//" width="500" height="281" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>

OJA from Arfon Smith on Vimeo.

And that's it. A journal with a nice web interface, an archive of the back and forth between reviewer and author, and a working peer review system. Simple. Beyond the lack of copyediting, everything you could want from a journal.6

It won't surprise anyone who has worked on software that we quickly got about 80% of the job done which is what you can see in the embedded video. Life then intervened - one of us got a new job, Peter ascended to Head of the School of Mathematics and Physical Sciences at the University of Sussex - and the idea foundered. In some ways, this is vindication for those who said that we'd missed out many of the costs. We're not sure, but if our editorial board had been paid for their work (as many are) then I suspect the journal would have got off the ground. But I'm also certain that our distraction didn't have much to do with the specific model we were promoting, which we still think is worth sharing.

We're releasing all the code that you'd need to implement our plan. There's currently two codebases to take a look at:

  • The first is the software running in the screencast which implements the downloading from the arXiv, GitHub import and some basis user interfaces.
  • The second is our latest work which is far more focussed on getting the permission model really hammered out (and properly tested) so while it's far less impressive to review (there's no UI) we're spending most of our time work on this codebase and will be porting in functionality from the old application as we progress.

We hope people will build on this work (frankly, we hope someone will help Peter and maybe ourselves finish the OJA) and use it for their own purposes. Once peer review is abstracted from the journals themselves, it makes it much easier to imagine looking beyond the paper - referees could look at a figure and caption, at a short note on a topic of urgent interest7 or code or some combination and give it a stamp of approval at least as easily as they can read a paper. There's no reason you shouldn't ask people to submit their Authorea papers, or data on FigShare or whatever post-publishing service you enjoy. Peer review can continue to be the imperfect but essential gatekeeper to scholarly discourse it is today, and evolve into new forms; extending what we've done to include public peer review, or review by committee, or formal author responses, or post-publication peer review or whatever shouldn't be hard. Nothing we've done is especially clever but once you start thinking of peer review as an external service, it's hard to stop. The plan is to make this an extremely simple hosted service that anyone could use - all suggestions, comments and code very welcome - head on over to TheOJ repository and open an issue - we'd love to talk.

  1. For a model, think about the way Travis keeps an eye on code ( and provides an image showing the status of the review :

  2. There are lots of arguments about how such a move would change things, who would benefit, what the real costs would be. Before you get into those arguments, remember that here we're just trying to build TSTTW as an experiment...

  3. There's similarly a lot of argument about whether the arXiv is sustainable. It's stuck around for thirty or so years now - it predates the web! - and is archivable by scholarly libraries, so by my lights it's at least as good as a commercial publisher by that standard. The question of permanent archiving of scholarly work doesn't belong to the publishers anyway - it's an issue for librarians and archivists, who could do a lot more if their budgets weren't spent on journal subscriptions...

  4. Authors who didn't want to submit to arXiv and make their paper public could just create the repo directly; we mused about - in the long term - charging for this service as making users pay for privacy seems like a nice model to fund things and also encourage open, sharing behaviour.

  5. A digital object identifier or DOI is the index that refers permanently to an academic article, issued by the publisher. We worked out it would have cost us a few hundred dollars to become a DOI issuing publisher, and about $1 per article. There are some carefully worded requirements about the format for references but nothing that's a show stopper.

  6. There remains of course the question of how to get people to submit to this new journal when promotion and so on is tied tightly to impact factor. Leaving aside the fact that that’s an argument about any new journal (and one eventually solved by having senior people like Peter commit to publishing in it), we thought that a decent web interface would be a strong incentive to switch. Editing in the browser! No more sharing of text files with collaborators! Roll up, roll up!

  7. The recent possible detection of primordial gravitational waves by the BICEP team is a nice case in point - this was released in a preprint on the arXiv and effectively peer reviewed on social media and in more than fifty notes posted to arXiv in the month or two since. All of this activity will eventually hit journals - but long after the conversation has moved on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.