The return on peer review

A while ago I took a decision to only publish in open access journals. I recently received two requests to review articles for journals. Peer-review is one of the great unseen tasks performed by academics. Most of us do some, for no particular reward, but out of a sense of duty towards the overall quality of research. It is probably a community norm also, as you become enculturated in the community of your discipline, there are a number of tasks you perform to achieve, and to demonstrate, this, a number of which are allied to publishing: Writing conference papers, writing journal articles, reviewing. 

So it's something we all do, isn't really recognised and is often performed on the edges of time. It's not entirely altruistic though - it is a good way of staying in touch with your subject (like a sort of reading club), it helps with networking (though we have better ways of doing this now don't we?) and we also hope people will review our own work when the time comes. But generally it is performed for the good of the community (the Peer Review Survey 2009 states that the reason 90% reviewers gave for conducting peer review was "because they believe they are playing an active role in the community")

It's a labour that is unaccounted for. The Peer Review Survey doesn't give a cost estimate (as far as I can see), but we can do some back of the envelope calculations. It says there are 1.3 million peer-reviewed journals published every year, and the average (modal) time for review is 4 hours. Most articles are at least double-reviewed, so that gives us:

Time spent on peer review = 1,300,000 x 2 x 4 = 10.4 million hours

This doesn't take into account editor's time in compiling reviews or chasing them up, we'll just stick with the 'donated' time of academics for now. In terms of cost, we'd need an average salary, which is difficult globally. I'll take the average academic salary in the UK, which is probably a touch on the high side. The Times Higher gives this as £42,000 per annum, before tax, which equates to £20.19 per hour. So the cost with these figures is:

20.19 x 10,400,000 = £209,976,000

[Update - Alan Cann gave me this link, which puts the cost at £1.9Billion a year]

I accept this is subject to lots of objections, it may well be less, but then again, many reviewers are often eminent, and thus further up the pay scale, so it could be more. It seems a reasonable conclusion based on the data we have. So that's over £200million a year that academics are donating of their time to the peer review process. This isn't a large sum when set against things like the budget deficit, but it's not inconsiderable. And it's fine if one views it as generating public good - this is what researchers need to do in order to conduct proper research. But an alternative view is that academics (and ultimately taxpayers) are subsidising the academic publishing to the tune of £200 million a year. That's a lot of unpaid labour.

Now that efficiency and return on investment are the new drivers for research, the question should be asked whether this is the best way to 'spend' this money? I'd suggest that if we are continuing with peer review (and its efficacy is a separate argument), then the least we should expect is that the outputs of this tax-payer funded activity should be freely available to all.

And so, my small step in this was to reply to the requests for reviews stating that I have a policy of only reviewing for open access journals. I'm sure a lot of people do this as a matter of course, but it's worth logging every blow in the revolution. If we all did it....

Comments

[AJ Cann]

A few links:

Unpaid peer review is worth £1.9bn
http://www.timeshighereducation.co.uk/story.asp?sectioncode=26&storycode=402189

Cost of peer review exceeds the cost of giving every researcher a grant
http://scienceoftheinvisible.blogspot.com/2009/04/cost-of-peer-review-exceeds-cost-of.html

Peer Review - A Love Poem by Google
http://scienceoftheinvisible.blogspot.com/2010/03/peer-review-love-poem-by-google.html

[twitter.com/cdmilligan]

Your decision only to review for open access journals is completely consistent with your publishing practice, perfectly justified.

When I peer review, part of the motivation for me is to reinforce my faith in the system - I treat other researchers papers in the same way I'd like my own to be judged.

I was wondering how practical deciding to publish only in open access journals has proved - it is easy when you are the primary author, but have you compromised when someone else leads, or have you been able to argue your case successfully.

(I couldn't find your original post on publishing in open access journals: perhaps you answered this point there).

[Erik Duval]

I review quite a bit too: an additional benefit for the reviewer is early access to research results. But it often becomes a bit complicated to refer to the paper you reviewed later on, because few journals or conferences do decent review follow-up. The result is that, as a reviewer, you often don't know what other reviewers think about the paper, wether it eventually got accepted and published (possibly with a different title), etc. It seems to me that there is a lot of wasted opportunity there...

BTW, do you only review for journals where also the review process is open? I have rather mixed feelings about that, so am curious to hear about your experiences...

[Martin]

@AJ - thanks for the links, that 1.9billion figure looks high (as you said on twitter), but may take into account further costs such as the editorial time, changes by the writers in response to review, etc. I also only took the modal time for review which was 4 hours, the median (they didn't give the mean) was 6, which would bump mine up a bit.

@Colin - I couldn't find my post where I said I would only write for open journals either, which means I didn't write one (strange, I was sure I'd blogged it somewhere). To be honest, I don't write many papers now, the blog being the my main outlet so it hasn't proven to be too big an issue. I haven't really encountered probs with co-authors so far - our recent paper on digital scholarship was in an open journal, and everyone seemed happy with that, but then I tend to be working with like-minded people.

@Erik - no, I haven't gone that far. I have mixed feelings too, and while I'm happy to experiment, I don't think I should impose a best practice in that respect yet. But I do feel that open access to the content is a minimum now if people want me to provided free content.

[Neelakantan Nair]

You write about this from the reviewers perspective. I tend to see it from the student/common man's perspective. I had blogged about this problem a while earlier here:

http://technicallypossible.wordpress.com/2010/06/04/academic-papers-should-be-free/

I'm new to the whole research side of things and just getting a hang of things. In computer science the problem is probably more, I am starting to feel. Because as far as I have seen, conferences are the ones with more reputation than journals. But most papers which are available for free in some form if you search hard enough through university databases mostly :)

[leighblackall]

To extend the illustration, Reed Elseiver reported a pre tax profit in the Academic Journal devision of ~ £247m in 2008. (see comments to this post for the link)

[leighblackall]

Sorry, hypertext html didn't go through. Here it is again: http://leighblackall.blogspot.com/2009/12/academic-publishing-scam.html

[Bob L. Sturm]

This is a thought provoking post, but I think there is more to the "return on investment" than just quality control for closed-access journals. How about the way the peer review process (usually) provides thorough and free feedback to researchers? I have rarely submitted a review and recommendation for rejection that didn't result in a stronger and acceptable resubmission. And in my own submissions I have rarely received any feedback that was not helpful in making my research and its presentation more thorough, convincing, and broad. Ironically, the worst review I have received is an accept with nothing more stated; the best I have received is a reject, with an insightful explanation of why my approach is flawed (kind of like getting to peek in the back of the book at the answers to the odd numbered questions). Also, as a peer reviewer, I usually get to see how the other reviewers see the problems, and what they found that I missed, or vice versa. I feel that this is how I am paid back by performing peer review, whether or not the publishing company provides open-access or not.

[Martin]

@Neelakantan - yes, exactly my point, I wasn't really focusing on the reviewers, ultimately it is the users I am concerned with - why do all this free activity and then let it be locked away from people who can afford access?

@Leigh - that's some good invective there - you are right to angry. I hadn't really appreciated the point you make about those outside the US/UK are also subsidising these companies which are largely based in those two countries.

@Bob - that's a different point. I agree about the value of peer-review to the individual and the community (although I think there is an interesting discussion to be had about whether this is the only way to achieve these results now). The publishers don't do anything to facilitate this process (that is done by academics, for free). I would take your conclusion and look at it the other way - would any of the benefits you name not exist for an Open Access journal? In which case, why do the work that allows a large multi-national to profit and lock away content, when you could get the same benefits and have the content open to all?

[pluton]

what about no more old-fashion peer review at all? Scientists would still write papers on their preferred subjects, then upload them on an open archive (arxiv, HAL...) with access to the useful piece of code (open languages preferred, like Python). Readers would be able to reproduce the results (I have to admit that this is not possible in many experimental fields), compare with their own methodology and comment the paper on a open forum on the side. That would be continuous social peer-reviewing instead of the current process that is not really capable of filtering mistakes. That would also filter out papers for which the strategy seems interesting but cannot be reproduced.

[Martin]

@Pluton - yes, I've advocated post-review or simply just let the good stuff rise up, approach. That is a slightly different argument - I think the whole peer-review process is up for debate. But deeply embedded cultural practices change slowly (and we may decide that peer-review really is the best approach for research), and a first step seems to be at least to ensure that all publications are openly available.

[Austin Bond]

Thanks for the very interesting post Martin. I wonder though whether your estimates are high enough? I was advised recently by an academic that only about one in every ten articles she reviewed was approved for publication (whereas your figures are based on the number of published journal articles only). I support the idea of post-review ... I wonder how many good ideas have been lost by overly stringent reviewers? I remember a post I read a while ago suggesting that many of our notable educational theorists may not have passed peer review if they were submitting their articles today.

[Just another Internet Poster]

if we all did...

1. secrets for anything profitable might be witheld.

2. petitions for subsidies would favor the 1.9 side rather than the .2 side for taxation run by a comittee which would likely [historically] require administrative fees t oproduce glossy pamphlets justifying their existence, purpose, and PSA's for the potato farmer who pays their rent.

3. the quality of open-source scientific, medical, information technology, and agricultural information might improve.

4. people who are not qualified might become compensated for reviewing information that is over their heads.

[R3beccaF]

Good Xtranormal take on this issue
http://www.youtube.com/watch?v=GMIY_4t-DR0
'Your manuscript has been accepted by the journal I own. Just sign here...'