Posted by s.hettrick on 21 January 2015 - 11:00am

By Lachlan Coin, Academic Karma and Associate Professor, University of Queensland.

In 2010, Jeremy Fox and Owen Petchey proposed an innovative idea – fix peer review by introducing a peer review currency, which they called PubCreds[1]. Fox and Petchey noted that peer review suffers from a tragedy of the commons , in which "individuals have every incentive to exploit the reviewer commons by submitting manuscripts, but little or no incentive to contribute reviews. The result is a system increasingly dominated by cheats (individuals who submit papers without doing proportionate reviewing), with increasingly random and potentially biased results as more and more manuscripts are rejected without external review." Their solution was to privatise the commons by introducing a currency which is earned by reviewing and spent by getting reviewed.

Symptoms of the tragedy of commons in peer review

One of the main symptoms is slowing down communication of science. Fox and Petchey describe other symptoms, including an increasing tendency for journals to peer review only a small fraction of papers received, resulting in greater randomness in what eventually gets published. Another symptom is editors inviting many more reviewers than necessary in order to secure the minimum number necessary (anecdotally ~5x as many).

Other solutions to tragedy of commons

Garrett Hardin, when he identified the tragedy of the commons in 1968 [2], concluded that appeals to altruism do not work, and that the only solution is "mutual coercion, mutually agreed upon". This conclusion seems to fit the status quo in peer review, where most reviewing is done on the basis of friendly editorial coercion.

It's worth commenting very briefly on post-publication peer review. Post-publication peer review addresses one of the main symptoms – slowing down scientific communication – by publishing the manuscript before it is reviewed. By itself, this does not address the underlying tragedy. Open peer review – often a feature of post-publication peer review – does address the tragedy by providing a platform for the reviewer to generate a citable output demonstrating their expertise. However, open peer review also carries a (real or perceived) associated cost of potential animosity from the author, which for many reviewers outweighs the incentive.

PubCreds – a peer review currency

Petchey and Fox suggest solving the tragedy by introducing a peer review currency (PubCreds) . This would require academics to accept enough review assignments to support the rate at which they are submitting papers. Petchey and Fox identified lots of potential issues to think through: what about early career researchers who haven’t had the chance to accumulate PubCreds, what about multi-author papers, what about re-reviews, is an overdraft allowed, are account balances publicly viewable?

The big unresolved question remains how do we establish such a currency? This is not trivial – a currency has no value until you can use it to buy something. Petchey and Fox suggest a top-down approach, i.e. publishers mandate a currency. But publishers, with 30-40% profit margins, have very little incentive to introduce a peer-review currency which would unlikely yield any financial gain but would incur significant costs.

Academic Karma as an implementation of PubCreds

Academic Karma is essentially a community driven implementation of PubCreds. We call the peer review currency karma, reflecting the underlying philosophy that if you carry out good quality, timely peer review for others you can expect they will do the same for you. In order to make the implementation work, we built from a reviewing platform which integrates with the reviewing procedure at almost any journal. As a reviewer you can submit a review to any journal through the system to earn karma. As an editor, you can invite reviewers and co-ordinate the process using Academic Karma.

Ongoing challenges

The biggest challenge Academic Karma faces is automated peer review systems at the journals. Currently the reviews generated using Academic Karma are transmitted via email. This works fine for journals where the email actually goes to the handling editor (which tends to be the higher impact journals), but this is not how high-throughput peer review systems work. These systems work by minimising the interaction between editor and reviewer (replacing genuine correspondence with automated emails) to the extent that an email from the reviewer to editor can jam the process.

We are making progress in better integrating Academic Karma better into publishers' processes. Currently, we make sure that the review forms at Academic Karma match the journal review forms,; and that all correspondence is copied to the editorial office as well as the handling editor. To date we have facilitated 39 reviews at Academic Karma for journals from a range of publishers including Gigascience, Biomed Central, PloS and Nature, but we hope to expand this in 2015!

References

1.Jeremy Fox and Owen L. Petchey 2010. Pubcreds: Fixing the Peer Review Process by “Privatizing” the Reviewer Commons. Bulletin of the Ecological Society of America 91:325–333. http://dx.doi.org/10.1890/0012-9623-91.3.325

2. Hardin, Garrett. “The tragedy of the commons.” science 162.3859 (1968): 1243-1248. http://dx.doi.org/10.1126/science.162.3859.1243