RORI’s interesting report on the future of peer review noted a lack of cross-over of ideas between peer review in funding and peer review in journals. Having worked on both, I have ideas.
Lotteries for journals
Conditional lotteries are becoming popular with funders. After we were publicly made of fun of for suggesting them with the Australian NHMRC in 2015, they are now all over the world. They are popular because they reduce biases, save time, and appear to increase diversity. They may even speed up scientific discovery.
Conditional lotteries could be used at journals.
I’ve never been part of an editorial meeting, but I can imagine that after peer review, many papers fall in a difficult “grey zone” of uncertainty. The papers rated as terrible can be rejected, the papers rated as great can be published, whilst the uncertain papers could be randomised, possibly even by throwing them down the stairs.
There would be two benefits.
- The journal would have an almost perfect randomised trial of the effect of publishing in their journal. They would need to track the control papers that they didn’t publish, which are very likely to eventually be published elsewhere. Then, after a year or so of follow-up, run a bibliometric comparison of readers, downloads, citations and Altmetric scores. Assuming a positive result, the journal would be able to boast about their selection of papers.
As a longer-term outcome, the journal could examine the impact of publication on the authors’ careers. Papers in high-profile journals are often thought to be career defining.
- The journal might publish more groundbreaking papers. The papers in the grey zone might be the risky papers that sometimes end up being surprisingly impactful. When humans decide, risky papers might more often lose out. A lottery is blind to any conservativism.
Lotteries are used by funders because their funding is finite. Some might not see the need for lotteries at an online journal where there is plenty of space. But many journals want to create scarcity to develop an exclusive brand with a low acceptance rate.
Distributed peer review at journals
Another idea that could be transplanted from funding to journals is distributed peer review. In this system, anyone that applies also needs to review. If you don’t review, your application is rejected. That’s a very strong incentive to review. Distributed peer review stops the freeloaders, of which there are many. It greatly eases the administrative headache of finding reviewers. Because every reviewer wants to get the results quickly, it has sped up peer review and funding decisions. So far, there’s be no clear gaming. It might even improve the quality of peer review because every reviewer should be familiar with the funding criteria.
Distributed peer review could be trialled at journals.
As batches of papers in the same field arrive and avoid desk rejection, the corresponding author would be sent an email explaining that their paper will be peer reviewed but, in return, they need to provide two peer reviews in next two weeks. They could nominate co-authors to do the reviews and be given a list of papers to choose from.
A big concern is that the extra work would discourage authors and the journal might lose business. However, it might also become a more popular journal, because the reviews will happen quickly. There would likely be some administrative headaches when an author team did not provide the reviews in time and was kicked out. There could also be gaming, as reviewers try to kill their competition. Authors might get AI to do their reviews, although this is a problem that goes beyond distributed peer review. Anyone providing AI reviews would be kicked out, although determining that with high certainty is difficult.
As peer review is under strain and journals are struggling to find reviewers, I think this system is worth trialling. If you expect a journal to review your papers, you should be willing to review in return.
Some journals already have a semi-informal distributed peer review, as I’ve often been asked to review for a journal soon after submitting. I am more inclined to accept these requests because I feel an obligation to return the favour. I also think that I might get a more sympathetic hearing in the editorial meeting.