There’s been another call to “Bring back panels!” for the NHMRC, this time published in Science. These calls normally appear on social media on the day of major funding announcements, when the majority of people find out that they’ve been unsuccessful and vent their frustrations.
There are some scientists who, when their applications are rejected, don’t consider that it was bad luck or that there’s some flaw in their ideas. Instead, they conclude that the system must be wrong and therefore must be changed.
If panels were brought back without an increase in the funding pool, then the same thing would happen with a different flavour. Perhaps it would be “get rid of panels!” or “bring back interviews!” or “we MUST have expressions of interest!”, etc.
Science?
It’s disappointing that an article can be published in Science without a single reference to the literature. There’s a decent amount that’s been published on panels in funding. For example the review What do we know about grant peer review in the health sciences? has 49 mentions of the word “panel”. If the authors had read the literature, they would see that the evidence for panels is tepid, with only moderate differences between independent reviewers and panels. It’s not clear that these differences are worth the extra time and cost of panels.
Data from the NHMRC shows that the percentage of reviewers who reported being a highly suitable match went from 27% in the panel system to 75% to 90% in the system using independent reviewers. That’s an enormous increase and a strong reason not to use panels in my opinion.
The trial we need
What we need is a randomised trial of panels versus independent reviews. We would need to wait perhaps a decade for the evidence to accrue, but we should then know which system was more able to select the best science.
The cost of a trial would be around $750,000. I think that would be worth spending to inform a system that spends billions every year.
Sadly, this trial will likely never happen, and it won’t be blocked by the costs. The main barrier is that too many researchers would not consent to take part, making the management of any trial impossible.
Only 2 out of 3 researchers consented to take part in a randomised trial of funding where we passively monitored them. This weakened our results and we got hammered for this by journal reviewers.
I am certain that many researchers would not consent to take part in a funding trial where they might receive an inferior system. It’s ironic that we ask patients to do this all the time, but many of us are unwilling to do it ourselves.
It’s not like a drug trial, where if a patient does not consent you just try the next patient. For a trial of funding, we’d need the whole pool of applicants to consent.
The only hope is that a private funder would make consenting to a trial a condition of applying. Patients in drug trials don’t have a chance of getting the experimental drug unless they agree to take part. Similarly, The BMJ reserve the right to run experiments on papers submitted to their journals.
Without using evidence inform the design of funding systems we are left with gut feeling. But many scientists are not objective when it comes to funding. When we examined the randomised funding used by the New Zealand Health Research Council, 78% who won funding thought it was a good system, compared with only 44% of those who missed out.
For a lot of scientists, the best system is the one that funds them. I ran one of the only scientific studies on panels and I came away thinking that panels were not worth the money and also had a lot of negatives, including sinking grants, gaming scores, settling personal scores, and “off the record” conversations. If panels looked great based on the evidence, then I’d be the first to be beating the panel drum.
“Bring back panels!”, nah. “Bring back evidence!”