Last week, I blocked out two hours of protected time in my diary for “grant writing”. I’ve done this before, but the difference this time was that at the end of two hours I had a nearly finished NHMRC Ideas Grant.
Of course I used AI. I used PRISM, a new free tool from OpenAI designed for academic writing. I gave PRISM the Ideas Grant criteria, a document on what makes a good application, and a title and an aim. That’s all. The first draft only needed around twenty additional prompts to make minor improvements. I had little intellectual input.
I used an aim and title of one of my failed applications, so I was able to compare with my own purely human work. The AI writing had a clear flow, whereas I got bogged down in “what ifs”. It was easy to read, confident and potentially compelling. It did not read as obviously AI. With a bit of polishing, this could win funding.
UPHEAVAL
I’ve been an AI sceptic because I’ve trialled it in research projects and have often been disappointed. However, the models are improving at a confronting pace.
If good grant applications can be instantly written, then application numbers will increase causing success rates to fall even further. There could be a vicious spiral, with desperate researchers submitting ever more applications and trying their luck in areas where they have little track record.
More applications will make it even harder to find peer reviewers. More time-poor reviewers will use AI to do their reviews. At worst, the system could become robots talking to robots.
AI detection is unreliable and disqualification on AI text alone is unfair to non-native English speakers who use it to improve diction. Hallucinated references will be fixed and my generated application cited papers that were relevant and accurate.
WARNINGS FROM ELSEWHERE
If you think I’m being too gloomy, then look at the publication system with the recent explosion in template-written papers and “fast churn” papers that mine open data. These robotic papers get through peer review and into high-ranking journals.
There’s another relevant warning from the job market, where applicants and employers are both turning to AI, creating an “AI-slop hellscape”.
ENTIRELY NEW FUNDING SYSTEM
Peer review for funding has been criticised as “peer preview” because reviewers must predict what projects will be successful, which is much harder than assessing completed work. We could shift our limited human peer review resources to examining what scientists have done, not what they or AI promises. For example, imagine a research group completed a high-quality experiment within the promised time frame with a budget of $1 million. At completion, they would submit their publications and a report for peer review. A good review and final score would mean they would be awarded more conditional funds to spend in future funding rounds.
Importantly, a good review would be conditional on whether the research was competently completed, not if some new treatment was “statistically significant”. This would shift the incentives away from “publish or perish” and towards reproducible research.
It would also create a much-needed feedback loop, as researchers who repeatedly did not complete or publish their research would have less funds, whilst those who did what they promised would be rewarded.
IMPERFECT
There are flaws in this retrospective funding system, particularly for early career researchers and for risky research. However, all funding systems struggle to reward these groups because information is lacking. For early career researchers, we lack information on their nascent skills. For risky or blue-sky research, information on highly uncertain outcomes will never be available.
We need to consider a scenario where all funding applications have little information because they are mostly AI written. We need to find alternatives that reward human scientific endeavour, not a few hours of interaction with AI.