Mistakes, I’ve made a few, but then again, not too few to mention

Statistics can be hard

Here’s a great quote about working with statistics from the fantastic statistician David Spiegelhalter: “I am often asked why people tend to find probability a difficult and unintuitive idea, and I reply that, after forty years researching and teaching in this area, I have finally concluded that it is because probability really is a difficult and unintuitive idea” (Spiegelhalter 2019).

Someone who attended my one-day statistics refresher course expressed disappointment in their evaluation that after a day’s training they did not now feel like an expert. Perhaps if I’d given a course in Microsoft Paint I could have handed out certificates that rightly said “certified expert”, but statistics is not something you can master in a day.

Mistakes

One consequence of statistics being hard is that mistakes are made. Far too often researchers use the wrong statistical methods, or use shoddy data, or misinterpret the statistics.

And I’ve made mistakes too. Just last week I noticed a mistake in my R code that substantively changed the results of a survey analysis. Luckily I spotted this before we sent the paper for review, but it was a near miss.

I made a big mistake in one of my first ever publications by using the wrong intra-cluster correlation (Underwood, Barnett, and Hajioff 1998), which was kindly corrected by Sally Kerry and Martin Bland (more from him below) (Kerry and Bland 1998).

I made a mistake in my most highly cited paper (Barnett, Pols, and Dobson 2004). I found this one years later when I was moving the code from an old web site to GitHub (Barnett, Van Der Pols, and Dobson 2015). The calculations were right on the web site, but wrong in the paper. I have no idea how this happened, possibly just a transcription error as I was fiddling with the parameters in the example between the web and the paper. This is a regrettable mistake because this paper has been widely cited and read.

I made a mistake in one of my most influential papers on the time that Australian researchers spend applying for funding (Herbert et al. 2013). Here I accidentally used the number of chief investigators rather than the distribution of chief investigators. This means the total time that researchers spend applying for funding should be 604 years not 551. A fairly big difference that does cause me chagrin. My comfort is that I think the impact of the paper would have been the same with either figure, as both are shocking.

I even made a minor mistake in my first blog article, as I didn’t label what the error bars were (fixed now).

There are likely other mistakes that I’ve missed and have been published because neither my colleagues or the peer reviewers spotted them. With bigger data sets with huge numbers of rows and columns, mistakes are now more likely. I think a big part of the current reproducibility crisis is caused by errors in statistical analyses.

Martin Bland

Back in 2002 when I was an early career researcher, I saw the great statistician Martin Bland give an hour-long talk about his own statistical mistakes. This remains one of the most memorable talks I have ever attended. I found it incredibly empowering to realise that someone so talented still made mistakes. I felt like Martin had given me a licence to fail, but that also gave me a licence to try.

This might sound like I’m encouraging slap-dash practices, as if statistics were karaoke where you get praised for having the nerve to do it rather than what you sound like. In reality there are real consequences for getting statistics wrong. To keep this blog topical, I’ve seen multiple legitimate warnings on Twitter just this week about people “having a go” with data on COVID-19.

I am still advocating for lots of background reading, lots of training (in theory and practice), and lots of data and code checking. But I’m also saying that you are likely to make mistakes given that we work in such a complex area.

Another benefit to making mistakes is that I am good at spotting mistakes when I review papers, because I’ve remembered the tell-tale signs of my past mistakes. A key part of Martin’s talk was about learning from his mistakes and then avoiding them.

The final curtain

I’m now going to brazenly compare myself to Martin and hope that my admissions of mistakes will similarly empower other statisticians. I also hope that these admissions do not destroy my career; Martin has survived pretty well!

For what else could I do? I’ve travelled each and every statistical by way (one-way and two-way), and more, much more than this, I analysed it my way.

References

• ← Previous Post
• Next Post →