Programme Contents

Reproducibility: why you should be worried?

Because, on simple statistical grounds, the majority of published research findings are likely to be false (Ioannidis, 2005), it is hardly surprising that the many studies cannot be replicated. Poor replication is problematic in biomedical science which commonly has low statistical power (e.g. Dumas-Mallet et al., 2016). We have examples from fields ranging from psychology to genetics. We have recently documented poor replicability for transcranial magnetic stimulation (TMS) and reproduced the finding for transcranial electrical stimulation (tDCS) (e.g. Héroux et al., 2017).

It is additionally worrying that researchers surveyed volunteered that others in the field used shonky research practices (such as failing to show data from all experimental conditions; selecting ‘responders’ to a protocol, and selecting statistics to optimise results). They even admitted to these practices themselves, but at a lower rate than their perceived prevalence for others. At the same time the researchers said such practices should be reported in publications! Our audit shows they were not. Apart from exposing a personal ethical conundrum, the practices push up the number of papers with false findings.

In an attempt to improve standards, in 2011 the Journal of Physiology and British Journal of Pharmacology both published editorial advice and 5 guidelines on standards for statistics and research presentation. Our precision audit of 200 randomly selected papers from the 4 years before and after 2011 showed that editorial advice failed to change practice: low quality statistical reporting and data presentation remained (Diong et al., 2018). Clearly guidelines alone are not enough to improve standards.

Diong J, Butler AA, Gandevia SC, Héroux ME. (2018) PLoS One 13, 8. doi: 10.1371/journal.pone.0202121

Dumas-Mallet E, Button KS, Boraud T, Gonon F, Munafò MR. (2016) R Soc Open Sci 4: 160254. doi: 10.1098/rsos.160254

Héroux ME, Loo CK, Taylor JL, Gandevia SC. (2017) PLoS One 12, 4. doi: 10.1371/journal.pone.0175635

Ioannidis JP. (2005) PLoS Med 2, e124. doi: 10.1371/journal.pmed.0020124