How new technology is being used to monitor statistics

New technologies are being developed to monitor published statistics. One of the newest and also most controversial is a software program that scans through published papers and finds, reports, and corrects inconsistencies. StatCheck, developed by Michèle Nuijten and her colleagues in Tilburg University, can scan an article and identify the type of test, degrees of freedom, test statistic, and P value. It can then find inconsistencies and report it to the user. I believe that this is creating a bright future for statistical monitoring. The first time StatCheck was used on a large conglomerate of published papers, it went through 30,717 papers to identify 16,695 that tested hypotheses using statistics and in half of those, it found at least one potential error. However, StatCheck is not perfect and can make mistakes. The former president of the Association for Psychological Science addressed the issues of StatCheck-like programs and what they are capable of if they make mistakes. He stated that uncurated and improper StatCheck results can damage the reputation of the authors and in some cases, may be seen as harassment. Another problem with StatCheck is that it cannot determine the validity of a paper. Even though there was a mistake, this could be due to a typo or a miscalculation that does not change the interpretation of the result. Thus, StatCheck results should be curated or investigated by individuals. As technology advances, more advanced statistical monitoring programs will be developed. This will only help improve the quality and reproducibility of results. In addition, tighter statistical regulations may help restore faith in those who have lost trust in scientific studies.

PubPeer, a user oriented discussion board, is a website allowing academics to engage in post-publication peer review. According to Retraction Watch, a system in place to report retracted papers, PubPeer has been a great ally in the fight against bad scientific paper publications. StatCheck analysis can now be viewed on PubPeer which has increased the number of papers to be wary of. However, there is a concern that this has only decreased the signal-to-noise ratio. Up until now the discussions on PubPeer were impactful at identifying critical flaws in the scrutinized article. Now, StatCheck is flagging papers with P-values rounded off to the wrong digit. The Red outlined image is a submission on PubPeer before StatCheck. The Unregistered User uncovered western blot images that were copy and pasted, a clear ethical infringement. The blue outlined image is from a StatCheck which points out a round off error. Instead of a p-value of 0.406, it should have been 0.40289. This doesn’t change the results of the hypothesis and the paper. (Images are located below)

Just to be clear, StatCheck is a great program that can be useful to tease out small mistakes. It is probably best used to do last minute checks before the authors submit their manuscripts. But, we must consider if this tool is causing more harm than good when people want to meta-analyze previous results. And on top of that, Having StatCheck score of 0 mistakes (which is also reported on PubPeer) can be misleading by making people think that the statistics are done correctly. Even though StatCheck agrees that their analyses should be curated, many people won’t do that unless they are the authors of the paper. Of all the responses that I have seen to StatCheck, all of them were by the authors of the paper. And finally, the biggest problem with StatCheck is that it can influence people to lose their faith in scientific research. Saying that 50% of the papers published have statistical errors can be quite misleading. The last thing that we need is to convince people that scientific research is inefficient and unreliable. Sure, there are bad papers published and improvements can be made, but looking at the big picture, science works. It is what provides us with knowledge about this world. Without the public’s support and trust in scientific research I would not be able to share this with everyone. In conclusion, StatCheck should be on your checklist to be checked-out, however it may not be the best tool for fact-checking.