As with any complicated problem, there are many potential changes to start curing the “reproducibility epidemic” and a solution will probably utilize intervention in many places. First and foremost, there is a necessity for more statistical understanding among scientists and more peer review on statistical methodology needed. In addition, as mentioned in the Vox article, the rise of post-publication peer review services like PubPeer and Pubmed Commons allows for continued public discussion and editing related to study design and statistical rigor following a paper’s release. The rise of independent science media outlets like Retraction Watch allows the scientific community and general public to be better aware of malfeasants and plagiarism in publication, and has also brought to light the extent to which the retraction process is arbitrary and unregulated across journals. And finally, the rush to publication likely prevents the necessary completion of corroborating experiments that would limit the amount of poorly supported science that is published.
Monday, January 18, 2016
Irreproducibility: a bug or a feature?
Scientific research has an image problem. Recent identification of irreproducibility issues in biomedical and psychological research has brought to light a seeming epidemic of unreliability in academic research. However, in order to evaluate this problem, it’s important to think about how reliable science is designed to be in the first place. As odd as it sounds, irreproducibility is a necessary part of the scientific process. With any hypothesis test, even in the most strongly designed studies with potential for causative outcomes, there exists the possibility of missing a real effect or having a false hit. For historical reasons, a 5% false positive rate and 20% false negative rate have become a standard for study design in academic research, but adhering to these characteristics alone is not sufficient to limit the potential for risk in publishing. In an ASBMB blog, Jeremy Berg does a nice job of summarizing how seemingly stringent statistical conditions can still lead to an unreliable experimental finding.