Monday, January 18, 2016

Irreproducibility: a bug or a feature?

Scientific research has an image problem.  Recent identification of irreproducibility issues in biomedical and psychological research has brought to light a seeming epidemic of unreliability in academic research.   However, in order to evaluate this problem, it’s important to think about how reliable science is designed to be in the first place.  As odd as it sounds, irreproducibility is a necessary part of the scientific process.  With any hypothesis test, even in the most strongly designed studies with potential for causative outcomes, there exists the possibility of missing a real effect or having a false hit.  For historical reasons, a 5% false positive rate and 20% false negative rate have become a standard for study design in academic research, but adhering to these characteristics alone is not sufficient to limit the potential for risk in publishing.  In an ASBMB blog, Jeremy Berg does a nice job of summarizing how seemingly stringent statistical conditions can still lead to an unreliable experimental finding.
As with any complicated problem, there are many potential changes to start curing the “reproducibility epidemic” and a solution will probably utilize intervention in many places.  First and foremost, there is a necessity for more statistical understanding among scientists and more peer review on statistical methodology needed.  In addition, as mentioned in the Vox article, the rise of post-publication peer review services like PubPeer and Pubmed Commons allows for continued public discussion and editing related to study design and statistical rigor following a paper’s release.  The rise of independent science media outlets like Retraction Watch allows the scientific community and general public to be better aware of malfeasants and plagiarism in publication, and has also brought to light the extent to which the retraction process is arbitrary and unregulated across journals.  And finally, the rush to publication likely prevents the necessary completion of corroborating experiments that would limit the amount of poorly supported science that is published.


1 comment:

  1. I enjoyed your blog post because it highlights something that I think is a valuable observation on how science works. In my opinion, retractions are just a part of the scientific process. This stems from the fact that we typically accept a 5% false-positive rate and a 20% false negative rate. By this definition alone, we can expect that about 5% of the papers published are going to be retracted or have irreproducible results. Now, we can get angry and say that scientists falsified data and they don't know what they are doing, or we can accept that this is a part of the scientific process. Indeed, there are some clear instances of situations where individuals falsified data, but what about the instances where this is not clear? Should these papers be retracted or should they be left alone because it is unclear if the data is wrong or if there is a confounding factor leading to the irreproducibility? The latter could lead to the next break-through in a particular area if the confounding factor is identified.

    I also think that irreproducibility, and in the end retractions, is partially due to the push for shorter materials and methods sections. While writing my own manuscripts, I notice routinely that my PI "makes the methods more concise" but loses a lot of the detail that would be necessary to accurately repeat the experiment. Indeed, my PI typically does this to shorten the article to fit into guidelines from the journal, but it can be unnerving to believe that someone may not be able to reproduce your results because one is forced to cut down on methods sections.

    Overall, I think that retractions are just a part of the scientific process, and as you say finally, the push to publish is so high that we tend to not spend the time ensuring that what we are publishing is true or determining if there are other factors that lead to the same or differing results.

    ReplyDelete