As with any complicated problem, there are many potential
changes to start curing the “reproducibility epidemic” and a solution will
probably utilize intervention in many places.
First and foremost, there is a necessity for more statistical
understanding among scientists and more peer review on statistical methodology
needed. In addition, as mentioned in the
Vox article, the rise of post-publication peer review services like PubPeer and
Pubmed Commons allows for continued public discussion and editing related to
study design and statistical rigor following a paper’s release. The rise of independent science media outlets
like Retraction Watch allows the scientific community and general public to be
better aware of malfeasants and plagiarism in publication, and has also brought
to light the extent to which the retraction process is arbitrary and unregulated across journals. And
finally, the rush to publication likely prevents the necessary completion of
corroborating experiments that would limit the amount of poorly supported
science that is published.
Monday, January 18, 2016
Irreproducibility: a bug or a feature?
Scientific research has an image problem. Recent identification of irreproducibility
issues in biomedical and psychological research has brought to light a seeming
epidemic of unreliability in academic research.
However, in order to evaluate this problem, it’s important
to think about how reliable science is designed to be in the first place. As odd as it sounds, irreproducibility is a
necessary part of the scientific process.
With any hypothesis test, even in the most strongly designed studies
with potential for causative outcomes, there exists the possibility of missing
a real effect or having a false hit. For
historical reasons, a 5% false positive rate and 20% false negative rate have
become a standard for study design in academic research, but adhering to these
characteristics alone is not sufficient to limit the potential for risk in
publishing. In an ASBMB blog, Jeremy
Berg does a nice job of summarizing how seemingly stringent statistical conditions
can still lead to an unreliable experimental finding.
Subscribe to:
Post Comments (Atom)
I enjoyed your blog post because it highlights something that I think is a valuable observation on how science works. In my opinion, retractions are just a part of the scientific process. This stems from the fact that we typically accept a 5% false-positive rate and a 20% false negative rate. By this definition alone, we can expect that about 5% of the papers published are going to be retracted or have irreproducible results. Now, we can get angry and say that scientists falsified data and they don't know what they are doing, or we can accept that this is a part of the scientific process. Indeed, there are some clear instances of situations where individuals falsified data, but what about the instances where this is not clear? Should these papers be retracted or should they be left alone because it is unclear if the data is wrong or if there is a confounding factor leading to the irreproducibility? The latter could lead to the next break-through in a particular area if the confounding factor is identified.
ReplyDeleteI also think that irreproducibility, and in the end retractions, is partially due to the push for shorter materials and methods sections. While writing my own manuscripts, I notice routinely that my PI "makes the methods more concise" but loses a lot of the detail that would be necessary to accurately repeat the experiment. Indeed, my PI typically does this to shorten the article to fit into guidelines from the journal, but it can be unnerving to believe that someone may not be able to reproduce your results because one is forced to cut down on methods sections.
Overall, I think that retractions are just a part of the scientific process, and as you say finally, the push to publish is so high that we tend to not spend the time ensuring that what we are publishing is true or determining if there are other factors that lead to the same or differing results.