A Student's Reaction to the Reproducibility Crisis

A common fear among academic scientists today is that many of the latest and greatest findings, which cost years of work and millions of mostly tax-payer dollars, are not actually true. Even as a new graduate student, I have already come across the challenge of reproducing other's work in my own research. The author of a recent Scientific American article claims that the lack of reproducibility in science is why science actually works, why progress can be made, and how science evolves. The author takes the position that if scientists worked until their research was valid (measured by the gold standard of reproducibility), then nothing would ever be published. His conclusion is that scientists should be honest with themselves, their peers, and the public about the impossibility of perfectly reproducible science, and the likelihood that untrue findings will be published with great frequency. I feel that the author of this article confuses validity of data with data that doesn't support the original hypothesis. It's true that a scientist must never allow himself to become so committed to his model that he forces his data to match it (either through outright fraud or innocent tinkering), however, a scientist should always seek to obtain and publish data, that to the best of his ability represents the truth, whether that matches the original hypothesis or not. Perhaps I am still early enough in my scientific career that my naive hopefullness has not yet been replaced by this author's cynicism, but I feel that this author's position sets the bar for scientists too low. As scientists, we must do the best we can with our current knowledge and technological capabilities, acknowledge the limitations of what we are capable of, and accept that the findings we discover may later be disproven by other scientists who are just trying to do the same thing. 

Why are negative results impossible to publish?

One of the critical pillars of scientific research is that it is reproducible and free of bias.  That is how other scientists and the public are able to validate and trust new scientific work.  More recently than ever, with the age of “publish or perish” upon us, it seems that new research is not held to the same level of scrutiny that it was decades ago.  This is largely because there are not enough qualified people to do the unpaid work of rigorously peer-reviewing papers.  Research funders and university executives have created an environment such that the number of publications is just as (if not more) important the quality of them, as this is one of the most important criterion for awarding tenure-track positions or research funding.

            Additionally, publishing negative results in scientific journals is very challenging, if not impossible as journals claim that negative data fails to draw readership.  You’d be hard pressed to find any recent peer-reviewed journal article where the entire results of the study were that negative.  While I understand that new scientific work needs to “prove” something novel, negative research results are often not included, or are spun (sometimes using immoral statistical methods) to make the data appear positive, leading to massive amounts of bias in the field. 

            What some people fail to see is that, in some cases, there is nothing wrong with publishing negative results.  There are even cases where they can greatly benefit society, such as for debunking public health myths.  Pharmaceutical research is an area where publishing negative results is incredibly important, yet these results are so often passed over, because a paper will never be selected for a journal if the sum of the research is that “these 1000 molecules are ineffective at agonizing the receptor.” In that field, having the need for new drugs to come to the market to make money for the companies can lead to additional bias, as the FDA requires drugs to be stringently tested in clinical trials.

            All in all, the publish and perish culture of today’s scientific research coupled with the inability to publish negative results is leading to significant issues with bias and reproducibility that will become detrimental if change is not enacted.