Incentive to Care

Scientific discovery and technological innovation can do and have managed extraordinary feats. However, today we hear so much questioning the reliability of the findings, and countless resources have been essentially wasted funding projects that never reach fruition. When we examine the system of scientific discovery and publication on paper, we find that it is a rigorous process that requires careful planning and execution of experiments meant to answer questions. The same question must be answered from multiple angles, proven and re-proven with each proof repeated to ensure that the manuscript sent to the reviewers is the best work the lab can offer. Multiple reviewers must then scrutinize the results and methods and send feedback often involving the original authors to run more experiments to cover any holes that might exist in the work. Finally, after publication, the article in question offers just one small answer to a problem still layered in questions, and it is the responsibility of other researchers to retest these data as they try to find their own answer to the problem.

Why then, with so many checks and balances, does this system seem to fail? In the article from The Economist “Trouble at the lab”, the author explores some of the specific issues that lead to the above problems with one of the major problems being the lack of incentive for researchers to engage in proper scientific practice. The culture of science, especially in the academic setting, follows a mantra of “publish or perish”, and journals incentivize positive and novel findings over replications of experiments or negative findings. These positive findings are much more likely to have a lower statistical power than the negative results that are found, meaning that more bias is published and fewer useful results. Additionally, other researches spend countless hours and dollars trying similar kinds of experiments not knowing that those methods have already been tried. But what researcher can afford to try to publish all their negative results or try every replication that’s in the relevant literature?

Looking at Dan Ariely’s “The Honest Truth about Dishonesty”, we can see that the human tendency to look after one’s own interest is phenomenon that is as omnipresent as it is complex. Applying some of the experimental conditions to those of the everyday conditions that many scientists face, I cannot blame any one scientist for behavior. In Ariely’s experiments, when the participants see another test taker (the actor) who very obviously cheated on the short math exam and easily profited from it, the incidence of cheating rose drastically. People compete with each other, not with integrity, for survival, and the same applies to a scientist. He might know he needs to replicate an experiment, but there’s only so many lab hours and reagents, and the draft to be sent out needs that last final spark to push it through as opposed to another replicant of a previous Western Blot. In a system where we feel like we’re being wronged, the laboratory lifestyle being very easy to imagine as one of those systems, it’s much more conceivable to justify self-promoting behavior because it’s the only way to compete with one’s colleagues who are engaging in the same practices.

However, this does not need to be the end-all for this story. More and more today, there are resources and entities seeking to remedy these problems we find in the scientific community by incentivizing behavior such as publishing methods, data, and negative results. A fellow blogger, Katherine Bricker, references in her piece the journal Cell’s mandate for investigators to list their exact methods and reagents in their “Star Methods” tab. In Ariely’s work, he found that when he asked students to recite the Ten Commandments before taking the exam, the incidence of cheating dropped to 0% astoundingly. Taking the time to remind investigators and scientists of their obligations to truthful and rigorous scientific practice and actually offering incentive for them to do so, we can change this tendency and start using our time and money more effectively and lay the foundation for stable and meaningful science in the future.

https://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

http://www.cell.com/star-methods

https://www.youtube.com/watch?v=G2RKQkAoY3k

Interdisciplinary Analysis as a Possible Method for Bias Reduction and Reproducible Science

It’s time science took a step back and looked at its methodology from other perspectives. As mentioned from other blog posts and articles, it is clear that the scientific culture favors positive results, paradigm-shifting headlines, and subpar reproducibility standards. These negative aspects are often discussed separately, but perhaps would be best considered together as part of the culture of academic scientific discovery. Then we can recognize the inherent difficulty in changing any one of those issues. Changing culture is not easy and involves the breakdown of pride.

I suggest that science stop thinking its problems are unique and look at how other fields deal with inherent human characteristics to make objectively sound building blocks. Compare this to the building of skyscrapers. There is unavoidable room for human error. However, our skyscrapers do not collapse frequently enough for us to be afraid of them. In The Checklist Manifesto by Atul Gwande, this example is shows how using construction workers use checklists to prevent human errors. He applies this to the operating room, a place so technical a general checklist would not be expected to work. Yet, when a checklist is put in place to remind operating teams of simple tasks such as administration of preoperative antibiotics, patient complications declined significantly. The checklist changed the culture of the operating room to allow the nurses to feel comfortable stopping physicians from proceeding with the surgery if they did not adhere to the checklist.  This intern allows for more reproducible patient outcomes. 

In the problem at hand, reproducibility can be mitigated through publication “checklists” for critical information regarding reagents and methodology. This clearly does not take out human error, just as it doesn’t remove human error from the performance of a heart transplant, but it mitigates major human biases and increases predictable outcomes.

The book also refers to aviation for how to make a good checklist- including “is each item not adequately checked by other mechanisms”. Our current “checklists” for tenure-track faculty may encourage bias towards certain criteria. For example, high impact journal publications may be regarded more highly than well-documented and thorough research projects producing negative data. Tenure checklist modifications may include criteria for a complete evaluation of select publications rather than simply journal titles.

Atul Gwande’s work shows how looking to other fields can reveal novel approaches to seemingly complex human errors. If we always simply shied away from changing culture, where would we be today? Well, we’d have a lot more unnecessary postoperative infections, to say the least.

We Aren't Paid to Replicate

When applying for government funding for scientific research in the United States, most agencies require a section describing the novelty of the project. If it isn't very innovative or ground breaking, chances of being funded are very slim; our government prioritizes the promise of new discoveries and breakthroughs over repetition. Scientific journals also focus on novel research and rarely publish papers that describe the replication of an older study. In general, the scientific community does not care to read about negative findings. This is reinforced by popular culture, which likes to emphasize the exciting and dramatic bioscience findings rather than remain objective and guarded about new discoveries until they are verified by other research groups. 

These expectations in academic institutions also contribute to the emphasis on novel findings. In order to be awarded tenure, a junior faculty member must receive multiple large grants and be senior author on publications in respected journals. These pressures and the competition for funding mean that most scientists don't have the time or the money to further verify their results by independent verification. Scientists have little incentive, and do not face much scrutiny even if later studies call into question the original results. However, this is becoming a crisis as more and more research indicates that only a small fraction of scientific papers can be easily replicated by other laboratories.

Recently, organizations have formed that are trying to address these issues. The most notable is a program run by the Science Exchange Network called the Reproducibility Initiative.  Scientists can pay to have their studies replicated by an independent and skilled group of researchers, who are then able to publish their results (positive or negative) in a PLOS journal. While this is expensive, it may ultimately save money by shifting research away from false leads.