Incentive to Care

Scientific discovery and technological innovation can do and have managed extraordinary feats. However, today we hear so much questioning the reliability of the findings, and countless resources have been essentially wasted funding projects that never reach fruition. When we examine the system of scientific discovery and publication on paper, we find that it is a rigorous process that requires careful planning and execution of experiments meant to answer questions. The same question must be answered from multiple angles, proven and re-proven with each proof repeated to ensure that the manuscript sent to the reviewers is the best work the lab can offer. Multiple reviewers must then scrutinize the results and methods and send feedback often involving the original authors to run more experiments to cover any holes that might exist in the work. Finally, after publication, the article in question offers just one small answer to a problem still layered in questions, and it is the responsibility of other researchers to retest these data as they try to find their own answer to the problem.

Why then, with so many checks and balances, does this system seem to fail? In the article from The Economist “Trouble at the lab”, the author explores some of the specific issues that lead to the above problems with one of the major problems being the lack of incentive for researchers to engage in proper scientific practice. The culture of science, especially in the academic setting, follows a mantra of “publish or perish”, and journals incentivize positive and novel findings over replications of experiments or negative findings. These positive findings are much more likely to have a lower statistical power than the negative results that are found, meaning that more bias is published and fewer useful results. Additionally, other researches spend countless hours and dollars trying similar kinds of experiments not knowing that those methods have already been tried. But what researcher can afford to try to publish all their negative results or try every replication that’s in the relevant literature?

Looking at Dan Ariely’s “The Honest Truth about Dishonesty”, we can see that the human tendency to look after one’s own interest is phenomenon that is as omnipresent as it is complex. Applying some of the experimental conditions to those of the everyday conditions that many scientists face, I cannot blame any one scientist for behavior. In Ariely’s experiments, when the participants see another test taker (the actor) who very obviously cheated on the short math exam and easily profited from it, the incidence of cheating rose drastically. People compete with each other, not with integrity, for survival, and the same applies to a scientist. He might know he needs to replicate an experiment, but there’s only so many lab hours and reagents, and the draft to be sent out needs that last final spark to push it through as opposed to another replicant of a previous Western Blot. In a system where we feel like we’re being wronged, the laboratory lifestyle being very easy to imagine as one of those systems, it’s much more conceivable to justify self-promoting behavior because it’s the only way to compete with one’s colleagues who are engaging in the same practices.

However, this does not need to be the end-all for this story. More and more today, there are resources and entities seeking to remedy these problems we find in the scientific community by incentivizing behavior such as publishing methods, data, and negative results. A fellow blogger, Katherine Bricker, references in her piece the journal Cell’s mandate for investigators to list their exact methods and reagents in their “Star Methods” tab. In Ariely’s work, he found that when he asked students to recite the Ten Commandments before taking the exam, the incidence of cheating dropped to 0% astoundingly. Taking the time to remind investigators and scientists of their obligations to truthful and rigorous scientific practice and actually offering incentive for them to do so, we can change this tendency and start using our time and money more effectively and lay the foundation for stable and meaningful science in the future.

https://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

http://www.cell.com/star-methods

https://www.youtube.com/watch?v=G2RKQkAoY3k

Sharing is caring? Open source data as a solution to the reproducibility crisis

As many others on this blog have already discussed, the reproducibility crisis is a serious concern shared by many scientists. While some blame the current culture of scientific achievement and others blame a widespread misapplication of statistics, the exact reasons behind this crisis are difficult to determine. In all of these discussions on scientific reproducibility, the question still remains: how do we fix it?

One proposed solution is sharing data. As Jeff Leek discusses in his article, there is much debate and fear about the idea of sharing dating for increased transparency and discovery. For many years, scientific findings have been shared in journals, where researchers present the results and interpretations of their studies via descriptions and figures. This method has been the fundamental means of scientific progress, allowing scientists to build discoveries off of the foundation of others. However, it is increasingly debated whether papers are enough – in an increasingly connected world, should scientists also share their raw data, allowing others to truly dive into the analyses performed as well as search for new findings of their own. While open-source data could open up a new world of discovery, there are also potential risks: data-sharers could lose an advantage in their field if others publish findings before them and without credit and data-analyzers could potentially misinterpret or improperly use the dataset without proper training. There are both pitfalls and advantages to data sharing, but as the science community begins to acknowledge and address the reproducibility crisis, open source data is a very viable solution.

What does open-source data look like in practice? In the field of neuroscience, there are several organizations and research groups pioneering data sharing. One such group, Neurodata Without Borders, attempts to address the logistical problems of sharing data. One obstacle to open-source data is that different research groups use very specialized techniques and store data in various distinct ways that can be difficult for a potential data analyst to understand. The Neurodata Without Borders pilot project attempts “to develop a unified, extensible, open-source data format for cellular-based neurophysiology data.” With a unified database, this organization aims to make data-sharing accessible and practical for scientists across the globe. In another pioneering effort to facilitate data sharing, a group of neuroscience laboratories across the world recently came together to form the “International Brain Lab.” This lab is a giant collaboration and project of reproducibility, where laboratories in various locations will use the same tasks and protocols to develop a standard model of neural processing. The International Brain Lab’s “standard protocol attempts to address all possible sources of variability…. from the mice’s diets to the timing and quantity of light they are exposed to each day and the type of bedding they sleep on. Every experiment will be replicated in at least one separate lab, using identical protocols, before its results and data are made public.” With solutions such as these, perhaps the trend of irreproducibility in science will be replaced with a more positive trend of collaboration and unity in scientific discovery.

Response to “Science has lost its way, at a big cost to humanity”

A few years ago, the Los Angeles Times published an article titled “Science has lost its way, at a big cost to humanity.” In the article, the author claims that billions of dollars in funds are at risk of being lost due to dishonesty in science. The author cites the studies published by Amgen and a group at Bayer Healthcare which reported that most of the findings of the papers which they were basing their biotech and pharmaceutical research could not be reproduced. The falsehoods produced, the author suggests, are the result of scientists wanted to produce exciting data and the current peer review process.

               Being that we, the scientists, are the targets of this accusation, it is easy to dismiss these claims as the concerns of a layman. We understand that falsehood is inherent in a competitive publish-or-die environment but accept it as a necessary evil for the gears of scientific progress to turn. With results comes funding and with funding comes more results. The results which are dishonest or biased may persist for some time but eventually those studies coming behind those results will not be able to stand on their own and will call the validity of the first study into question. And while the peer-review process is not perfect in ridding dishonest studies from science, it is still valuable in deterring a lot of it.

               However, I feel that concerns of a layman are sometimes telling of problems to which those within a field are blind, apathetic, or complicit. The competitive publishing environment which is the norm for science worldwide has serious issues which do hinder progress, ultimately. On top of that, many scientists do not question this system or consider any alterations to the system which may improve research for everyone.

               Despite this, however, I feel the author does overlook a few things. First, the author seems to cast bad light on scientists for not following up most studies as these studies by the large biotech and pharmaceutical companies revealed so much about the reproducibility of these other major studies. Most scientists do not work for multi-billion dollar companies which can afford to devote time and resources to checking other peoples studies. The world of science is mostly a web of independent researchers. Additionally, the studies they were checking were cancer and blood research which often require study of live animals, studies which can take enormous amounts of time and labor for a single lab to complete. To his credit though, the author did include a quote in his article which stated that research seeking to just check another lab’s work would likely not get funded. Second, the author seems to imply that scientists think that peer review is sufficient to promote honest research which I feel is not the belief of most scientists. Reproduction of results is really one of the best indicators of good science, and I do not believe most scientists ignore it in favor of belief in the infallibility of peer review. As it stands, peer review is one of the most efficient ways to check others’ work before it is presented to the larger community. However, I do believe that some sort of amendment to the peer review process is called for to reduce bias in choosing what does and does not get published (without hindering the speed of publication dramatically).

Overall, I feel this article, while having some misunderstandings of the field, is a good reminder to scientists that our methods can always be improved upon and that there are people outside of our research bubble which do care about and are affected by the work we do.

Interdisciplinary Analysis as a Possible Method for Bias Reduction and Reproducible Science

It’s time science took a step back and looked at its methodology from other perspectives. As mentioned from other blog posts and articles, it is clear that the scientific culture favors positive results, paradigm-shifting headlines, and subpar reproducibility standards. These negative aspects are often discussed separately, but perhaps would be best considered together as part of the culture of academic scientific discovery. Then we can recognize the inherent difficulty in changing any one of those issues. Changing culture is not easy and involves the breakdown of pride.

I suggest that science stop thinking its problems are unique and look at how other fields deal with inherent human characteristics to make objectively sound building blocks. Compare this to the building of skyscrapers. There is unavoidable room for human error. However, our skyscrapers do not collapse frequently enough for us to be afraid of them. In The Checklist Manifesto by Atul Gwande, this example is shows how using construction workers use checklists to prevent human errors. He applies this to the operating room, a place so technical a general checklist would not be expected to work. Yet, when a checklist is put in place to remind operating teams of simple tasks such as administration of preoperative antibiotics, patient complications declined significantly. The checklist changed the culture of the operating room to allow the nurses to feel comfortable stopping physicians from proceeding with the surgery if they did not adhere to the checklist.  This intern allows for more reproducible patient outcomes. 

In the problem at hand, reproducibility can be mitigated through publication “checklists” for critical information regarding reagents and methodology. This clearly does not take out human error, just as it doesn’t remove human error from the performance of a heart transplant, but it mitigates major human biases and increases predictable outcomes.

The book also refers to aviation for how to make a good checklist- including “is each item not adequately checked by other mechanisms”. Our current “checklists” for tenure-track faculty may encourage bias towards certain criteria. For example, high impact journal publications may be regarded more highly than well-documented and thorough research projects producing negative data. Tenure checklist modifications may include criteria for a complete evaluation of select publications rather than simply journal titles.

Atul Gwande’s work shows how looking to other fields can reveal novel approaches to seemingly complex human errors. If we always simply shied away from changing culture, where would we be today? Well, we’d have a lot more unnecessary postoperative infections, to say the least.

Bias and Reproduciblity

An essential piece of the scientific method is forming a hypothesis, an educated guess regarding the outcome of an experiment that you have not yet performed. In doing so, we create bias before we even pick up a pipette – we want our results to be statistically significant so that we can reject the null. So that our experiment means something - so that we can publish (preferably in a high impact journal). To graduate, to secure funding, to make an impact... As scientists, we don’t just want to publish - we need to publish for the sake of our careers. But as scientists, we also have a responsibility to ensure that what we are publishing is true, reproducible, and free from bias. So how do we merge those two ideas? How do we ensure that we are performing high-quality research while also meeting the pressure to publish (preferably in a high impact journal)?

Science is not easy, and sometimes science just doesn’t work. Not every experiment goes as we expect it to. Sometimes our results don’t make sense – or worse contradict the story that we are trying to tell. Then when our experiments do go as we “want” them to and we are able to publish, we face the problem of reproducibility.

In his piece about the reliability of scientific research, Jeremy Berg writes of a study performed by the pharmaceutical industry in which only 10-25% of the key findings of published preclinical cancer research could be reproduced by independent scientists. While those results may be shocking to someone outside of science, as someone that has tried and failed to replicate a published method (more than once) I’m not surprised.

I think recognizing that these problems with both bias and reproducibility exist is important, and we as scientists need to find a solution to overcoming it. I’m not sure what that solution is, but in 2016, Cell implimented their STAR★METHODS requiring all publications to provide a detailed account of their methods including the reagents that were used and I think that this is a good step in the right direction.