Hidden Biases: Looking for Bias in Unexpected Places

           Assuming that we, and all of our colleagues, are striving to conduct unbiased science, I often forget to look for bias in obvious places. Sure, there’s the scientist trying to make his or her research appear more important, the prototypical bias we have come to expect, but what about the biases we can’t control?

            This article fromNPR points out that bias can be virtually impossible to avoid. To make a long story short, a researcher found that the results of his experiment had been skewed by natural hormone differences between men and women. Depending on the gender of the person administering the experiment, the results were different.

            Of course this doesn’t mean that one gender gets inherently better results than the other, but it does illustrate an important idea. Not only do we have to be proactive in controlling our own personal biases, but also we have to look for external factors that may affect or otherwise disrupt results.

            I wouldn’t have expected gender of the experimenter to affect results of an experiment. This could be the tip of the iceberg. Does the color of the experimenter’s eyes matter? What about the perfume the experimenter was wearing? Of course some external factors are inevitable, and the point of the article wasn’t to condemn scientists of either gender. The article was a simple plea that scientists report these kinds of external factors.

            To me, this requires two fold effort from scientists: First, we need guidelines on the myriad external factors that can affect our science. Second, we need to be vigilant, reporting every factor that could possibly affect results, no matter how seemingly inconsequential. On the surface, it sounds like a pain. I understand that. It isn’t easy to keep copious and detailed notes. Reporting like this will, almost certainly, be tedious. Still, if there is any chance that a factor could affect reproducibility, it should be recorded.

            More than anything, I think the real call to action is to think about hidden ways that bias could exist in our work. I think our experiments and our credibility will be all that much better for it.

The Elephant in the Room – Irreproducibility

As a scientist-in-training, it’s hard to ignore the issues of bias and reproducibility in the field. Whether I am at an ethics workshop or keeping up to date with my social media, I always seem to come across an article reminding me that scientists are living in dark times. But how much of an issue is irreproducibility? According to an article in Nature, 90% of 1,576 researchers surveyed think there is a reproducibility crisis, with 52% believing it’s a significant crisis. As for my field, a research article published in PLoS Biology by Freedman et. al. estimates that at least 50% of preclinical research is irreproducible. As someone who works with mouse models, this figure hits close to home. It turns out that differences in mouse feed and mouse microbes can actually make it difficult to replicate in vivo mouse experiments. If this itself wasn't a problem, the authors of the PLoS Biology article also estimated that this results in about $28 billion dollars spent a year on irreproducible research. In a time where funding is extremely tight, this amount is ridiculous. All of these issues are also added on top of the fact that lack of reproducibility is costing us time. Even though it is important that biomedical research be replicated, having to replicate most of the research already out there is taking time away from more pressing issues and could hinder progress. Although there are factors that can't be controlled which might lead to lack of reproducibility, the Nature article also pointed out that most of the factors that do contribute to this problem are those that we can control. It’s hard to believe that scientists have created more problems than they have solved.  

After doing more research on the topic, I believe that the problem of reproducibility and bias in science is significant and can’t be underestimated. It is difficult to accept that a lot of research in my field can’t be reproduced, but, this does not mean that all hope should be lost. Work is already underway to help tackle some of the causes of irreproducibility. I applaud Emory for implementing ethics seminars and requiring an ethics program to be completed before graduation. I also applaud scientific journals such as Nature and Science, and funding bodies such as the NIH for taking action to ameliorate the situation. Ultimately, however, I believe that these changes won’t be enough in the long run. To truly work towards ending this problem, we need to start at the most basic level – with the scientist. This is the reason why I particularly liked Jeff Leek’s article, his suggestions seemed very reasonable and not too tedious to implement. One suggestion that was particularly interesting to me is to stop publicizing our scientific results as miracle solutions. This directly feeds into Julia Belluz’s Vox article about “revolutionary” cancer drugs that are not really what they claim to be. This could be a result of our bias, which is another important issue. Scientists, much like everyone else, are not immune to having biases. While we can’t always avoid having them, recognizing them and overcoming them is already a step in the right direction. Finally, I believe we need to work together on changing the culture around publishing, which places a lot of stress on scientists and definitely contributes to irreproducibility. I am confident that with this new generation of scientists can make headway on these problems!

Irreproducibility from Dishonesty or Technical Incompetence?

While Dan Ariely is an exceptionally good speaker, I didn’t find the entire Dan Ariely video overly relevant to our discussions on bias and irreproducibility in science. The majority of the video dealt with overt and pre-meditated dishonesty in an attempt to intentionally deceive others. In reality, the instances of pre-meditated, outright fraud and data manipulation are very low in science. Rather, I think that most instances resulting in irreproducible science are a result of both poor technical execution and improper experimental “optimization” (i.e. “optimizing” the experiment to give you the results that you expect to receive). I have seen many examples of technicians and young scientists disregarding experimental results as technical failures simply because the results were not easy to interpret at supporting their expected outcome.

Secondly, the individuals doing the conscious and deceptive lying were motivated by their own short term personal gain. The experiments he and his team conducted were very specific and the outcomes represented very immediate and short term gain, and I believe these experiments represent a biased design if one is trying to apply the conclusions to irreproducible science. The expected reward of scientific fraud is not immediately realized, nor does the risk of exposure vanish immediately.

However, I did find the section regarding bankers and mortgage backed securities (MBS) particularly relevant to some of the cognitive biases that are introduced at all stages of one’s career in science. The example given regarding being rewarded for supporting MBS leading to individuals developing a bona fide belief that MBS are inherently good, specifically coupled with other cognitive biases regarding a belief in markets as self correcting, etc. The belief in a hypothesis handed down from a PI, mentor, post doc for whom a young scientist has a lot of respect can result in the development of similar biases. And subsequently, introduce many biases into experiments resulting from both a belief in the hypothesis and a belief in science in general.