Bias and Reproduciblity

An essential piece of the scientific method is forming a hypothesis, an educated guess regarding the outcome of an experiment that you have not yet performed. In doing so, we create bias before we even pick up a pipette – we want our results to be statistically significant so that we can reject the null. So that our experiment means something - so that we can publish (preferably in a high impact journal). To graduate, to secure funding, to make an impact... As scientists, we don’t just want to publish - we need to publish for the sake of our careers. But as scientists, we also have a responsibility to ensure that what we are publishing is true, reproducible, and free from bias. So how do we merge those two ideas? How do we ensure that we are performing high-quality research while also meeting the pressure to publish (preferably in a high impact journal)?

Science is not easy, and sometimes science just doesn’t work. Not every experiment goes as we expect it to. Sometimes our results don’t make sense – or worse contradict the story that we are trying to tell. Then when our experiments do go as we “want” them to and we are able to publish, we face the problem of reproducibility.

In his piece about the reliability of scientific research, Jeremy Berg writes of a study performed by the pharmaceutical industry in which only 10-25% of the key findings of published preclinical cancer research could be reproduced by independent scientists. While those results may be shocking to someone outside of science, as someone that has tried and failed to replicate a published method (more than once) I’m not surprised.

I think recognizing that these problems with both bias and reproducibility exist is important, and we as scientists need to find a solution to overcoming it. I’m not sure what that solution is, but in 2016, Cell implimented their STAR★METHODS requiring all publications to provide a detailed account of their methods including the reagents that were used and I think that this is a good step in the right direction.

Scientific Bias Across Different Fields

As a scientist, it is increasingly more common to hear about papers being retracted after being published and about how journals are becoming more demanding in terms of what they require for submission. Many postdoctoral fellows and professors joke about how it is notoriously impossible to repeat experiments published in high-impact journals that were conducted at other research institutions. One explanation regarding the issue of reproducibility could be that the conditions at every institution are different, leading some researchers to obtain positive results, while others obtain negative results. However, the issue of reproducibility raises a more concerning issue: how much bias is introduced into experiments by researchers?

Science is driven by the underlying need to obtain significant results. Publications are crucial to advancing a scientific career and it is difficult to publish without having statistically significant data. Can the need for obtaining significant results cause a researcher to introduce bias into their experiments? According to Daniele Fanelli, a researcher at Stanford who studies publication bias, bias is greater in the social sciences, such as psychology, economics, and sociology, when compared to the physical sciences, such as chemistry and biology. 

Some fields, such as psychology, seem to be inherently more prone to experimenter bias. Although Fanelli suggests that the social sciences tend to have more bias than the physical sciences, that doesn’t mean that the physical sciences are bias free. In some cases, the physical sciences can introduce bias through experimental design, such as testing enough animals until significance is reached, or through the presentation of the data. No matter what field a researcher is working in, it is important to keep in mind how bias can be introduced into experiments and an effort must be made to reduce bias as much as possible.

https://www.washingtonpost.com/news/in-theory/wp/2017/03/31/how-biased-is-science-really/?utm_term=.018e6cf3b4b5