Combating Bias and Irreproducibility in Research

Recent reports by the news media have begun to shine light on a serious issue facing the scientific research community: bias and irreproducibility in research are very real problems that often result in wasted resources. During my time as a research technician in a highly competitive academic environment, I frequently witnessed the ‘forces’ that I believe are responsible for these problems: 1) a ‘publish-or-perish’ culture that encourages academic faculty (particularly junior faculty) to rapidly and continuously produce exciting results, sometimes at the expense of being painstaking in their research efforts; 2) scientists’ lack of substantive knowledge regarding how to use statistics to appropriately design experiments and analyze and interpret data; 3) opaque communication of experimental methodology; and 4) lack of communication of negative experimental results.

Moving forward, concrete steps can and should be taken to avoid bias and minimize irreproducibility wherever possible. While the first ‘force’ mentioned above would be impossible to change quickly because it would require a dramatic shift in the deeply ingrained culture of science, I believe the second, third and fourth ‘forces’ could be readily addressed by 1) enhancing scientists’ training in statistics, 2) raising standards for data transparency, rigor and reproducibility, and 3) encouraging more communication between scientists. Fortunately (and also likely in response to increasing public attention to bias and irreproducibility in research), leading scientific journals and grant-funding institutions have already started to push scientists toward more rigorous statistical treatments of data and greater transparency in methodology. I’m optimistic that these efforts will help to positively shape the scientific research landscape such that current problems with bias and irreproducibility will be increasingly mitigated.

For the specific purpose of increasing transparency in scientific methodology and enhancing communication between scientists, another effort that I think could benefit the scientific community would be to establish an online forum dedicated to each scientific paper. In each forum, the different scientists attempting to reproduce the results of an original scientific paper could directly communicate both with one another and with the authors of the original paper. This forum could be used as a ‘safe space’ to ask for help in trouble-shooting problems that arise, to ask for clarifications in methodology, and, most importantly, to report conflicting or negative results. This approach has already been implemented to great effect by some research groups, e.g. the Zhang group for the development of CRISPR/Cas9 gene editing systems.

In sum, bias and irreproducibility in research are finally receiving their due attention from the public as real obstacles that must be addressed and overcome by the scientific community. I’m hopeful that current and future efforts undertaken by the scientific community to address issues in bias and irreproducibility will continue to enhance the quality of published scientific research.

Planet Money Podcast

Hey everyone, after our blog posts earlier this week I found an interesting podcast regarding scientific bias and replication errors. It's about 20 minutes long and a good overview of the subject.

NPR’s planet money, episode 677: The Experiment Experiment

http://www.npr.org/sections/money/

Trust me, I'm a scientist.

I am sure, as scientists, we have all uttered this phrase at least once. There is a certain pride and comfort that comes with claiming to be part of such an elite field of seemingly noble pursuers of knowledge, who altruistically devote their time to making discoveries and finding cures that will benefit mankind for years to come. But really, can we be trusted? I have felt my confidence in this shake every time a colleague has uttered this statement one time too many. Maybe it is the imposter syndrome speaking, but can we really claim to know all of the facts, especially in a discipline that is constantly challenging the status quo and reexamining the “facts” of yesteryear? (Need I remind anyone of how many iterations of the atom we have produced and we are still trying to get it right?)

I am not saying that scientists should not be trusted, but what we would most benefit from would be a change in the context of this phrase. Do not trust me to have all of the answers, but do trust me to approach the question with an open-mind towards a multitude of testable possibilities in the hopes of eliminating a few. Do not trust me to find the silver bullet, but do trust me to have a potential treatment for a subset of the tested subjects. Do not trust me to produce positive results that always support my hypothesis, but do trust me to honestly present my findings, the good, the bad, and the unsexy. This is the kind of trustworthy scientist I want to be, honest and enthusiastic, but nonetheless not blinded by my own bias towards being right.

But if it were simply our own biases we were fighting, this problem of unreliability in science would be manageable with our current peer-review system. Unfortunately, the push for flashy results that provide anecdotes to the human races’ biggest problems (cancer, anyone?) is all too strong of a pull to keep the biasness at bay. The publish-or-perish culture of scientific journalism and the hierarchy it creates is a powerful force that can make all but the most steadfast of scientists to conform to a system that rewards the biggest ripple makers, regardless of whether they are made with nuggets of truth.

So as scientists, let’s resolve to be trustworthy. But let’s do this by accepting our disproven hypotheses as worthwhile findings, by being okay with making only incremental steps towards cures, by welcoming critique of our experiments so that our research can improve. By knowing that the unbiased representation of all of our findings will set the foundation for a new set of truth to be laid.

Written in reaction to:

1.     Horvath, J. The Replication Myth: Shedding Light on One ofScience’s Dirty Little Secrets. Scientific American Guest Blog, Dec 4,2013

2.     Trouble at the lab: Scientists like to think of science asself-correcting. To an alarming degree, it is not. The Economist, Oct 19, 2013

3.     Belluz, J. Why you can't always believe what you read inscientific journals. Vox.com, Sept 1, 2015

4.  Dan Ariely's TED talk on deception

Implications of Limited Funding on Sound Science

The two articles that I focused on were The Economist article, “Trouble at the lab” and the Scientific America article “The Replication Myth: Shedding light on One of Science’s Dirty Little Secrets”. Both articles comment on prevalence of research bias in current scientific discoveries, though the articles reach drastically different conclusions on the impact that these biases will have on public perception and future scientific endeavors. While both The Economist and Scientific America articles argue that the error is inherent in the peer-review construct, my belief coincides with that of the Scientific America article in that irreproducibility in research is not inherently bad, but it is the nature in which this irreproducibility occurred that dictates its benefit to the scientific community.

The Economist article raises some troubling trends in peer-reviewed science. The article cites that “Fiveyears ago about 60% of researchers said they would share their raw data ifasked; now just 45% do.” My idealized view of science relies on two basic premises: one that the scientific method is sound and rigorous and second, that the science is conducted in a collaborative atmosphere. Limited funding creates an atmosphere that is in direct conflict with both of these premises. It fosters an environment that favors speed and novelty. It favors those who publish first and favors those that publish results not yet seen in its respective field, regardless of the fact that it could contradict a number of more rigorously researched findings. The peer review process can do little to correct it because it too is subject to these two shortcomings. Without the adequate time needed, they cannot offer a thorough assessment of the data presented to them. I am not inherently against irreproducible data. Like the ideas presented in the Scientific America article, I believe that this data can “allow science to evolve”. It can foster debate, conversations, and fruitful discussions, and through these interaction, we can further our understanding of science. However, when data is irreproducible because of negligent, haphazard, and irresponsible conduct, it does nothing to further the conversation, but destroys public perception of science. In doing so, further limits scientific funding and in turn, limits the soundness of the science. 

Bias: Inevitability and Mitigation

As a graduate student in a microscopy lab, bias in data analysis and presentation constantly lurks in the back of my mind. Deciding how to fairly quantify and represent data that is highly qualitative is a constant struggle for our lab. In fact, the problem of research bias has troubled me since my very first hypothesis-based research project, a high school science fair project in which I struggled with selection bias, confounding variables, and my own flawed expectations that my data should match my hypothesis. These types of bias and myriad more are profiled by Pannucci and Wilkins, but simply recognizing our often inevitable biases will not be enough to minimize its damage. As Jared Horvath discusses, “In actuality, unreliable research and irreproducible data have been the status quo since the inception of modern science”; as such, we have a responsibility as scientists to confront our innate bias, the greatest threat to our credibility. As Dan Ariely says in his TED Talk, checking our expectations and intuitions should be the first step to improving our morality and our research. Is there any perfect solution beyond being aware of our bias? I don’t know. Bias is a huge and multifaceted challenge. But, I think working to improve education in this area and increasing public access/publication of negative data is a good place to start.

I would also like to share a resource not listed in the suggested readings that will be of great interest for those wanting to learn more on these topics. This past Friday, NPR’s Planet Money podcast released and episode entitled “The Experiment Experiment”. In this episode, the hosts talk with Brian Nosek, a researcher at the University of Virginia, about a massive study he led to examine the reproducibility of psychology studies. Briefly, Nosek found that only 39% of 100 psychology experiments in the top journals were able to be reproduced. Like many of the other sources in the recommended reading list, Planet Money explores some of the causes of unreproducible data, such as the pressure for scientists to “publish or perish”, and the bias of journals to only publish positive data, and the lack of publicity for negative data. If you enjoyed Ariely’s TED Talk, this podcast episode is well worth a listen as it is another engaging platform in which to explore the inevitable challenges of bias.

Different types of unreliable research call for different solutions

Our readings this week looked at multiple facets of unreliable research: Deception/cheating, unreplicability, and un-generalizability.  

 

Dan Ariely discusses cheating in the context of Wall Street, but it is easy to see how the behaviors he teases out in a laboratory setting with undergraduates could be relevant for data faking in science as well. Ariely discusses the finding that the propensity to cheat increases dramatically when one believes members of one’s “in-group” (in this case, Carnegie Melon students who see a student with a Carnegie Melon hoodie cheating). Ariely also discusses how under most circumstances, people would only cheat a little bit. He described this as a “personal fudge factor”, related to an individual’s desire to see his or herself as a good, moral person. You can easily imagine how this might translate into a research setting; the belief that your idea is correct (even if the data don’t support that) could lead one to believe that they are only cheating a little bit rather than a lot (which, to be clear, they are.)

Unreliable research can also result from from sloppy, improper use of statistics or poor data collection processes which find “significance” where there is none. An article for the Economist summarizes this problem quite well. This is in many ways the most frustrating area of unreliable research because it is the one that feels most correctable. We can disincentivize cheating, though realistically there will still be people who cheat. But rigorous and appropriate statistical analysis should be achievable for every published paper, if journals are willing to have every paper reviewed by a statistician (and realistically, every University is willing to foot the bill via increased paper submission fees).

Finally, un-generalizable research is research that was properly collected and analyzed, but is sensitive to small changes in experimental conditions. I would argue that these studies are the “features” of unreliable research, as discussed in Jared Horvath’s piece for Scientific American. While research that doesn’t generalize may be disappointing to those who thought they were studying a widespread phenomenon, this research still ultimately adds to our knowledge base.

These three types of unreliability are different in both their causes and solutions, and thus should not be lumped together. Cheating, or faking data, is completely unacceptable and should have serious professional consequences. Poor use of statistics can be corrected with education, and through structural mechanisms such as professional statisticians being employed by journals specifically to review the data analysis methodology of the paper. Un-generalizable research should be revealed through replications specifically designed to test the generalizability of the phenomenon: Using a different mouse line, a different age range, data collected at a different time of day. And unlike the previous two types of unreliable research, un-generalizable research shouldn’t reflect poorly on the researchers who conducted the experiments; instead it’s a familiar call back to the drawing board.

How to become a qualified future Scientist

After viewing the video of deception and reading articles about unreality in science, I feel more pressured of becoming a qualified future scientist than ever.  This is good as it makes me, a graduate student in biomedical science, to think deeply about what's important to carry on along our way of pursuing a scientific career.

As mentioned in Dan Ariely's video, there might be a "fudge factor" which underlines our cheating behaviour.  If people signed the honor code before the test, their cheating behaviour was greatly reduced.  It signifies the importance of awareness.  The awareness that biased research might cause very "bad impact" should be beared in our minds before we design our research, during our data collection and at pre-publication status.  "Bad impact" might be the failure of a drug, the waste of time and cost into the research, or the damage to our reputation in the field. As a student in cancer research, attempting to find a "cure" to pediatric brain cancer, one "bad impact" might be creating the "false hope" to those young patients and their parents or family.

Other than being aware of possible bad impact, it is also important for us biologists to be armed with basic concepts of statistical knowledge. In Jeremy Berg's blog and the article of "Trouble at the lab", they mentioned that scientists tend to produce false positive data and barely publish false negative data, which is more reliable from the perspective of statistics.  This may explain for the most biomedical non- reproducible experiments.  If we knows better about statistics, we would at least not overstate our conclusions considering the possibility of producing false positive result is not that low.  Though no statement is 100 percent true in science as mentioned by Jared Horvath, there must be ways for us to improve our methodology in conducting and communicating science, for example, better interpreting our data.  Nowadays, peer reviews call more attention on statistical explanation, which might attribute to more involving of statistics in biological research.

Now realizing that we are under the age of "publish or perish", we should avoid being pushed to nowhere by the tides of publication stress.  A good start to change might be the awareness of possible bad impact on our society by uncareful research and the integrating of biostatistics in biomedical science.

The statistics of statistics

The largest issue, in my opinion, with irreproducibility and bias is scientific research isn’t a lack of awareness, and it is not even the fact that these problems exist. The greatest problem is that there is a lack of understanding of why, and a lack of self-awareness that you personally are capable of bias. To my point of awareness of bias and irreproducibility that is not the issue. As obvious in the homework assignment, there have been many articles “shedding light” on the issue. It is so common that the term irreproducible data is more like a running lab joke then a real day-to-day concern. Most people blame the “perish or publish” culture, which puts a lot of pressure on publishing data as soon as possible, the vague (either on purpose or not) methods sections, and also the use of statistics. The pressure to publish will never go away. The concern with a lack of transparency in methods section is also an issue that arises mostly due to the high pressure to publish, but also because there are small things that make a large difference, and the research is not even aware of these. After my admittedly short time in science (6 years) I do believe that this issue has started to be addressed, and may be a more personal then systematic issue.  

The argument for blaming the use of statistics is a complicated one. This is because statistics can be extremely powerful and necessary, particularly as researchers move toward analyzing large data sets and “-omics” type studies. The main argument is that statistics is either used to freely or in a basic understanding, or that complicated statistics are used to make data seem more significant then truly are. However, as pointed out in Jeremy Berg’s blog, the issue is that scientists do not really understand how the statistics work. Importantly, they do not understand the bias that is inherent in the statistics, and why a significant fraction of experiments cannot be repeated exactly.

If you fully understand the statistics on how easily data could not be reproducible it is easier to swallow the thought that irreproducible data is common place, and has always been a part of scientific research. This is argued by John Horvath, where he stresses that if we accept that most of the data published is false or irreproducible, we can then strive to focus on what is true. As scientists we are taught always to question data or idea, and this does not end just because something is “statistically significant”.  However, if we as scientists can accept this and focus on the ideas being presented, and how we can use the small amount of “true” data to move science further then the irreproducibility is not as huge an issue as we once thought, as long as we are being honest with our data and eliminating personal biases. Meaning it is OK to accept that there is a small level of irreproducibility, but we cannot add to it with our personal biases, otherwise that small level will become very large.

This leads to an issue of awareness. Being aware of our ability to bring bias into our research. As Dan Ariely pointed out in his TED talk, “a lot of people cheat a little bit”. We all have what he refers to as a fudge factor. We are willing to cheat just a little bit, but in general most people (and here I’m really saying most scientists) do not make up data, we simply allow our biases to creep into our experiments both in design and analysis. Dan states that one of these is because of our social norms, we as a scientific community need to educate researcher on how to avoid experimental biases, and we need to accept that our intuition is not always correct.

How to change a field that seems doomed with bias

With the many recent articles siting bias as an extreme impediment to the purpose of science, it is difficult to not stand on top of the cafeteria table and call B.S. to the entire field. Although this seems drastic, faith in science and the scientists that perform it become more and more futile as knowledge of how bias has infiltrated research becomes known. An article published in The Economist lists several ways that biased and unrepeatable research can become published. How can we change an entire field that consists of researchers partial to find their hypotheses correct, reviewers that don’t have enough time to complete an accurate assessment of the science, or journals that rarely publish needed and essential repetition of previous results?  

The psychology of science is complex, and it seems that bias is unavoidable, especially in the “publish or perish” mentality that exists in the field of academia. Jared Horvath, in an article in Scientific American, states that bias and the lack of reproducible research is not just a recent phenomenon, but can be seen for many centuries previous and among even the most lauded scientists, including Galileo, Dalton, and Einstein. So again, how are we to avoid something that has been innate in our field for centuries?

We as scientists must acknowledge that science not properly designed, analyzed, reviewed and repeated exists and is pervasive in our field, even among our own institutions, departments, and laboratories. We also must acknowledge that this kind of science is not trustworthy and can mislead not only the scientific field but the general public to hope for cures that might not exist—wasting hope, time and funding on biased hypotheses and results. Although changing a field seems impossible, I believe it starts in one laboratory that is willing to fight for good scientific practices (click here for a list of biases common in design and analysis of research written by Drs. Pannucci and Wilkins).

If we truly want to change the field to one that is trustworthy and contains good, unbiased scientific practices, we must become our own “bias police” in which we are adamant about self-critiquing and repeating experiments from our laboratories. As we begin to transform our own laboratories, we can begin to hope for the transformation of our entire field.