Irreproducibility from Dishonesty or Technical Incompetence?

While Dan Ariely is an exceptionally good speaker, I didn’t find the entire Dan Ariely video overly relevant to our discussions on bias and irreproducibility in science. The majority of the video dealt with overt and pre-meditated dishonesty in an attempt to intentionally deceive others. In reality, the instances of pre-meditated, outright fraud and data manipulation are very low in science. Rather, I think that most instances resulting in irreproducible science are a result of both poor technical execution and improper experimental “optimization” (i.e. “optimizing” the experiment to give you the results that you expect to receive). I have seen many examples of technicians and young scientists disregarding experimental results as technical failures simply because the results were not easy to interpret at supporting their expected outcome.

Secondly, the individuals doing the conscious and deceptive lying were motivated by their own short term personal gain. The experiments he and his team conducted were very specific and the outcomes represented very immediate and short term gain, and I believe these experiments represent a biased design if one is trying to apply the conclusions to irreproducible science. The expected reward of scientific fraud is not immediately realized, nor does the risk of exposure vanish immediately.

However, I did find the section regarding bankers and mortgage backed securities (MBS) particularly relevant to some of the cognitive biases that are introduced at all stages of one’s career in science. The example given regarding being rewarded for supporting MBS leading to individuals developing a bona fide belief that MBS are inherently good, specifically coupled with other cognitive biases regarding a belief in markets as self correcting, etc. The belief in a hypothesis handed down from a PI, mentor, post doc for whom a young scientist has a lot of respect can result in the development of similar biases. And subsequently, introduce many biases into experiments resulting from both a belief in the hypothesis and a belief in science in general.

Why you always lying?

Do scientists lie? Yes everyone lies. Why do scientists lie? We, like our genes, are selfish. Does it matter that scientists lie? 

If we can trust Dan Ariely’s talks, then his research demonstrates that people in general lie and cheat marginally (and often) without feeling like they are dishonest or bad. Fair enough. We can all think of times, or at least I certainly can (‘No officer I don’t know how fast I was going’; ‘Yes gas station attendant I am over 21’), we have lied for minor personal gains.

Apparently situations where there is a conflict of interest, a distance between the lie and direct monetary outcome, and people you identify with are also lying lead to more misbehavior. The first two parameters are met by scientists. It is obviously in the interest of a scientist to perform research with interesting results that gets published, and although publishing is connected to monetary gain it isn’t a direct transaction. But many would argue that the third parameter, other scientists lying about their data, is not something that happens. Perhaps not so blatantly. Ariely goes on to discuss asking golfers if they have picked up a golf ball and moved it. No. Kicked it while looking the other direction? Of course. For scientists there is a distinct possibility for a similar scenario. Have you ever falsified data? No, that is repugnant. Have you ever used statistics you didn’t fully understand to analyze your data and make them look good? ......

The kicker is that dishonesty in science is neither new nor always problematic. Jared Horvath describes how important scientists from Galileo to Millikan produced research that is not replicable yet helped to push our collective understanding forward. Still it could be argued that falsification is occurring at a greater rate in contemporary science. In “Trouble at the Lab” John Ioannidis is described as saying that the majority of published findings are false. The author further notes that very few articles are retracted.

So is the process of science failing in our new age? No. A scientific paper should not be expected to be 100% correct. Hell one of the main lessons being beat into our bones as grad students is that we should always be ferociously looking for errors while reading papers. Just because a paper has mistakes does not mean that it contains no useful information. The presence of useful information implies the paper shouldn’t be retracted. Yes this means that to obtain useful information from a paper you need more than a lay understanding of the field, but the entire point is that we perform research on the cusp of our society’s understanding. The health of the scientific enterprise should not be measured by how many mistakes there are in published articles, but by how much progress we are making toward improving our society. 

Integrity vs Science Career

“Everyone learns from science; it all depends on how you use the knowledge.”  Quoted by Grissom from the tv show, CSI, this quote is applicable to the controversy of bias and reproducibility in science.  Science is so valuable that there is immense pressure to publish data in top-tier journals and obtain grants during a period when the percentage of grants funded is extremely low.  It’s a cycle that can quickly steamroll out of control.

The pressure to deliver ground-breaking results can lead to publications with “mixed-up images, mislabeling, faulty descriptions, and inexplicable discrepancies” as was determined to occur in July 2014 when two papers published by a Japanese group regarding the STAP phenomenon were retracted from Nature. The responsibility of the false data was so immense that it caused one of the scientists to take his own life.  While this is an extreme example, it demonstrates the repercussions of falsifying data and the importance of integrity in science.  One’s name is always tied to his or her work even after death, thus their integrity can always be viewed through published literature.

While reproducibility may be viewed as the gold standard in science, medicine shows that personalized medicine is the current trend.  In a way, this is applicable to the world of science, as every animal facility and laboratory is different.  Experiments conducted in one environment may not be accurately reproduced in another equivalent environment at a different institution.  Despite the lack of reproducibility at different institutions, I do believe it is very important to have the science reproduced at in house.  Doing a study once is not enough to maintain one’s integrity.  After all, one’s integrity will take a person further in life than a science career.

Not only does the responsibility of accurate scientific communication fall on scientists, but it also is the responsibility of the media to precisely report findings when reporting about new findings.  As pointed out in, “Half of the cancer drugsjournalists called ‘miracles’ and ‘cures’ were approved by the FDA,” “about 55 percent of cases” using superlatives related to cancer treatments were made by journalists.  This leaves about 30% percent of cases where doctors, hospitals and universities used the superlatives.  This shows an important lack of communication between media and the science community but also further shows the falsification of data from scientists.  It provides false hope to those truly suffering with a disease.  Thus, the media and medical communities must work more closely together to avoid such controversies and maintain the integrity of all parties involved.