Little Cheaters

Dan Ariely in his talk, ‘The Honest Truth About Dishonesty’ at The Amaz!ng Meeting 2013 introduces the concept of little cheaters, that is, people who are dishonest in ways that they consider small enough to maintain personal morality while still reaping benefits of dishonesty. This concept was derived from studies in the general population suggesting that scientists too are privy to such behavior, but what implications does this have for science?

The most likely effect of dishonesty in science is irreproducibility. If experiments are planned, executed, interpreted, or reported with even the slightest amount of dishonesty, they are impossible to repeat by others. Consequences extend beyond those who seek to replicate to those attempt to build on the existing work as they would be working off likely incorrect information. Such deception is clearly undesirable but eliminating it can be difficult as perpetrators may not always be aware of their deception because they perform it while convinced of their morality. This is further compounded by the inherent conflict of interest that exists in all scientists. Every researcher holds stake in the success of their work: graduate students benefit from publishing papers and graduating early, senior investigators gain career advancement and increase their marketability for grant funding by presenting positive results. All these factors color the objectivity of researchers making it harder to recognize the subtle ways in which they can be dishonest such as inflating the meaning of their findings or omitting unfavorable results. Proper statistics should be able to check this bias but it is no secret that many laboratory scientists are not sufficiently conversant in statistical methods.

What then, does the combination of dishonesty, bias, and poor statistical knowledge mean science is doomed? Should presenting work be put off until these problems are eliminated? No. Rather, science needs to be redefined as the work in progress that it is and not the subject of irrefutable answers as perceived by many. Efforts should be taken certainly, to minimize blatant falsehood in published work, but it should also be acceptable to not be quite certain. Scientists will be more likely to shed their little cheater identity when it is fine to have work that does not completely make sense.

Why you always lying?

Do scientists lie? Yes everyone lies. Why do scientists lie? We, like our genes, are selfish. Does it matter that scientists lie? 

If we can trust Dan Ariely’s talks, then his research demonstrates that people in general lie and cheat marginally (and often) without feeling like they are dishonest or bad. Fair enough. We can all think of times, or at least I certainly can (‘No officer I don’t know how fast I was going’; ‘Yes gas station attendant I am over 21’), we have lied for minor personal gains.

Apparently situations where there is a conflict of interest, a distance between the lie and direct monetary outcome, and people you identify with are also lying lead to more misbehavior. The first two parameters are met by scientists. It is obviously in the interest of a scientist to perform research with interesting results that gets published, and although publishing is connected to monetary gain it isn’t a direct transaction. But many would argue that the third parameter, other scientists lying about their data, is not something that happens. Perhaps not so blatantly. Ariely goes on to discuss asking golfers if they have picked up a golf ball and moved it. No. Kicked it while looking the other direction? Of course. For scientists there is a distinct possibility for a similar scenario. Have you ever falsified data? No, that is repugnant. Have you ever used statistics you didn’t fully understand to analyze your data and make them look good? ......

The kicker is that dishonesty in science is neither new nor always problematic. Jared Horvath describes how important scientists from Galileo to Millikan produced research that is not replicable yet helped to push our collective understanding forward. Still it could be argued that falsification is occurring at a greater rate in contemporary science. In “Trouble at the Lab” John Ioannidis is described as saying that the majority of published findings are false. The author further notes that very few articles are retracted.

So is the process of science failing in our new age? No. A scientific paper should not be expected to be 100% correct. Hell one of the main lessons being beat into our bones as grad students is that we should always be ferociously looking for errors while reading papers. Just because a paper has mistakes does not mean that it contains no useful information. The presence of useful information implies the paper shouldn’t be retracted. Yes this means that to obtain useful information from a paper you need more than a lay understanding of the field, but the entire point is that we perform research on the cusp of our society’s understanding. The health of the scientific enterprise should not be measured by how many mistakes there are in published articles, but by how much progress we are making toward improving our society. 

Bias is human nature

Whenever I think about the issue of bias and irreproducibility in science, there are two quotes that come to mind. 

“73.6% of all statistics are made up.” – Mark Suster 

The second quote was popularized by Mark Twain, who attributed it to Benjamin Disraeli: 

“There are three kinds of lies: lies, damned lies, and statistics.” 

How do these quotes relate to the issues of reproducibility and bias in science? The irony of the first quote is it is itself a made up statistic, meant to demonstrate that people will parrot figures without first validating their veracity. The second quote highlights that statistics can be deceitful if misrepresented. Combine misleading statistics with the repetition of false information, and you have a crisis in the validity and reproducibility of scientific data. You do not have to go far to find proof of this phenomenon: This article discusses the source of hype around new cancer drugs, which stems from both journalists and scientists repeating statistics without understanding the full context of the situation. Yet, it is not just scientists who do this. How many times have you or a Facebook friend read a statistic and then repeated it, without understanding where that number came from? Misleading facts combined with repetition without confirmation means it is very easy to fool ourselves into thinking there is something in the numbers when in actuality, there is nothing.

To demonstrate how easy it is to fool ourselves, take a look at the graph below: 

These graphs look related, right? An r value of .666 is not terrible. Let’s add some labels.

Do you believe this graph? It seems pretty reasonable, right? But let’s look at what the graph actually represents.

Surprise! This a spurious graph where the two variables have nothing to do with each other, yet look related because of the way the data is represented.

The point is, statistics is tricky. It is easy to ignore facts and justify what we want to see, especially when it benefits us. I think this may be a big reason why science is currently in a data crisis; it is not necessarily out of intentional malice, but rather because human beings are inherently bias, and we inherently make connections, false or not, between data sets. Of course, there are those who intentionally falsify data or manipulate data to fit their theories, but that’s a whole other topic.

Graph source