"Publish or Perish", Human Nature, and Media Hype--A Bad Cocktail

The "publish-or-perish culture" that dominates science today has pushed the field in an undesirable direction. But despite the fact that much of the science we see published today is inconsistently reproducible, this is not all entirely due to the "publish-or-perish" culture, nor maleficent cherry-picking motivated by self-gain. The truth behind the current situation is much more benign, although equally worrisome and harmful.

The first explanation simply lies in the imperfection of science. An article on Scientific American reveals that even the scientists we consider greatest had experiments that led to irreproducibility upon others’ attempts to repeat it. Rather than stigmatize irreproducibility and experiments that didn’t “work,” it would be more beneficial to open up discussion and generate a space where it is possible to talk about this issue and resolve it, as is being done in PubPeer. By making discussion about the more fallible aspects of science open, researchers wouldn't feel pressured to only report experiments that “worked” and hide those that didn’t. Consequently, experiments that had statistically insignificant or unexpected results could serve as advice or inspiration for future research.

The effects of the stigma against insignificant results and irreproducibility are compounded by human nature. More insidious and difficult to detect are our conflicts of interest and tendencies for dishonesty. While it is easier to detect conflicts of interest in others, as Dan Ariely posits, it is incredibly difficult to detect it in ourselves. Sometimes we simply think we are furthering science by eliminating outliers that are hiding the significance of our results which we believe to be correct. It is difficult but necessary to take a step back and realize that it not for the benefit of science for us to do so, but for our own.

The situation is exacerbated by the media. By reporting medical “breakthroughs” and “miracles” in studies that sometimes weren’t even done in humans, science is portrayed as a lot more all-powerful than it actually is. This adds to the atmosphere of pressure for researchers to meet that same misleading bar.

Bias in science needs to be dealt with from square one. Rather than allow ourselves to fall prey to the pressures of the scientific community to publish and our own well-meaning intentions to illustrate hypotheses we believe should be true, we need to accept that sometimes science isn’t infallible, breakthroughs don’t happen as often as the media reports, and experiments that don’t yield significant results aren’t failures.

Little Cheaters

Dan Ariely in his talk, ‘The Honest Truth About Dishonesty’ at The Amaz!ng Meeting 2013 introduces the concept of little cheaters, that is, people who are dishonest in ways that they consider small enough to maintain personal morality while still reaping benefits of dishonesty. This concept was derived from studies in the general population suggesting that scientists too are privy to such behavior, but what implications does this have for science?

The most likely effect of dishonesty in science is irreproducibility. If experiments are planned, executed, interpreted, or reported with even the slightest amount of dishonesty, they are impossible to repeat by others. Consequences extend beyond those who seek to replicate to those attempt to build on the existing work as they would be working off likely incorrect information. Such deception is clearly undesirable but eliminating it can be difficult as perpetrators may not always be aware of their deception because they perform it while convinced of their morality. This is further compounded by the inherent conflict of interest that exists in all scientists. Every researcher holds stake in the success of their work: graduate students benefit from publishing papers and graduating early, senior investigators gain career advancement and increase their marketability for grant funding by presenting positive results. All these factors color the objectivity of researchers making it harder to recognize the subtle ways in which they can be dishonest such as inflating the meaning of their findings or omitting unfavorable results. Proper statistics should be able to check this bias but it is no secret that many laboratory scientists are not sufficiently conversant in statistical methods.

What then, does the combination of dishonesty, bias, and poor statistical knowledge mean science is doomed? Should presenting work be put off until these problems are eliminated? No. Rather, science needs to be redefined as the work in progress that it is and not the subject of irrefutable answers as perceived by many. Efforts should be taken certainly, to minimize blatant falsehood in published work, but it should also be acceptable to not be quite certain. Scientists will be more likely to shed their little cheater identity when it is fine to have work that does not completely make sense.

Why you always lying?

Do scientists lie? Yes everyone lies. Why do scientists lie? We, like our genes, are selfish. Does it matter that scientists lie? 

If we can trust Dan Ariely’s talks, then his research demonstrates that people in general lie and cheat marginally (and often) without feeling like they are dishonest or bad. Fair enough. We can all think of times, or at least I certainly can (‘No officer I don’t know how fast I was going’; ‘Yes gas station attendant I am over 21’), we have lied for minor personal gains.

Apparently situations where there is a conflict of interest, a distance between the lie and direct monetary outcome, and people you identify with are also lying lead to more misbehavior. The first two parameters are met by scientists. It is obviously in the interest of a scientist to perform research with interesting results that gets published, and although publishing is connected to monetary gain it isn’t a direct transaction. But many would argue that the third parameter, other scientists lying about their data, is not something that happens. Perhaps not so blatantly. Ariely goes on to discuss asking golfers if they have picked up a golf ball and moved it. No. Kicked it while looking the other direction? Of course. For scientists there is a distinct possibility for a similar scenario. Have you ever falsified data? No, that is repugnant. Have you ever used statistics you didn’t fully understand to analyze your data and make them look good? ......

The kicker is that dishonesty in science is neither new nor always problematic. Jared Horvath describes how important scientists from Galileo to Millikan produced research that is not replicable yet helped to push our collective understanding forward. Still it could be argued that falsification is occurring at a greater rate in contemporary science. In “Trouble at the Lab” John Ioannidis is described as saying that the majority of published findings are false. The author further notes that very few articles are retracted.

So is the process of science failing in our new age? No. A scientific paper should not be expected to be 100% correct. Hell one of the main lessons being beat into our bones as grad students is that we should always be ferociously looking for errors while reading papers. Just because a paper has mistakes does not mean that it contains no useful information. The presence of useful information implies the paper shouldn’t be retracted. Yes this means that to obtain useful information from a paper you need more than a lay understanding of the field, but the entire point is that we perform research on the cusp of our society’s understanding. The health of the scientific enterprise should not be measured by how many mistakes there are in published articles, but by how much progress we are making toward improving our society. 

Honest Science

       We like to assume that the scientific literature always reflects the truth. However, there is a growing recognition that scientific findings are not guaranteed to be reproducible, causing many to question the validity of published results. This has led to dismay, exemplified in a 2016 Nature poll reporting that, of the 1,576 researchers who participated, 52% agreed that there is a ‘significant crisis’ of reproducibility in scientific research. Irreproducible research is a valid concern, since making scientific progress would seem to be difficult without the ability to corroborate results.

       Some scientists argue, however, that this lack of reproducibility in science is nothing new. As discussed in an article by Dr. John Horvath in Scientific American from 2013, unreliable research has been common since the beginning of modern science, and indeed is necessary for scientists to push the boundaries of what we know. Experiments that do not go as expected or hypotheses that are eventually proved to be incorrect are often what lead to novel discoveries. This is not a popular idea though, because it could potentially cause the public and funding agencies that may not understand the true pace of science to lose trust in the effectiveness of scientific research.

        Regardless of whether or not scientific findings are more or less reproducible now than in the past, the conclusion from this discussion is the same: we need to ensure that we prioritize being upfront and honest about experimental results. It is easy to allow personal opinions and biases affect the way results are portrayed in scientific literature and to the public. This human tendency to present results in the best light possible is amplified by the growing pressure to publish and the intense competition for funding and jobs. Everyone involved in the scientific process, from graduate students to PIs to journal editors need to be encouraged to make truth and honesty a priority in the way research is conducted and portrayed.