What it means to be scientific

The word “scientific” is one that I have been thinking about a lot recently. It comes up in casual conversation all the time, as in “that’s not very scientific” to mean that something is uncertain, or “she has a very scientific mind” to describe someone who is organized and meticulous. I cringe internally when I hear this phrase, because, I think, science is not very scientific. The public perception of science is that it is methodical, linear, organized; that we as scientists think of a question and that science gives an answer. It is no surprise that they think this, because it’s how we present our work both in academic journals and to the public. We create a story when writing a paper or presenting our work, making it sound like we knew what was going to happen all along and that the conclusions we drew were inevitable. But we know that this is not how science works. In reality, we come up with a question and a possible way to find the answer; our idea doesn’t work, or gives us an answer we weren’t expecting, or the project is more difficult than we anticipated – science is a messy, nonlinear, unplanned process.

This disconnect between the reality of conducting science and the story we tell to the public creates two problems. First, it elevates scientists to a level in our society where we are almost mythical beings, solving the unanswered questions of the universe.  Second, it creates public distrust when we start to talk about the problems of bias and irreproducibility in science, or when we have to walk back claims we’ve made. How could such a clear process result in results that can’t be obtained twice? How could such pure people, working to explain the world to everyone else, be biased? While we as scientists have to grapple with how to make our results replicable and what irreproducibility means for our fields, I think that being more honest about how science works would help to prevent the sense of disillusionment the public feels when we are inevitably revealed to be imperfect.

For instance, scientists, doctors, and journalists should stop overstating the impact and significance of our work. As explained in this Vox article, we often describe our findings as miraculous or curative when in fact they have simply moved the field forward one small step. A quick google search revealed that we’re going to cure HIV in the next three years, when in reality human CRISPR trials are much further off. Instead of telling the public that we’ve cured HIV or any particular disease, we can be more honest and tell them that we’ve unlocked another small piece of the 10,000-piece jigsaw puzzle – that doesn’t make it less exciting that we’re curing HIV in mice, it just makes it less confusing for the public when HIV isn’t cured in 2020. As graduate students, we can be honest with our friends and family about the realities of daily life as a scientist rather than painting a rosier picture of the scientific process. We can also help them to interpret findings in the media and explain when results are truly outstanding versus being an interesting novel finding.  

Finally, we can also be more honest with ourselves about the role our discoveries play in the larger scientific community. My favorite analogy for science is that “solving” any question is like pushing a boulder up a mountain. No individual is going to get the boulder to the top, but we will each push the boulder a bit further. This does not mean that our work is not important. This does not mean, for those of us who study problems relevant to human health, that our work will not help real people someday. But if we can be honest with ourselves about the relative insignificance of our individual discoveries, I think it will help to keep science closer to the pure ideal of the pursuit of unbiased knowledge.

Seeking Integrity on the Shoulders of Giants

One of the books that has most shaped my view of the world is Mother Night by Kurt  Vonnegut. In it, the narrator describes his philosophy of how people’s minds are like clockwork, with every fact we know serving as a tooth in a cog. In many ways, though, we are missing teeth. Either we are unintentionally blinded to truth because of the environment in which we were raised, or we purposely deny truths because they do not fit nicely with our mental constructs. He says, “I… will say that I have never tampered with a single tooth in my thought machine, such as it is. There are teeth missing, God knows -- some I was born without, teeth that will never grow. And other teeth have been stripped by the clutchless shifts of history -- But never have I willfully destroyed a tooth on a gear of my thinking machine. Never have I said to myself, 'This fact I can do without.’” As a scientist, my greatest quest is to find truth and to try to fill the gaps in our understanding of the world. Yet, one of my greatest fears is that, knowingly or unknowingly, I am missing teeth in my gears.

Our first lecture with TJ Murphy struck home with me not only because it brought to light the “reproducibility crisis,” but it also reminded me of my part in it. I see that we today are standing upon the shoulders of giants. Everything that we know and believe to be truth is based upon the years of work of those before us. Though I am astounded by the brilliance, creativity, and passion of our predecessors, a part of me is also frightened. How much of what we take for granted is actually truth? In what ways do I contribute to the reproducibility crisis, and where does my own blindness come into light? How can I help promote an environment of scientific integrity? Christopher Pannucci and Edwin Wilkins cite the many different forms of biases that can distort an investigator’s ability to assess their findings, so I must confront and address my biases head-on before starting a project. As stated in The Economist, science is not necessarily self-correcting, so my fellow researchers and I must begin the movement to be more open about science as a powerful, yet fallible, tool for approaching our world’s many questions.

The Paradox of Trust in Science

As an undergraduate student, I find that most, if not all, of my lectures incorporate information which students naturally presume is credible, reliable evidence. Very rarely does a professor inquire about the shortcomings of data or urge us to approach publications from a critical perspective. Upon reading a PubPeer article and learning that 25% of 120 randomly selected published papers revealed image data errors made me ponder why we constantly accepted literature as truth. I was frequently under the impression that if a study was published, it underwent a rigorous review process and was subsequently immortalized in an impressive journal....thus what was there to question? It was only after I joined a lab when I began to pause and truly assess studies within the growing scope of my knowledge. I also became increasingly aware of one of the greatest challenges facing research scientists: striking a balance between the demanding pressure to publish and being conscientious about data production throughout. 

The case of the overhyped medical press does not surprise me, however. I personally have fallen prey to such articles, namely those that occupy headlines with "breakthrough findings" that daily use items are carcinogenic. It's no secret that journalists are cognizant of how to grasp the layman's attention, albeit with content that isn't scientifically sound. Often, such articles are followed by disclaimers that, for example, the hot new weight loss drug working "miracles" is contingent upon x, y, z and/or has yet to be tested on humans. 

On another note, it is reassuring to know that individuals are making strides towards developing peer feedback initiatives such that widespread expertise can be offered (PubPeer) or in the case of PubMed Commons, establish an ongoing review system. But, there is much more that needs to be done to achieve the level of scientific credibility humanity deserves. As we discussed in lecture, statistics' primary objective is to identify and avoid bias, therefore it is imperative that as a scientific community, we understand and implement the appropriate tests/tools to prevent unacknowledged flaws from persisting throughout our comprehension. Further, we must better integrate sound statistics knowledge with our perception of the research process - specifically as elucidated by Bayesian analysis, what a 5% false positive rate of scientific hypotheses means for our ability to reproduce given results. I hope that all contributors to this dynamic field can eventually become more well-versed in statistics to better uphold ethical standards and ultimately, allow the public to regain its trust in the currently misleading realm of scientific research.  

The statistics of statistics

The largest issue, in my opinion, with irreproducibility and bias is scientific research isn’t a lack of awareness, and it is not even the fact that these problems exist. The greatest problem is that there is a lack of understanding of why, and a lack of self-awareness that you personally are capable of bias. To my point of awareness of bias and irreproducibility that is not the issue. As obvious in the homework assignment, there have been many articles “shedding light” on the issue. It is so common that the term irreproducible data is more like a running lab joke then a real day-to-day concern. Most people blame the “perish or publish” culture, which puts a lot of pressure on publishing data as soon as possible, the vague (either on purpose or not) methods sections, and also the use of statistics. The pressure to publish will never go away. The concern with a lack of transparency in methods section is also an issue that arises mostly due to the high pressure to publish, but also because there are small things that make a large difference, and the research is not even aware of these. After my admittedly short time in science (6 years) I do believe that this issue has started to be addressed, and may be a more personal then systematic issue.  

The argument for blaming the use of statistics is a complicated one. This is because statistics can be extremely powerful and necessary, particularly as researchers move toward analyzing large data sets and “-omics” type studies. The main argument is that statistics is either used to freely or in a basic understanding, or that complicated statistics are used to make data seem more significant then truly are. However, as pointed out in Jeremy Berg’s blog, the issue is that scientists do not really understand how the statistics work. Importantly, they do not understand the bias that is inherent in the statistics, and why a significant fraction of experiments cannot be repeated exactly.

If you fully understand the statistics on how easily data could not be reproducible it is easier to swallow the thought that irreproducible data is common place, and has always been a part of scientific research. This is argued by John Horvath, where he stresses that if we accept that most of the data published is false or irreproducible, we can then strive to focus on what is true. As scientists we are taught always to question data or idea, and this does not end just because something is “statistically significant”.  However, if we as scientists can accept this and focus on the ideas being presented, and how we can use the small amount of “true” data to move science further then the irreproducibility is not as huge an issue as we once thought, as long as we are being honest with our data and eliminating personal biases. Meaning it is OK to accept that there is a small level of irreproducibility, but we cannot add to it with our personal biases, otherwise that small level will become very large.

This leads to an issue of awareness. Being aware of our ability to bring bias into our research. As Dan Ariely pointed out in his TED talk, “a lot of people cheat a little bit”. We all have what he refers to as a fudge factor. We are willing to cheat just a little bit, but in general most people (and here I’m really saying most scientists) do not make up data, we simply allow our biases to creep into our experiments both in design and analysis. Dan states that one of these is because of our social norms, we as a scientific community need to educate researcher on how to avoid experimental biases, and we need to accept that our intuition is not always correct.

Follow the Data, Not the Hypothesis

The scientific community is built upon the idea that that the best way to answer a question is through logical experimental design, redundant testing and unbiased analysis of results. Many members of the scientific community sought out its membership to sequester themselves from the ideological, emotional and political turmoil that so frequently drives major discussions in the rest of society. We often see ourselves as the cogent alternative to such trivialities.  However, there are two very important points that we should keep in mind whenever we feel start to feel high and mighty:

Scientists are human beings. And as human beings we are susceptible to all the flaws of human perception and quirks that put us at risk for misrepresenting our results. One particularly concerning tendency is that of confirmation bias, the propensity to look for evidence only in the support of the expected result. Often, discussions of confirmation bias are trotted out on the world stage to illuminate large conflicting results, but it is equally at play when a scientist chooses to quantify this cell instead of that cell because it looks more like how it should. 

Just as concerning is the human tendency to cheat--just a little-- if it is beneficial and difficult to prove. Dan Ariely, a behavioral economist, discusses this fudge factor in depth during his TED Talk about our "buggy" moral code. Of specific interest is his evidence that we are much more likely to fudge our own results if it is an accepted part of our in-group, our community. Therefore, if a practice (even a wrong practice) is common among scientists, it is likely to be perpetuated, especially if it is perceived as beneficial.

This conveniently leads us to point number two.

Humans suck at statistics. We do. We can't conceptualize our chance to win the PowerBall. We can't understand how big a million is. And, most definitely we can't understand what statistical analyses actually mean. Too often, scientists perform statistics in a retrospective and perfunctory, fill-in-the-blank way. There is a dearth of understanding in how to analyze statistical results because people can't easily tell the difference between an occurrence and a significant occurrence, or how numbers relate to each other.

Perhaps this is why sources report that less than half (and that's being generous) of published scientific studies can be replicated, which could contribute to the growing trend of public disbelief in scientific findings. As scientists, it is hard not to take this personally or to get angry when confronted with a person who misuses data to fuel their own argument. But really-- is there any difference between a congressman reformatting data to assault planned parenthood and you avoiding some questionable data to get a Nature paper?