How to think about Statistics and Confidence Intervals (for a p-value-centric scientist)

Introducing Statistics and Confidence Intervals

Statistics is, to me, man’s way of recognizing that we are imperfect and doing our best to control for it. We try to reduce bias at every level of experimentation, from study design to statistical analyses, but because this is a man-made technique of reducing man’s impact on the work that we do as scientists, it is only as effective as we are. It is the same as a computer- a computer is only as powerful and smart as the person who is running it. As such, we need to make ourselves as unbiased and as well-educated as possible in order to trust the conclusions that we draw. It is easy (and only human) to overlook many of the possible variables and situations that can cause our data to look a certain way that have nothing to do with the experimental treatment that we wish to test (and many times, that which we think we are successfully testing!).

The problem with statistics is that many times, we think we know more than we do. We are overconfident in our hypotheses and in our conclusions, and we yell on top of the data (with asterisks) instead of letting the data speak for itself. It is not enough to execute a well-designed experiment. It must be interpreted correctly as well in order to make inferences about the world around us, which is the ultimate goal of experimentation. For example, the p value is touted as the “end-all-be-all” of scientific (statistical) significance. If p<0.05, then we conclude that our treatment is working and we should get a Nature paper. However, in many cases, these small p values still beg the question, WHO CARES? If something is statistically significant, it does not mean that it is clinically relevant. Additionally, the scientific community receives (or should receive) a lot of flak for the weight they give to p values, when in fact what we should be reporting most of the time is a confidence interval. The confidence interval is intimately related to the p value, but it gives far more information and is a more accurate and informative description of the data. People do not understand p values and many times, they do not stop to think closely enough about confidence intervals either. Below are two graphs I have selected from a biostatistics lecture by Patrick Breheny illustrating the differences that result from your choice of confidence level and how they are intuitively very simple, if one takes the time to think about them…

Now, one of these graphs shows a 95% confidence interval, and the other shows an 80% confidence interval. If you think about just the values, you would (wrongly) assume that an 80% confidence interval is “worse” than a 95% confidence interval because 80 is less than 95. However, the definition of a confidence interval is that there is a X% chance that your interval contains the population mean. So, in order for you to be more sure that your interval will contain the true population value, you must widen the interval. Therefore, a 95% confidence interval is actually larger than an 80% confidence interval, but you are more confident that it contains the true population mean. Understanding this somewhat simple but very important concept is essential to generate and interpret scientific data. This course has illustrated this concept and the importance of statistics very well and I will make sure to keep this in the back of my mind throughout my career.

Is bias a necessary evil?

Biases are ubiquitous. And because they are ubiquitous, we must embrace them for we cannot escape. This is especially true to science. I see biases as vital and necessary components of science. I often see bias portrayed as being bad--rightfully so as bias can really screw us over as portrayed by the articles I read. But from reading these articles, I wanted to find reasons for how bias can be good. Perhaps I am misunderstanding the connotation of the word "bias" or applying the word subjectively? No matter what, here are some reasons why I think bias could be a good thing:

1. Biases, aka hypotheses, are the impetuses for projects. Scientist must have a central belief, which are biased by our expertise, past life experiences, our colleagues, mentors, and etc, to which we frame our scientific questions. I believe that these biases provides the momentum for the creation of projects and propel discovery. Without our constantly changing biases science would not be in perpetual motion. 
2. Biases allow us to be more critical, allowing for the advancement of science. We are taught as scientist to always question what we see, what we read, and what we hear. We would not have such critical minds if we did not have a bias that something published is not always true or causative. I think that such skepticism pushes science forward. 
3. Biases force us to do better science. The point of publishing is share your discoveries with supportive evidences that try to minimize biases. Because we have these biases and want our results to be as objective as possible, we design "controlled" experiments. Thus, bias forces us perform scientifically valid experiments and analyze data that can best confirm our hypotheses.

Here are my thoughts on how bias is a necessary evil in science. Without them it may be hard to pose a scientific question, make it impossible to be more critical, and most importantly, perform and analyze truly honest and objective experiments. Since there are many examples of how bias can be detrimental to science, I just wanted to be a devil's advocate and provide some reflections about how bias can actually be good for science.

The Morality of Reproducible Data

            As academic scientists, we of course are invested in our research, and (ideally) want to leave our own mark on our respective fields. The data that we generate and publish do not just contribute to personal edification, but also the understanding of a topic on a global scale. With the wide accessibility of scientific journals, data is consistently reanalyzed and published findings applied to new experiments in a growing international research network. Keeping this in mind, it is more critical than ever before that the research community emphasize the importance of sound (and complete) data and experimental reproducibility.

            In his TED talk, Dan Ariely discussed how it was more likely for every person in a room to cheat a little than for just one to completely cheat. Students would give themselves a “4” in lieu of the “2” they deserved, presumably so that they would get a slightly better reward while still maintaining a degree of self-respect. This is sadly applicable in the research world, in the form of “cherrypicking” data or only withholding any negative findings from papers. Findings may often be smudged or spun in a certain light, or an incomplete story presented, not nearly enough to warrant a retraction, but just enough that the findings may not be entirely trustworthy. I agree with Jared Horvath’s Scientific American article in that funding provides a constant pressure for scientists to focus first on generating marketable data, and second on generating complete or valid data. However, while funding is a legitimate concern, I do not think that it is an excuse to perform unviable research or twist results. Granted, this is easy for me to say as a graduate student with a guaranteed stipend, but labs that produce questionable data do more than fail to contribute to science; they actually detract from ongoing research. False data can mislead other researchers who may use these findings as a baseline for their own projects, which in turn could possibly fail or lead to more misdirection. The withholding of negative data could lead other labs to pursue these and waste precious grant money rediscovering what should already be public domain.

            After reading some of these articles, it seems that it should be easier than ever to make sure that research is well-executed, given the formation of organizations, such as the PLoS ONE New Reproducibility Initiative and PubPeer. While these opportunities should be taken with a grain of salt, they seem like a viable means for experts to help fact-check or ensure that results hold true. I do think there is a critical difference between difficult and irreproducible experiments, in that some procedures may have a low success rate due to the necessity for high level of technical skill or specialized setup. However, if even a group of experts in the same field cannot recapitulate a finding, something is likely at fault with the underlying experimental strategy or the published data.