Patterns Real and Imagined

One of the interesting topics covered in Harvey Motulsky’s first few chapters is the argument that probability is not intuitive because people tend to identify patterns even when none are present. He provides the example of basketball players being perceived as more likely to make or miss their next shot based on their current “streak” of successful or unsuccessful shots. As a side note, I think this is a rather poor example of the point, because it suggests that whether a basketball player makes or misses a basket is based on random chance as opposed to all the other factors that go into it. A better example is the randomly generated table provided on page 5, which could be interpreted as depicting patterns. As Motulsky points out, humans are adept at identifying patterns because it is evolutionarily advantageous to do so. I would propose that scientists are more perceptive than the average person to hints of patterns, as we are trained to detect regularities that point us to the underlying mechanisms that govern our world. That also means that we are decidedly prone to introduce bias into our work even with the best of intentions. If in the course of an experiment we start to see a trend emerging, we tend to look harder for more data that fit that trend. I discovered this during my first pilot experiments that measured disease severity in mice, and I have performed all subsequent experiments of this type blinded to genotype. Blinding is not a universal fix for this problem, though. Another instance of possibly spurious pattern recognition in data that comes to mind is multimodal populations. If you look at a scatter plot and see points clustered in what seem to be two groups, it is tempting to think that perhaps they reflect a bimodal response to a variable. Flow cytometry is another area where this can occur, as it is often possible to identify numerous populations that seem to express different combinations of marker intensity. In the complexity of biological systems, the possibility that these “patterns” in the data represent truly distinctive physiological entities is very real, and especially in more variable systems such as human studies or experiments with outbred animals, it is not at all unlikely that subsets of individuals could exhibit different responses to treatments that are based on underlying physiological differences. For instance, studies relevant to our lab’s work have found that a subset of depressed patients exhibit high levels of inflammatory markers and that their depressive symptoms can be improved with anti-inflammatories(Raison 2013). So it is important for scientists to recognize and pursue patterns that may lead to outcomes like this. But we must also recognize the potential for bias that comes if we choose to focus only on one perceived population of a dataset that “behaves better,” and also the potential to miss interesting findings by subdividing populations to the point that we lose experimental rigor.

How to think about Statistics and Confidence Intervals (for a p-value-centric scientist)

Introducing Statistics and Confidence Intervals

Statistics is, to me, man’s way of recognizing that we are imperfect and doing our best to control for it. We try to reduce bias at every level of experimentation, from study design to statistical analyses, but because this is a man-made technique of reducing man’s impact on the work that we do as scientists, it is only as effective as we are. It is the same as a computer- a computer is only as powerful and smart as the person who is running it. As such, we need to make ourselves as unbiased and as well-educated as possible in order to trust the conclusions that we draw. It is easy (and only human) to overlook many of the possible variables and situations that can cause our data to look a certain way that have nothing to do with the experimental treatment that we wish to test (and many times, that which we think we are successfully testing!).

The problem with statistics is that many times, we think we know more than we do. We are overconfident in our hypotheses and in our conclusions, and we yell on top of the data (with asterisks) instead of letting the data speak for itself. It is not enough to execute a well-designed experiment. It must be interpreted correctly as well in order to make inferences about the world around us, which is the ultimate goal of experimentation. For example, the p value is touted as the “end-all-be-all” of scientific (statistical) significance. If p<0.05, then we conclude that our treatment is working and we should get a Nature paper. However, in many cases, these small p values still beg the question, WHO CARES? If something is statistically significant, it does not mean that it is clinically relevant. Additionally, the scientific community receives (or should receive) a lot of flak for the weight they give to p values, when in fact what we should be reporting most of the time is a confidence interval. The confidence interval is intimately related to the p value, but it gives far more information and is a more accurate and informative description of the data. People do not understand p values and many times, they do not stop to think closely enough about confidence intervals either. Below are two graphs I have selected from a biostatistics lecture by Patrick Breheny illustrating the differences that result from your choice of confidence level and how they are intuitively very simple, if one takes the time to think about them…

Now, one of these graphs shows a 95% confidence interval, and the other shows an 80% confidence interval. If you think about just the values, you would (wrongly) assume that an 80% confidence interval is “worse” than a 95% confidence interval because 80 is less than 95. However, the definition of a confidence interval is that there is a X% chance that your interval contains the population mean. So, in order for you to be more sure that your interval will contain the true population value, you must widen the interval. Therefore, a 95% confidence interval is actually larger than an 80% confidence interval, but you are more confident that it contains the true population mean. Understanding this somewhat simple but very important concept is essential to generate and interpret scientific data. This course has illustrated this concept and the importance of statistics very well and I will make sure to keep this in the back of my mind throughout my career.

Error Bar Misinterpretation

Nature Methods published an article fairly recently that explains common misconception surrounding error bars. If you're like me, I thought error bars was something I could easily look at and understand. Don't they just represent the likelihood of variance between my replicate samples? Well, first, what are you using your error bars to represent? Standard deviation, standard error of the mean, or a confidence interval? This article (which provides interactive supplementary data where you can see the raw data used in the discussion as well as make your own data), provides examples of how data can look very different (or even not significant) depending on the way the error bars are represented.

A very common misconception is that a gap between bars means that the data are significant while if the bars overlap they are not significant. That is not the case, and again, it all depends on the type of bars you choose to use. 

An an example, figure 1 (above, n=10) shows how error bars cannot be compared. The left graph shows what happens to the p-value when the error bars from SD, SEM, and 95% CI are adjusted to the same lengths. The right graph shows what happens to the size of the error bars when adjusting to a significant p-value (p=0.05). As you can tell, just because SEM error bars do not overlap does not indicate significance and just because SD error bars do overlap does not mean that the data are not significant. 

When showing data with error bars it is important to be clear about which measure of uncertainty is being represented in order for the reader to be able to interpret the results properly.

Short summary:

SD: represents variation of the data and not the error of your measurements.

SEM: represents uncertaintiy in the mean and its dependency on the sample size.

CI: represents an interval estimate indicating the reliability of a measurement.