From colored marbles to real science experiments: looking at your data with truthfully

When you hear the word statistics, what is the first thing that comes to mind? A coin that is flipped over and over again?  A die game that you seem to never win and always blame bad luck? Perhaps built up frustration about logging into Prism and not being able to figure out which graph you need to present your data?  

Statistics seems to start off easy. The professors get excited and they say, “Let’s start with a coin!” And then they progress to “Oh! We can move to pretty, colored marbles!” And by the end of it, you’re asking yourself what in the world this has to do with the mice that are downstairs that have various treatments with various time points to measure various readouts at.

Well it all starts with coins and colored marbles. Our pretty bag of colored marbles contains known distributions of different colored marbles; therefore, it is a simple probability about which one you will choose out of the bag at random. Now take your mouse experiment, in which the distributions of the various responses are unknown. The bag of colored marbles is now all the mice in the world, but a typical experiment might have only 20 mice. How are you to determine that a response in a few of your 20 mice is not just random, but indicative of a significant result?

According to the central limit theorem, the larger our sampling, the more the distribution of responses forms a normal distribution centered around a mean. This is one large assumption that has many assumptions hidden within it. First, we assume that we can measure a mean and standard deviation from our data. We also must assume our sampling is completely random and independent. Further, we assume that with increased sampling (more mice), the standard deviation of the mean becomes smaller and smaller, allowing us to be more confident that the sample mean we are measuring (our 20 or so mice) is nearing the population mean (all the mice in the world). From the beginning, we are already assuming a lot.

From there, it gets even a bit hairier. Based on the normal distribution created from our sampling (after a few assumptions), significance is then determined by comparing a preset threshold of error to the probability of obtaining a particular result. If the probability is fairly low for obtaining a result, then it is more likely to pass below our error threshold and become significant. If it does not pass below our error threshold, there is too much error involved with claiming its significance and is more likely to have occurred by chance. It's important to see here that our definition of significance depends upon error.    

From the outside looking in, statistics seems like a black box in which data go in and significant results come out, but upon further analysis, we simply make assumptions, sample populations and then infer. Although the premise is simple, it is critical to remember that all our inferences about significance are based on “unlikelihoods” that could have occurred by chance alone and consist of many assumptions that might not have been met. A proper understanding of the statistical analyses done to yield particular results is extremely important in determining how confident we can be in those results.