Showing posts with label significance. Show all posts
Showing posts with label significance. Show all posts

Monday, April 11, 2016

From colored marbles to real science experiments: looking at your data with truthfully



When you hear the word statistics, what is the first thing that comes to mind? A coin that is flipped over and over again?  A die game that you seem to never win and always blame bad luck? Perhaps built up frustration about logging into Prism and not being able to figure out which graph you need to present your data?  

Statistics seems to start off easy. The professors get excited and they say, “Let’s start with a coin!” And then they progress to “Oh! We can move to pretty, colored marbles!” And by the end of it, you’re asking yourself what in the world this has to do with the mice that are downstairs that have various treatments with various time points to measure various readouts at.
Well it all starts with coins and colored marbles. Our pretty bag of colored marbles contains known distributions of different colored marbles; therefore, it is a simple probability about which one you will choose out of the bag at random. Now take your mouse experiment, in which the distributions of the various responses are unknown. The bag of colored marbles is now all the mice in the world, but a typical experiment might have only 20 mice. How are you to determine that a response in a few of your 20 mice is not just random, but indicative of a significant result?
According to the central limit theorem, the larger our sampling, the more the distribution of responses forms a normal distribution centered around a mean. This is one large assumption that has many assumptions hidden within it. First, we assume that we can measure a mean and standard deviation from our data. We also must assume our sampling is completely random and independent. Further, we assume that with increased sampling (more mice), the standard deviation of the mean becomes smaller and smaller, allowing us to be more confident that the sample mean we are measuring (our 20 or so mice) is nearing the population mean (all the mice in the world). From the beginning, we are already assuming a lot.


From there, it gets even a bit hairier. Based on the normal distribution created from our sampling (after a few assumptions), significance is then determined by comparing a preset threshold of error to the probability of obtaining a particular result. If the probability is fairly low for obtaining a result, then it is more likely to pass below our error threshold and become significant. If it does not pass below our error threshold, there is too much error involved with claiming its significance and is more likely to have occurred by chance. It's important to see here that our definition of significance depends upon error.    
From the outside looking in, statistics seems like a black box in which data go in and significant results come out, but upon further analysis, we simply make assumptions, sample populations and then infer. Although the premise is simple, it is critical to remember that all our inferences about significance are based on “unlikelihoods” that could have occurred by chance alone and consist of many assumptions that might not have been met. A proper understanding of the statistical analyses done to yield particular results is extremely important in determining how confident we can be in those results.

Tuesday, January 19, 2016

15 Seconds of Significance



As pointed out by many, a reigning mentality in science is “publish-or-perish.” The pressure to publish results, positive ones at that, is high. And as a graduate student, I find myself being taught that this how science is conducted.
To earn my PhD, I have to publish.
The publications generated during my years in graduate school are supposed to be a way for me to demonstrate what I have learned. Sadly, publications are frequently plagued by significance bias.
The bias of significance can mean many things. The straightforward interpretation is that of statistical significance. Any positive data published must reach significance, and often times it is reached unethically. I agree with Victoria Stodden, scientists often do not use appropriate statistical methods to interpret their data. But how a scientist comes to use the wrong method is varied and is normally born of ignorance and choice. Inappropriate methods and fishing for significance is a frequent occurrence in science, and it is often caused and perpetuated by a scientist’s pressure to publish.
Another aspect of the significance bias is the drive for something to be novel. What is often valued in science is something that shakes the foundation of its field. Not so surprisingly, many publications that do this are often untruthful, irreproducible or flawed. Replication of data is important to determine the truth in a situation and no scientist should be afraid of it. Horvath stated that, “If replication were the gold standard of scientific progress, we would still be banging our heads against our benches trying to arrive at the precise values that Galileo reported. Clearly this isn’t the case.” I disagree, replication is a gold standard for the progress of science. Replication by many is simply more evidence that what was observed is real, and we can come to accept it as fact and move on from it. And yet, science as a community rewards those who publish in high impact journals, even if there data is wrong. The rewards of more funding and notoriety just reinforce the “publish-or-perish” mentality.
At present, science is stuck in the mentality of “publish-or-perish.” It is unlikely, that this and our obsession with significance will ever change, until science is no longer approached as a for profit business.