Scientists, especially graduate students, have become too focused and driven on results being statistically significant. We play statistical significance up to be all-important in science; most of our experiments and projects focus on finding some difference. If we don’t get the results that are “statistically significant”, we feel like failures and that something went wrong. Maybe I am generalizing too much of my own experience in graduate school, but bear with me. Graduate school is notoriously viewed (well, at least by me) as “soul-sucking.” I believe that much of these feelings of hopelessness and failure originate from the moment you press “Analyze” on Prism and see “ns.” Imagine how different graduate school would be if that feeling of failure were eliminated…how would things be if we took every negative result and no longer viewed it as a dead end or a reflection of our abilities as scientists? What if when we saw “ns” we could feel joy and not distress? I feel like our success in graduate school is defined by statistical significance; without a p<0.05, our hard work means nothing. When was the last time that any of us went to a thesis defense that focused on non-significant results? Why has it become that a statistically significant result is necessary to earn our doctorate? Would our education be at a disadvantage if were not required to present statistically significant data?
In a way, statistical significance helps to remove bias by allowing for quantification and comparison of results in order to look for a difference. Statistics and calculating a P value are what allow western blots to be informative and unbiased. Without P value, there would likely be variation in what some would say “looks” like a difference between two groups. Science needs statistical significance. However, statistical significance has also created bias in the way that we approach problems. The need for statistical significance prevents us from exploring concepts and hypotheses that may turn up to be of no significance. The need for statistical significance may also lead a researcher (without proper statistical training) to increase the n of their experiment to the point where a p value of <0.05 is inevitable. It has become unacceptable to just say no significance; we force our P value to mean something, even if it’s just “trending” towards significance. Statistical significance and p values both eliminate bias as well as create it.
I feel that people don’t actually think about what “statistically significant” means; all a P value can tell us is the probability that we could see that a result of the same magnitude if the null hypothesis were true. It cannot actually tell us how likely the alternative hypothesis is true. Thus, we need to stop defining the importance of our work by the P value. Motulsky brings up that colloquialisms may contribute this problem of focusing on statically significance a P values. We associate the term significance with importance, which is incorrect when interpreting statistics. In order to interpret statistics, one must understand the theory and definitions of the terms used. Then, and only then, can we understand that statistics does not interpret the importance of our experimental results; it only allows us to accept or reject the null hypothesis. We can no longer define our work and goals by “statistical significance”; instead we should be seeking scientific importance.