Quick and Dirty Significance

Thought about logically and mathematically, it is unsurprising that confidence intervals and p-values are related. In the calculation of confidence intervals, for example in a t-test, the same t statistic that is calculated in order to determine a p-value is used to calculate confidence intervals. However, it is rare to see “statistical significance” addressed in terms of a confidence interval, instead of the more commonly calculated p-value. We know that confidence intervals quantify the random error seen in experimentation, and that they represent the range of values within which the true population value lies with whatever confidence we decided to use. A value outside a 95% confidence interval is unlikely to be the true population value, since we can say that the true population value lies within our interval with 95% confidence. Expanding this to the idea of “quick and dirty significance” means looking at our calculated confidence interval and determining whether our null value is contained within it. Given a 95% confidence interval, a null value outside of it has a low (<=5%) probability of being the true value. On the other hand, if our null value is included within our confidence interval, the null is at least somewhat consistent with the observed data, and further analysis is needed. Wayne W. LaMorte of Boston University School of Public Health recommends thinking of the relationship between a p-value of 0.05 and a 95% confidence interval in terms of an “embrace.” Values within the confidence interval “arms” are “embraced,” and therefore are not rejected. Null values within these arms will have calculated p-values greater than 0.05, and so are determined to not be statistically significant. Alternatively, if the 95% confidence interval does not contain the null value, there is no “embrace,” null hypothesis is rejected, and the p-value will fall below 0.05.

Furthermore, confidence intervals in multigroup comparisons can also be used as a “quick and dirty” check for significance. If intervals do not at all overlap, the 95% confidence intervals allow you to say with at least 95% confidence that there is a significant statistical difference between the groups. Large overlaps, on the other hand, support a lack of statistical significance, and p-values above 0.05. However, this method is slightly more “dirty” in that interval overlap can be as much as 25% and still be statistically significant! In these cases, it is best to conduct general calculations of p-values and be absolutely sure of your results.