Thought about logically and mathematically, it is
unsurprising that confidence intervals and p-values are related. In the
calculation of confidence intervals, for example in a t-test, the same t
statistic that is calculated in order to determine a p-value is used to
calculate confidence intervals. However, it is rare to see “statistical
significance” addressed in terms of a confidence interval, instead of the more
commonly calculated p-value. We know that confidence intervals quantify the
random error seen in experimentation, and that they represent the range of
values within which the true population value lies with whatever confidence we
decided to use. A value outside a 95% confidence interval is unlikely to be the true
population value, since we can say that the true population value lies within
our interval with 95% confidence. Expanding this to the idea of “quick and
dirty significance” means looking at our calculated confidence interval and
determining whether our null value is contained within it. Given a 95%
confidence interval, a null value outside of it has a low (<=5%) probability
of being the true value. On the other hand, if our null value is included
within our confidence interval, the null is at least somewhat consistent with
the observed data, and further analysis is needed. Wayne W. LaMorte of Boston University School of Public Health recommends
thinking of the relationship between a p-value of 0.05 and a 95% confidence
interval in terms of an “embrace.” Values within the confidence interval “arms”
are “embraced,” and therefore are not rejected. Null values within these arms
will have calculated p-values greater than 0.05, and so are determined to not
be statistically significant. Alternatively, if the 95% confidence interval
does not contain the null value, there is no “embrace,” null hypothesis is rejected,
and the p-value will fall below 0.05.

Furthermore, confidence intervals in multigroup comparisons
can also be used as a “quick and dirty” check for significance. If intervals do
not at all overlap, the 95% confidence intervals allow you to say with at least
95% confidence that there is a significant statistical difference between the
groups. Large overlaps, on the other hand, support a lack of statistical
significance, and p-values above 0.05. However, this method is slightly more “dirty”
in that interval overlap can be as much as 25% and still be statistically
significant! In these cases, it is best to conduct general calculations of
p-values and be absolutely sure of your results.

## No comments:

## Post a Comment