With my last post about nonparametric tests that are the
equivalent to ANOVA tests, and focusing primarily on the Kruskal-Wallis Test, I
next wanted to write a blog post about what sort of post-hoc analysis are
available to us for these experiments with nonparametric data sets.
For the
most part, there seems to be little consensus among statisticians regarding
what sort of comparison tests to perform post hoc on nonparametric data sets.
The best answer I’ve been able to find that specifically states something to
the effect of “When you wish to run a post hoc test on nonparametric data, you
can use any of these tests…” provides three options. Those three are the Mann
Whitney U test, the Nemenyi test, and a modified Bonferroni Dunn correction.
First,
let’s discuss the Mann Whitney U test. As previous described in my Kruskal
Wallis post, the Mann Whitney U test is used to compare differences between two
independent groups that have an ordinal or continuous dependent variable, but
are not normally distributed. A simplified way to think about the Mann Whitney
U test is to consider it to be the nonparametric equivalent to the independent
t test (but this isn’t always the case). However, the Mann Whitney test
provides us the opportunity to draw varying conclusions about our nonparametric
data depending on what we assume about our data’s distribution, which is unlike
the independent t test. These conclusions from the Mann Whitney test can range
from stating whether two populations differ at all, to providing a difference
in group medians. The four assumptions we can make using the Mann Whitney U
test are:
1.
Your dependent variable is ordinal or continuous
2.
Your independent variable should be two
categorical groups that are independent of each other.
3.
Your observations should be independent of each
other.
4.
Your two independent variables are not normally
distributed.
The only problem with the Mann Whitney U test is that while
it provides a lower type II error, there is also the risk of potentially high
type I error.
The Nemenyi test is a bit more
simplified. This is a post hoc test intended to find groups within a
nonparametric data set that differ after a statistical test of multiple
comparisons has successfully rejected the null hypothesis. This null hypothesis
would be that the comparisons show data groups to be similar. This is,
unfortunately, as much as I could find and make sense of regarding this test,
but I’d love to hear about anything anyone else uncovers about this test. This
test is a “middle road” of these three options with a moderate type I and type
II error risks.
Finally, the Bonferroni adjustment
is exactly what you’d expect it to be. In a past blog about multiple
comparisons, I discussed exactly what the Bonferroni correction is. Without too
much detail, this test is essentially used to perform multiple comparisons
without sacrificing your type I error by dividing your threshold for alpha by
the total number of comparisons you’d like to make. Using this correction means
the alpha for each comparison is very small to conserve the set type I error
threshold for the entire comparison set. Unfortunately, the Bonferroni
correction is still very conservative of type I errors at the expense of
statistical power.
