Multiple Comparisons: How do we compare in Non-Parametric Data?

With my last post about nonparametric tests that are the equivalent to ANOVA tests, and focusing primarily on the Kruskal-Wallis Test, I next wanted to write a blog post about what sort of post-hoc analysis are available to us for these experiments with nonparametric data sets.

            For the most part, there seems to be little consensus among statisticians regarding what sort of comparison tests to perform post hoc on nonparametric data sets. The best answer I’ve been able to find that specifically states something to the effect of “When you wish to run a post hoc test on nonparametric data, you can use any of these tests…” provides three options. Those three are the Mann Whitney U test, the Nemenyi test, and a modified Bonferroni Dunn correction.

            First, let’s discuss the Mann Whitney U test. As previous described in my Kruskal Wallis post, the Mann Whitney U test is used to compare differences between two independent groups that have an ordinal or continuous dependent variable, but are not normally distributed. A simplified way to think about the Mann Whitney U test is to consider it to be the nonparametric equivalent to the independent t test (but this isn’t always the case). However, the Mann Whitney test provides us the opportunity to draw varying conclusions about our nonparametric data depending on what we assume about our data’s distribution, which is unlike the independent t test. These conclusions from the Mann Whitney test can range from stating whether two populations differ at all, to providing a difference in group medians. The four assumptions we can make using the Mann Whitney U test are:

1.     Your dependent variable is ordinal or continuous

2.     Your independent variable should be two categorical groups that are independent of each other.

3.     Your observations should be independent of each other.

4.     Your two independent variables are not normally distributed.

The only problem with the Mann Whitney U test is that while it provides a lower type II error, there is also the risk of potentially high type I error.

The Nemenyi test is a bit more simplified. This is a post hoc test intended to find groups within a nonparametric data set that differ after a statistical test of multiple comparisons has successfully rejected the null hypothesis. This null hypothesis would be that the comparisons show data groups to be similar. This is, unfortunately, as much as I could find and make sense of regarding this test, but I’d love to hear about anything anyone else uncovers about this test. This test is a “middle road” of these three options with a moderate type I and type II error risks.

Finally, the Bonferroni adjustment is exactly what you’d expect it to be. In a past blog about multiple comparisons, I discussed exactly what the Bonferroni correction is. Without too much detail, this test is essentially used to perform multiple comparisons without sacrificing your type I error by dividing your threshold for alpha by the total number of comparisons you’d like to make. Using this correction means the alpha for each comparison is very small to conserve the set type I error threshold for the entire comparison set. Unfortunately, the Bonferroni correction is still very conservative of type I errors at the expense of statistical power.