Multiple Comparisons Testing : Bonferroni versus Sidak Methods

The last installment of my brief guide to multiple comparisons testing are the Bonferroni and Sidak correction methods. The Bonferroni method, or correction as it is sometimes referred to, is probably the simplest and easiest of the multiple comparisons tests to comprehend, (at least for me). In short, the correction adjusts the individual thresholds for p-values when multiple independent or dependent statistical tests are performed all at the same time. Based around the central problem with multiple statistical tests, which is that you burn your type-one error with every comparison you make, the Bonferroni method corrects for this problem in the most direct route possible. If you have X number of comparisons to be made, you simply take your threshold for type one error, (α), and divide it by X comparisons (α/X), which provides a much smaller threshold for the type one error for each individual comparison.

            To be put even more simply, when you use the Bonferroni method, the chance of knowing a result reported as significant is actually significant increases with the more comparisons you make.

            The Sidak (the first S is pronounced as a SH) method or correction for multiple comparisons was published in 1967 by the statistician and probabilist Zbyněk Šidák. Where the Bonferroni correction focuses on the threshold of type one error for each comparison made, the Sidak correction focuses on the opposite question. If the Bonferroni correction asks the question of “ What is the probability that my statistically significant result is actually statistically significant?” then the Sidak correction asks the question, “What is the probability that the differences are not statistically significant?”
            The example the Graph Padwebsite provides suggests to think about it in this way: If we want to make three comparisons, and each one is under the same threshold for type one errors, which is 5%, (α = 0.95, then 1-α = 0.05) and that means that the chance of all three comparisons being not statistically significant is equal to (0.95*0.95*0.95 = 0.8574). And thus the chance that one or more of the comparisons will actually be statistically significant is 1-0.8574 = 0.1426. The only catch with Sidak’s method for correction is that you have to assume that all of your individual tests are independent of each other.

There is plenty more math out there detailing how exactly the Bonferroni and Sidak methods differ, but the easiest juxtaposition I can discern is how they go about calculating α. Bonferroni sets α for each comparison based on the number of comparisons being done, and the Sidak method calculates an exact α all comparisons in a reverse-thinking method. Based on these, it is often considered that Sidak’s method produces α values that are less stringent than those of the Bonferroni corrections.