Tuesday, April 18, 2017

Pre-hoc vs Post-hoc Analysis: Whats the difference?

One of the points of emphasis for this semester has been the use of pre-hoc statistical analysis instead of post-hoc analysis when analyzing experimental data. We have been told over and over again that good statisticians plan statistical analyses before they start their experiment and never wait until after the experiment. But why is this the case? What makes pre-hoc analysis better than post-hoc analysis? The major answer to this question is that pre-hoc analyses allows for testing of a specific hypothesis while post-hoc analysis fits a hypothesis to an observed result. This difference may seem small but it has a large effect when conclusions are made from post-hoc analyses instead of pre-hoc analyses. When conclusions are made from post-hoc analyses, there is an inherent bias, as we are able to test the data in any way that produces a favorable result. In many cases, this leads to data dredging or in the worst cases, p-hacking. Pre-hoc analysis is incorporated into the experimental design and therefore the only testable hypotheses are ones specified by the experimental design. This allows us to make conclusions based on the null and alternative hypotheses, which is the basis of the scientific method. It can then be seen how most scientists should preferentially use pre-hoc analysis, however, all is not bad with post-hoc analyses. The one real benefit of post-hoc analyses is that they have the capacity to show patterns in the data that were not the primary objective of the study. This can be especially beneficial for exploratory science. The only caveat is that conclusions should not be made from post-hoc analyses, only hypotheses which can be further tested in a separate experiment. 

No comments:

Post a Comment