One of the points of emphasis for this semester has been the
use of pre-hoc statistical analysis instead of post-hoc analysis when analyzing
experimental data. We have been told over and over again that good statisticians
plan statistical analyses before they start their experiment and never wait
until after the experiment. But why is this the case? What makes pre-hoc
analysis better than post-hoc analysis? The major answer to this question is that pre-hoc
analyses allows for testing of a specific hypothesis while post-hoc analysis
fits a hypothesis to an observed result. This difference may seem small but it has
a large effect when conclusions are made from post-hoc analyses instead of
pre-hoc analyses. When conclusions are made from post-hoc analyses, there is an
inherent bias, as we are able to test the data in any way that produces a
favorable result. In many cases, this leads to data dredging or in the worst
cases, p-hacking. Pre-hoc analysis is incorporated into the experimental design
and therefore the only testable hypotheses are ones specified by the
experimental design. This allows us to make conclusions based on the null and alternative
hypotheses, which is the basis of the scientific method. It can then be seen
how most scientists should preferentially use pre-hoc analysis, however, all is
not bad with post-hoc analyses. The one real benefit of post-hoc analyses is that
they have the capacity to show patterns in the data that were not the primary
objective of the study. This can be especially beneficial for exploratory
science. The only caveat is that conclusions should not be made from post-hoc
analyses, only hypotheses which can be further tested in a separate experiment.
No comments:
Post a Comment