Monday, April 11, 2016
It's not you, it's P?
The challenge in interpreting statistical vs. clinical significance has come to the fore front of clinical and basic research alike. Deeply embedded in this conversation is the reliability of p-values and statistical significance as an exhaustive measurement for testing a result's validity. Interestingly, a recent statement released by the American Statistical Association has strongly suggested the growing need to steer science into a "post p<.05 era". The ASA's executive director compellingly argues that "the p-value was never intended to be a substitute for scientific reasoning" and that "Well-reasoned statistical arguments contain much more than the value of a single number and whether that number exceeds an arbitrary threshold." However, I personally believe that while this reasoning has a sound basis, it seems just a tad hypocritical that he makes such a large, sweeping generalization about the applicability of p-values while one of the reasons why p-values are viewed with scrutiny is because of their widespread-usage with out serious consideration for the specific scientific application. It's naive to think that the inclusion of p-values hasn't had a net-positive effect on the scientific community (especially in the infancy of modern research) by forcing one to evaluate the significance of their data, however arbitrary that line may have become in some fields today or may inherently be in certain scientific fields. Given this separate consideration, I think the ASA's statement would do better to reach out to to major scientific organizations and field-specific research leaders and to work with these parties to critically assess how to move-forward into this post p<.05 era but to do so with regard for creating sound statistical parameters on a field-by-field basis. Of course, this is a demanding request but if indeed statistical tests require the complete context of their application then shouldn't each scientific field work together to comprehensively establish not just some arbitrary set of standards e.g. .05 but a set of standards that accommodates the needs of that field. Perhaps, national societies and organizations would do well to utilize their national conferences as vehicle for instigating this shift in paradigm. As the ASA statement importantly highlights that the p-value was never meant to replace sound scientific reasoning it would seem equally important at this critical juncture to rely on the scientific reasoning of statisticians and field leaders to set a new precedent for evaluating data.