Showing posts with label p-hacking. Show all posts
Showing posts with label p-hacking. Show all posts

Tuesday, April 19, 2016

One-tailed or two-tailed?



I’d like to raise a question that’s occurred to me several times during this class. Many of the problem sets we’ve been given have required the use of a one-tailed t-test, and TJ has been pretty dismissive of two-tailed t-tests, saying that they’re used by people who aren’t willing or haven’t collected enough preliminary data to make a real hypothesis. Before this class, however, I was taught that two-tailed tests were preferable and that one-tailed tests were usually used inappropriately for, as TJ would call it, “p-hacking.”
I can see merit in both arguments. Typically, we have a pretty good idea of what we expect to happen in our experiments, and we’ve often done enough preliminary tests to formulate a hypothesis that does predict a change in only one direction. In that case, why not get as much rigor as we can from our statistical test and devote all the alpha allowance to testing our exact prediction? However, in my years of research, I’ve rarely been in a situation where I only cared about a change in one direction. If you’re testing a drug, you certainly want to know if it improves your disease condition, but everyone who might ever be affected by that drug hopes that you’re also making sure it doesn’t increase the severity of the disease. In my dissertation research, I’ve often predicted that knocking out an anti-inflammatory regulator in the gut will increase expression of pro-inflammatory mediators and decrease levels of barrier-promoting proteins, but I’ve frequently seen that what really happens is that the protein level of a pro-inflammatory cytokine increases, but its mRNA levels decrease because of negative feedback, or that the expression levels of a barrier protein increase because it’s being degraded more rapidly. If I tried to do statistical tests that only looked for change in one direction, I’d miss these realities even though my fundamental hypothesis about increased inflammation was correct.
If you decide to do a one-tailed test and see a change in the opposite direction, not only do you get no statistical info about it, but technically, if you want to evaluate whether that change is significant, you have to go back and repeat the study again with a different statistical plan. Now in reality, no one’s going to do that; they’re just going to go back and run a different statistical test on the same data in hopes of getting a significant result. At that point, you’re changing your stats for a p-value, no way around it. It’s even worse if you then repeat the test with a one-tailed in the opposite direction, because then you’re essentially performing a two-tailed test with an alpha of .1. If you did decide to go by the book and repeat the experiment planning for a two-tailed or a different one-tailed test, then at least you’re wasting time, lab resources, and taxpayer/donor money, and frequently also animal lives or human samples, which raises an ethical dilemma.
It seems to me that there are very few times when it would be appropriate to do a one-tailed test (for instance, you’re transfecting with an overexpression plasmid; you only care if there’s an increase in expression of your target, because if not, you’re going to repeat the transfection), and the rest of the time, you should just power your studies appropriately for a two-tailed t-test rather than risking missing a potentially important biological result.

Thursday, April 7, 2016

P-hacking and publication bias

iii. P values and statistical significance

(Note: names have been changed)
I glared at the data glowing back at me on the computer screen – the postdoctoral fellow who was mentoring me had mentioned finding a p-value for all the numbers I had. 
“Sorry, Florence, how did you want me to analyze this data again?” 
“Just calculate the SEM for each group, stick the numbers in PRISM, and then look for the p-value and see if the difference is significant.” Then she walked away. Something about a motor neuron prep and her mouse embryonic spinal cords sitting on ice for too long. 
I had no idea how to do what she wanted me to do. What is an SEM? Why aren’t we calculating standard deviation instead? Google searches brought up a myriad of statistics websites that attempted to explain how to derive the equation used to calculate an SEM, but with little context as to why you would even want an SEM in the first place. I tried to recall the statistics class I took sophomore year, but only came up with “the p-value indicates significant results.” …Right? I sat there questioning my own competence before continuing to toil over how to do the calculations on the numbers from my qPCR. When I finally got the numbers and graphs, I was dismayed that there was no significance. When Florence came around again, I showed her the results. 
“Oh. Well, that sucks. But there seems to be a trend. And we’re only at n=2, so I think if we just increase our n, it’ll probably be significant.”


Type of error bar
Conclusion if they overlap
Conclusion if they don’t overlap
SD
No conclusion
No conclusion
SEM
P > 0.05
No conclusion
95% CI
No conclusion
P < 0.05
(assuming no multiple comparisons)
 Rule of thumb provided by GraphPad's FAQ

How many of us have been put in a similar situation or have heard of a situation like this?

Without a strong background or understanding of statistics, I blindly trusted Florence’s logic and choice of statistical analyses – she was a postdoctoral fellow after all. She’s probably done more than two dozen of these kinds of statistics on her own data that granted her her Ph.D. She must know what she’s doing, I reasoned. But that was the danger of scientists who were improperly or inadequately trained to conduct statistical analyses: in hindsight, I realized that 1) few people (or even scientists, me included) actually understand what “significance” really means, and 2) as Motulsky puts it, “once some people hear the word significant, they often stop thinking about what the data actually show.” The scenario I recounted is something Simmons, Nelson, and Simonsohn (2012) termed “P-hacking,” a term that refers to attempts by investigators to lower the P value by trying various analyses or by analyzing subsets of data. Motulsky draws out two ways in which investigators do this: 1) by tweaking data (if one analysis didn’t give a P value less than 0.05, then they tried a different one) and/or 2) by changing the sample size post hoc (stopping data collection if the P value is less than 0.05, but collecting more data when the P value is about 0.05).

One study by Gotzche (2006) looked at comparing the number of publications that reported a P value between 0.04 and 0.06, hypothesizing that if results were published honestly, the number of publications reporting a P value between 0.04 and 0.05 and a P value between 0.05 and 0.06 should be similar. In the analysis, Gotzsche found that there were five times as many papers reporting P values between 0.04 and 0.05 compared to P values between 0.05 and 0.06. The emphasis on statistical “significance” equating as scientific significance ends up skewing the publication of results and data and creates publication bias. I really wonder if some scientists believe that inadvertently p-hacking is a legitimate way to conduct statistical analyses, or if some do it knowing that it is the improper way to generate "significant" results.


Perhaps the fix here is for journals to start requiring authors to submit a short cover note explaining the justification of the utilized statistics to corroborate that they understood why and how the statistical tools were chosen and used. In this way, it could force scientists to not only conduct reliable and properly designed experiments, but also to think more carefully about the interpretation of their results, rather than just trying to force or find significance that might not be there.