How p-values of underpowered studies select for exaggerated effect size

When we think of experimental power, we're usually in the mindset of setting things up to have  enough power to detect a treatment effect. In fact, we should also think about detecting an effect size accurately. It turns out that underpowered experiments not only make it less likely to detect effects, but are also more likely to lead to an exaggerated effect size.

Here's a simulation to illustrate this point.

We simulate a run of unpaired two-sided t-tests that compare a drug effect to a placebo effect at each of two different power levels. Because we define the parameters of the sampled population, we know for a fact that the response level to the drug is 350 response units (SD = 120). The response level to the placebo is 200 response units (SD = 120 also). The variables are assumed to be normally distributed.

Thus, in testing over the long run, the drug would be expected to have an average effect of 150 response units over that of placebo. How accurately do experiments conducted at low and high power estimate this effect size between the drug and placebo?

The experiment is simulated at low power (~50%, sample size per group = 6) and at high power (~90%, sample size per group = 15). An unpaired two-sided t-test is done for each of 1000 random samples under each of the two power conditions. At 50% power about half of the t-tests have a p-value of 0.05 or less, whereas 90% of the tests at higher power are positive.

Now let's look at the averages of the mean group response values only in those tests that correctly detect a "significant" difference between drug and placebo treatment.

In the low power simulation, the average drug effect is 375 response units and that for placebo is 176, for a difference of 199 response units. Thus, in the long run, the average estimate from the low power is inaccurate by 49 response units.

In contrast, the high power simulation yields response unit estimates of 353 and 197 for the drug and placebo groups, respectively. They are accurate to within 6 response units from what we coded in for the groups.

When using p-value based cut off criteria to make decisions on whether an experiment worked or not, we are at risk of making inaccurate estimates of the effect size of a treatment. The high powered tests not only detect more "positive" results, but they are also much more accurate!

Some in the biomedical world have called for increasing the stringency of type 1 error tolerance, setting the "significance" threshold at p< 0.01 rather than p<0.05 alpha. What effect do you think this will this have on effect size estimation of underpowered experiments?

p less than 0.05: habit or sound clinical judgment?

The p-value is the observed significance level of a result. In the context of hypothesis testing it is the smallest level of significance at which the null hypothesis would be rejected for a given test procedure and data set.  This allows for comparison with a chosen threshold of type one error, alpha, which is the probability of the rejecting the null hypothesis when it is actually true.  Thus a p-value is the probability of obtaining a test statistic value at least as contradictory to the null hypothesis as the value observed if the null hypothesis is actually true.  In terms of applicability, the p-value is used when determining if a test statistic value is statistically significant or not where the threshold of significance is defined by the minimum level of type 1 error one is willing to accept.  In other words, the null hypothesis can be rejected (i.e. result of a test statistic is significant) if the probability of obtaining a test statistic value at least as contradictory to the null hypothesis as the observed test statistic value is less than the maximum probability one is willing to accept that the null hypothesis is actually true in this context of rejecting it.

Now the threshold for significance accepted by most in the scientific community is an alpha of 0.05.  However, is that perhaps too large? If we are trying to determine if a particular treatment is successful at treating a disease is a 5% chance that the determination that this treatment is successful was a false positive acceptable?  Now a lot of this is dependent upon the clinical context.  If this were a case where a patient had little to no other alternatives to this treatment than it may be judged an appropriate threshold.  Alternatively if there are treatments that are equally as effective a higher threshold for significance may be necessitated.  Other factors aside from alternate treatment availability could be cost, risk of complications, and contraindications for future treatment options to name a few. It is trusted that clinical judgment is used when determining a threshold alpha, yet 5% seems to be the standard so much so that I suspect that it is used more out of habit rather than out of sound clinical judgment. 

Additionally, it is important to make an assessment with regards to the goals of the study and whether a type 1 or type 2 (false negative) error would be worse.  The significance of our statistical methods is focused on type 1 error, but if type 2 error is also or more critical that must be accounted for in the study design.  For example, the power of a study needed, the compliment of type 2 error, should be determined a priori which would allow for an appropriate sample size to be determined to ensure the study is not underpowered, thus reducing the chance of type 2 error.

The p-value selected should also depend on the nature of the statistical analysis.  For example, while for one test a p-value of 0.05 may seem reasonable if we begin to perform multiple tests and comparisons the actual error will increases over the collection of studies and it becomes more likely that among our results a false positive has occurred.  Thus in the case of multiple comparisons it is essential to select a lower p-value to compensate.

As a final note, as had been alluded to throughout this blog post, the p-value, although the current foundation of statistical hypothesis testing, does not tell the whole picture (e.g. power and effect size) and is perhaps used too frequently, as a matter of habit. It should be expected that researchers determine what they actually want to show with their analysis and what the implications of their results will be and then choose the appropriate statistical test and parameters (alpha, beta, n ect.) accordingly.

Quick and Dirty Significance

Thought about logically and mathematically, it is unsurprising that confidence intervals and p-values are related. In the calculation of confidence intervals, for example in a t-test, the same t statistic that is calculated in order to determine a p-value is used to calculate confidence intervals. However, it is rare to see “statistical significance” addressed in terms of a confidence interval, instead of the more commonly calculated p-value. We know that confidence intervals quantify the random error seen in experimentation, and that they represent the range of values within which the true population value lies with whatever confidence we decided to use. A value outside a 95% confidence interval is unlikely to be the true population value, since we can say that the true population value lies within our interval with 95% confidence. Expanding this to the idea of “quick and dirty significance” means looking at our calculated confidence interval and determining whether our null value is contained within it. Given a 95% confidence interval, a null value outside of it has a low (<=5%) probability of being the true value. On the other hand, if our null value is included within our confidence interval, the null is at least somewhat consistent with the observed data, and further analysis is needed. Wayne W. LaMorte of Boston University School of Public Health recommends thinking of the relationship between a p-value of 0.05 and a 95% confidence interval in terms of an “embrace.” Values within the confidence interval “arms” are “embraced,” and therefore are not rejected. Null values within these arms will have calculated p-values greater than 0.05, and so are determined to not be statistically significant. Alternatively, if the 95% confidence interval does not contain the null value, there is no “embrace,” null hypothesis is rejected, and the p-value will fall below 0.05.

Furthermore, confidence intervals in multigroup comparisons can also be used as a “quick and dirty” check for significance. If intervals do not at all overlap, the 95% confidence intervals allow you to say with at least 95% confidence that there is a significant statistical difference between the groups. Large overlaps, on the other hand, support a lack of statistical significance, and p-values above 0.05. However, this method is slightly more “dirty” in that interval overlap can be as much as 25% and still be statistically significant! In these cases, it is best to conduct general calculations of p-values and be absolutely sure of your results. 

The value of the P-value

Scientist tend to place a lot of importance in the P value. PIs loose grants over P values, grad students have nervous break downs over “insignificant data”, undergraduates desperately look for outliers they can redo to correct that p-value. Is the P-value really that important?

As we have learned in class this semester, the p-value is the probability of obtaining a result equal to or more extreme than predicted with the null hypothesis. So a low p-value indicates that it was unlikely that random chance gave you such an outlying result. Many scientist would stop when they see that low number, wipe the sweat off their brows, and submit their grant or paper.

However is a p-value of 0.051 really all that much worse than a p value of 0.049? Is the difference between those p-values really worth a nervous breakdown? The p-value is an arbitrary line drawn by scientists, and often young scientists fail to look beyond it. A p-value is just one statistical number that can be used in conjunction with other statistical values such as the mean, confidence interval, and error, to make a decision about the implications of an experiment.

A scientist needs to use his/her judgment to conclude if an experiment is worth pursuing further or if it is a waste of time; a significant or insignificant p value should not make that decision. An insignificant p-value does not mean the data was worthless, but only a clear minded scientist can determine what it actually means. Likewise, but much less considered, a significant p-value does not mean that the experiment showed a result worth celebrating over. Your significant p-value might indicate that your drug decreased the stress levels of mice 10%, but is a 10% decrease in mice stress really significant for any application?

These sorts of questions are what scientists should be thinking about. In one experiment, a p-value may hold a lot of weight, but in another experiment, the confidence interval might tell much more about the results. Scientist must be able to analyze their data, not just statistically, but critically to know what their statistics are actually telling them.

How to think about Statistics and Confidence Intervals (for a p-value-centric scientist)

Introducing Statistics and Confidence Intervals

Statistics is, to me, man’s way of recognizing that we are imperfect and doing our best to control for it. We try to reduce bias at every level of experimentation, from study design to statistical analyses, but because this is a man-made technique of reducing man’s impact on the work that we do as scientists, it is only as effective as we are. It is the same as a computer- a computer is only as powerful and smart as the person who is running it. As such, we need to make ourselves as unbiased and as well-educated as possible in order to trust the conclusions that we draw. It is easy (and only human) to overlook many of the possible variables and situations that can cause our data to look a certain way that have nothing to do with the experimental treatment that we wish to test (and many times, that which we think we are successfully testing!).

The problem with statistics is that many times, we think we know more than we do. We are overconfident in our hypotheses and in our conclusions, and we yell on top of the data (with asterisks) instead of letting the data speak for itself. It is not enough to execute a well-designed experiment. It must be interpreted correctly as well in order to make inferences about the world around us, which is the ultimate goal of experimentation. For example, the p value is touted as the “end-all-be-all” of scientific (statistical) significance. If p<0.05, then we conclude that our treatment is working and we should get a Nature paper. However, in many cases, these small p values still beg the question, WHO CARES? If something is statistically significant, it does not mean that it is clinically relevant. Additionally, the scientific community receives (or should receive) a lot of flak for the weight they give to p values, when in fact what we should be reporting most of the time is a confidence interval. The confidence interval is intimately related to the p value, but it gives far more information and is a more accurate and informative description of the data. People do not understand p values and many times, they do not stop to think closely enough about confidence intervals either. Below are two graphs I have selected from a biostatistics lecture by Patrick Breheny illustrating the differences that result from your choice of confidence level and how they are intuitively very simple, if one takes the time to think about them…

Now, one of these graphs shows a 95% confidence interval, and the other shows an 80% confidence interval. If you think about just the values, you would (wrongly) assume that an 80% confidence interval is “worse” than a 95% confidence interval because 80 is less than 95. However, the definition of a confidence interval is that there is a X% chance that your interval contains the population mean. So, in order for you to be more sure that your interval will contain the true population value, you must widen the interval. Therefore, a 95% confidence interval is actually larger than an 80% confidence interval, but you are more confident that it contains the true population mean. Understanding this somewhat simple but very important concept is essential to generate and interpret scientific data. This course has illustrated this concept and the importance of statistics very well and I will make sure to keep this in the back of my mind throughout my career.

ASA Statement on P Values

The American Statistical Association has released a statement on P Values. If you haven't read it, you should. There really isn't anything new here.

I've seen a few headlines already that have clearly misinterpreted the statement. Such as (I paraphrase), "ASA Declares the P Value Dead." Coupled with some variation of a dance around the carcass meme.

We're still sort of left wondering. How, when and why did the train jump the tracks?

Maybe it's too many people doing the scientific process and too few of them doing it grounded in a scientific philosophy. In other words, this one is on the scientists, not the statisticians.

Significance of Objectivity

As of late, reproducibility has been a hot topic in the field of science ethics. As I thought about our inherent bias and how we can combat human nature to overcome these biases, I found myself considering the system we are forced to work within. In a career that is competitive and driven by the old motto “publish or perish!”, there is a strong emphasis on both the quantity and the quality of publications. But therein lies the rub: how do we judge the quality of work that is to be published? Peers who are, hypothetically, both the experts and the competition in your field? Publishers more experienced with 20^th century English literature than the latest super resolution microscopy techniques? So as a community we establish rules to govern what constitutes “quality” research. How many technical and biological replicates did you do? Was your data significant?

By creating these rules and trying to fit them, valuable negative results are lost in lab notebooks as unpublishable. Worse, data is massaged until it fits a “significant” result. The journal of Basic and Applied Psychology has gone so far as to outright ban p- values from their publications! Novella wrote that the use of p-values encourages lazy thinking, as if the magical value of p<0.05 must mean your hypothesis is correct, when in reality that number is entirely arbitrary. The founders of PubPeer, an open and anonymous forum for discussing scientific work post publication, describe the difficulties of publishing quality work as “chas[ing] ‘metrics’” in the search for impact when we are “ruled by often incompetent kingmakers”.  These metrics are held up as the gold standard for success, but ultimately lead to the failure of good science in favor of winning the numbers game.

Statistics show the system is broken. According to an article in ASBMB today by Jeremy Berg, 75-90% of important cancer research is not reproducible. Another article showed that 25% of randomly selected publications in three cancer journals had major flaws in imaging data. I cite this specifically because imaging data and statistics seem to me to be the least understood analyses. Reviewers fail to see the problems in these areas, not only because of the ability of the researcher to manipulate that data, but also because the reviewers don’t understand the methods enough to think critically about them.

The system is broken, yes, but reading these articles also made it clear that people are thinking deeply about the issues we face and the best steps to take to fix the problems for the future.