Showing posts with label confidence intervals. Show all posts
Showing posts with label confidence intervals. Show all posts

Monday, April 11, 2016

Quick and Dirty Significance

Thought about logically and mathematically, it is unsurprising that confidence intervals and p-values are related. In the calculation of confidence intervals, for example in a t-test, the same t statistic that is calculated in order to determine a p-value is used to calculate confidence intervals. However, it is rare to see “statistical significance” addressed in terms of a confidence interval, instead of the more commonly calculated p-value. We know that confidence intervals quantify the random error seen in experimentation, and that they represent the range of values within which the true population value lies with whatever confidence we decided to use. A value outside a 95% confidence interval is unlikely to be the true population value, since we can say that the true population value lies within our interval with 95% confidence. Expanding this to the idea of “quick and dirty significance” means looking at our calculated confidence interval and determining whether our null value is contained within it. Given a 95% confidence interval, a null value outside of it has a low (<=5%) probability of being the true value. On the other hand, if our null value is included within our confidence interval, the null is at least somewhat consistent with the observed data, and further analysis is needed. Wayne W. LaMorte of Boston University School of Public Health recommends thinking of the relationship between a p-value of 0.05 and a 95% confidence interval in terms of an “embrace.” Values within the confidence interval “arms” are “embraced,” and therefore are not rejected. Null values within these arms will have calculated p-values greater than 0.05, and so are determined to not be statistically significant. Alternatively, if the 95% confidence interval does not contain the null value, there is no “embrace,” null hypothesis is rejected, and the p-value will fall below 0.05.
Furthermore, confidence intervals in multigroup comparisons can also be used as a “quick and dirty” check for significance. If intervals do not at all overlap, the 95% confidence intervals allow you to say with at least 95% confidence that there is a significant statistical difference between the groups. Large overlaps, on the other hand, support a lack of statistical significance, and p-values above 0.05. However, this method is slightly more “dirty” in that interval overlap can be as much as 25% and still be statistically significant! In these cases, it is best to conduct general calculations of p-values and be absolutely sure of your results. 

Counter-intuitive apophenia and common statistical problems


Introducing Statistics and Confidence Intervals

An interesting idea that I wanted to share was the concept of “Apophenia”, or the problem where humans find patterns and meaning where there is none. This idea got me thinking, and I went back and flipped through our textbook, and noticed several instances in the first section where the author discusses how humans tend toward counter-intuitive decisions. For example, Motulsky says, “Our brains have simply not evolved to deal sensibly with probability, and most people make the illogical choice.” When it comes to scientific research, the most obvious illogical choices made always seem to tie to the statistics reported to support a publishing’s claim. The two most common culprits with this problem are the P-value and confidence intervals (or more often, the lack thereof), and these are two basic concepts that were introduced early in the textbook.
But why are these two statistics problematic?
The more I researched about P-values, confidence intervals, and the statistical pitfalls of scientific research, the more I felt I was reading the same words and phrases. Everyone agrees unanimously and rallies behind this banner with the battle-cry for reproducible research, and for more stringent statistical reporting! But it was rare to find a proposal of any actual mechanism to correct the problems.
One article from Erika Check Hayden in NatureNews explored an interesting proposal from statistician Valen Johnson. Johnson developed a method to directly compare, “the P value in the frequentist paradigm, and the Bayes factor in the Bayesian paradigm”, and then re-examine select published data to see how the two juxtaposed. His results found that the common standard of p≤ 0.05 coordinates with a weak Bayesian factor, and that of the published data reviewed, “as many as 17–25% of such findings are probably false”. To ameliorate this problem, Johnson suggests, “to use more stringent P values of 0.005 or less to support their findings, and thinks that the use of the 0.05 standard might account for most of the problem of non-reproducibility in science — even more than other issues, such as biases and scientific misconduct”.
Another article I found from Nature News that was published in Medicine by Jean Baptiste-du-Prel et al. discusses when p-values and confidence intervals are most important individually, and when it’s important they’re reported combined. For example, one major point that is discussed is that, in clinical research, the over-stressed p-value is a problem most reports make, and Baptiste-du-Prel even states, “the investigator should be more interested in the size of the difference in therapeutic effect between two treatment groups […] rather than whether the result is statistically significant or not”. The take home message from this article was that both p-values and confidence intervals are intertwined statistics should be reported together to bolster credibility.
While these and other articles I found in my research were very interesting, I can’t help but think about how these more stringent statistical standards will be achieved, and what sort of impact raising the bar will have on new scientists just beginning. While I, personally, am excited at the idea of knowing all the papers I read genuinely are statistically significant, I can’t help but wander back to the problem that got me researching to begin with.

How much of the statistics published in scientific literature is apophenia that stems from lack of basic statistical knowledge?
In a “publish or perish” scientific arena, I can certainly understand why so many statistical mistakes can slip through the cracks, but that doesn’t really change the fact of why there is this statistical apophenia at all. Perhaps if instead of raising the bar for statistical stringency in scientific publishing, we should raise the standards for statistic education for scientists.  Then the problems of abusing the p-value, misinterpreting significance, and making unnecessary statistical decisions illogically, would correct themselves.

Sunday, April 10, 2016

Political Polls: A Quagmire of Sampling Errors

It is once again a presidential election year, which means mainstream media is practically covering our daily lives in a bloodbath of tortured poll results. Everywhere you look, there is yet another new opinion poll showing how the political pundits are faring against each other. However, before even looking at any of these results, we should be questioning how the survey was designed before it even began and how the data was actually collected.

The end goal of these political opinion surveys is to infer how the whole population of voters in the US will vote in November. However, sampling the opinion of every single American voter is completely impossible given the constraints of time, money, and willingness/truthfulness in every voter answering. So instead, polls must base their opinions on a smaller sample, a subset of individuals taken from the greater population. If the sample is chosen by random sampling, then the distribution of opinions of those people will be similar to that of the entire population. Randomization is key here. If there are any parts of the population that are intentionally (or unintentionally) excluded or over-represented in polling, then the sample distribution of opinions will not necessarily resemble that of the whole population. Problems in proper random sampling have been hotly debated for decades as discussed by Jill Lepore in a recent article in The New Yorker, and the debates have only grown more contentious in recent years with changes in technology. For example, many opinion polls are conducted by random dialing of phone numbers and collecting responses of whoever picks up. However, telephone response rates are currently in the single digits, and people who willingly pick up the phone often have very different political behaviors than the general public. Furthermore, random calling to cell phones was banned in the 1991 Telephone Consumer Protection Act, so calls are only going to land lines and not cell phones. And once again, the demographics of people who have land lines is not fully representative of the whole population, excluding younger voters in particular. After the UK elections in 2015, Nate Silver, the widely beloved political analyst behind FiveThirtyEight, highlighted a few recent examples of inaccurate political polls and some of the reasons to worry about bad sampling, such as the inability to rely on contact by phone, unreliable online polling methods, Americans withholding their true opinion in polls, and herd opinions swaying voters.

Although political polls aspire to have completely representative polling, an accurate sampling is nearly impossible in our current culture given the reasons above. To compensate for this, many political polls use methods that could be considered scientific heresy to scientists working in hypothesis-driven experiments. For non-probability sampling methods, pollsters make the assumption that each sample selected does not have an equal chance of being chosen from the population (i.e. younger voters with cellphones but no land lines are far less likely to be chosen in phone-based polling methods). There are various methods to compensate for the changes in variability, such as weighting the results of certain samples more than others. For example, a phone-based poll could weight the opinions of younger voters or minority voters more so than the opinions of older white voters more likely to have phone lines. But remember, although this seems like a practical method to “fix” political polls, it makes for really really bad science. Hypothesis-driven scientific studies only work on the assumption that all samples were RANDOMLY chosen from the population, i.e. the probability of choosing any one sample is equal for all samples in the population. Using non-probability sampling or not using true randomized selection throws this assumption out the window, runs it over with a semi-truck, and lights the remains on fire. Never trust scientific experiments using non-probability methods, they are scientifically unsound and completely useless when it comes to predicting anything of value about the whole population. And for political polls, make sure to investigate the weighting methods used and determine yourself how much you actually trust the reported results.

Another important feature of political polls is their sampling error. By pure chance, any sample poll of a population will have a slightly larger or smaller proportion of subjects voting a certain way compared to the true population vote. Pollsters only know the opinions of the small sample, not of the whole population, so the best way to show how sure you are that your sample represents the whole population is to calculate confidence intervals. A poll result percentage by itself is completely useless for making assumptions about the population without a confidence interval. By pure tradition, almost everyone uses a confidence interval of 95%. This means that the range of values reported has a 95% chance of containing the actual population value and a 5% chance that it does not contain the population value. For example, let’s pretend you polled 100 people about if they would vote for Bernie Sanders or Hillary Clinton in a state’s Democratic primary. If 54 of 100 voters in the sample say they would vote for Bernie, then the 95% CI for the sample is +/-9.77%, which means that the percentage range of 44.23% and 63.77% has a 95% chance of containing the population value of the proportion of voters who will vote for Bernie.


Increasing your sample size decreases your confidence interval, as illustrated below by PewResearch.This is because the confidence interval is roughly proportional to the square root of 1/n where n is the number of samples. So, bigger n leads to smaller confidence intervals. 

If you repeat the poll above for Hillary vs Bernie in 1000 sampled voters, and find 54% of voters +/-3.09% support Bernie to 95% confidence, then the percentage range of 50.91% to 57.09% is 95% likely to contain the population percentage of voters supporting Bernie. It’s still the same sample percentage, but the 95% confidence interval is much smaller.


If you want to take a closer look at current political polls, I highly encourage you to look at the compiled list of national polls on FiveThirtyEight and search through the methods sections to see how each poll plays with non-probability sampling weighting, random sampling methods, and determination of confidence intervals. The lack of standards between polls is both horrifying and fascinating and makes me question the results of all political polls, especially when methods aren’t clearly reported.

I’ll leave you with this one interesting snippet of the methods section of an NBC/SurveyMonkeypoll that actually acknowledges inherent bias in their selection methods and how they compensated for that in their methods.  It’s rare that a poll will actually acknowledge their bias like this, so enjoy this unusually honest example.














Saturday, April 9, 2016

I have confidence in confidence (intervals) alone!

Plain and simple, confidence intervals allow us to express the accuracy of our data. They're suitable for almost any type of biological measurement as you need three basic things: the sample mean, the standard deviation, and the sample size. Once you determine the degree of confidence you find acceptable for the experiment, you can use those three values to calculate a confidence interval (CI). 95% confidence is standard, though some situations call for a much more narrow CI, such as astrophysics or nuclear weapons. If you choose a conventional CI of 95%, then you are stating the range of values within which there is a 95% chance that the true population value lies.

The example of weather brought up in the Garfield comic is an easy one to think about in this context. If you want to state a high confidence interval, then you must either have many measurements, a small standard deviation in your measurements, or being willing to accept a very large range of values as your interval. Sure you can say confidently that the temperature in Atlanta, GA is between -40°F and 200°F at any given time, because it’s not meteorologically relevant to state otherwise. Say, instead, that you measure the temperature once an hour for 24 hours, so your sample size equals 24. Perhaps your sample mean is 72°F with a standard deviation of ±7°F. Even though your calculated sample mean is absolutely 72°F, you didn’t measure the temperature continuously all day long, so there’s a very real chance that your mean is an inaccurate representation of the actual fluctuations that occurred, particularly if the temperature quickly rose and then dropped again or vice versa. So you can calculate that you are 95% confident the true temperature mean for the past 24 hours is between 69.2°F and 74.8°F.

You must make many assumption about your data in order to calculate a confidence interval correctly. Namely, you assume that your sample is a random representation of the population, that your data are independent and unbiased, and that your data was accurately obtained. In the scenario I outlined, a faulty thermometer that measures 5 degrees warmer than the actual temperature would ruin the accuracy of your CI. The reality is biology isn’t always as easy to interpret as the temperature scenario I outlined, or to understand when there are problems with the data. What if your project is so novel that you don’t know saying the value could be between -40 and 200 is ludicrous? Maybe the value should be 60 and 80, but the time frame in which you’re capturing your data is so large that unknown biology is skewing your results and causing a large standard deviation?


95% confident sounds like such a sure thing, but it’s important to think critically about what these values are telling you. Is it actually biologically relevant?





Thursday, March 31, 2016

How to think about Statistics and Confidence Intervals (for a p-value-centric scientist)


Introducing Statistics and Confidence Intervals

Statistics is, to me, man’s way of recognizing that we are imperfect and doing our best to control for it. We try to reduce bias at every level of experimentation, from study design to statistical analyses, but because this is a man-made technique of reducing man’s impact on the work that we do as scientists, it is only as effective as we are. It is the same as a computer- a computer is only as powerful and smart as the person who is running it. As such, we need to make ourselves as unbiased and as well-educated as possible in order to trust the conclusions that we draw. It is easy (and only human) to overlook many of the possible variables and situations that can cause our data to look a certain way that have nothing to do with the experimental treatment that we wish to test (and many times, that which we think we are successfully testing!).

The problem with statistics is that many times, we think we know more than we do. We are overconfident in our hypotheses and in our conclusions, and we yell on top of the data (with asterisks) instead of letting the data speak for itself. It is not enough to execute a well-designed experiment. It must be interpreted correctly as well in order to make inferences about the world around us, which is the ultimate goal of experimentation. For example, the p value is touted as the “end-all-be-all” of scientific (statistical) significance. If p<0.05, then we conclude that our treatment is working and we should get a Nature paper. However, in many cases, these small p values still beg the question, WHO CARES? If something is statistically significant, it does not mean that it is clinically relevant. Additionally, the scientific community receives (or should receive) a lot of flak for the weight they give to p values, when in fact what we should be reporting most of the time is a confidence interval. The confidence interval is intimately related to the p value, but it gives far more information and is a more accurate and informative description of the data. People do not understand p values and many times, they do not stop to think closely enough about confidence intervals either. Below are two graphs I have selected from a biostatistics lecture by Patrick Breheny illustrating the differences that result from your choice of confidence level and how they are intuitively very simple, if one takes the time to think about them…
Now, one of these graphs shows a 95% confidence interval, and the other shows an 80% confidence interval. If you think about just the values, you would (wrongly) assume that an 80% confidence interval is “worse” than a 95% confidence interval because 80 is less than 95. However, the definition of a confidence interval is that there is a X% chance that your interval contains the population mean. So, in order for you to be more sure that your interval will contain the true population value, you must widen the interval. Therefore, a 95% confidence interval is actually larger than an 80% confidence interval, but you are more confident that it contains the true population mean. Understanding this somewhat simple but very important concept is essential to generate and interpret scientific data. This course has illustrated this concept and the importance of statistics very well and I will make sure to keep this in the back of my mind throughout my career.