Take that Jenny McCarthy! Statistical Tests Show Improvement in Vaccination Completeness

It hasn’t been too long since celebrity Jenny McCarthy let it be known that she is vehemently opposed to our current vaccines. She was one of the first celebrities to support the pseudoscientific view that vaccinations cause autism, and recently she has made it clear that she thinks anyone carrying a virus is deathly “sick,” in her comments made about former co-star Charlie Sheen, who has HIV. Uh, that’s not exactly how the fields of virology and immunology have deduced the process, Jenny.

Nonetheless, public health officials will soldier on because they recognize the benefits of vaccinating people, especially children, and that such vaccination prevents sickness even in the case of contracting the virus. One question I’ve always had as a bench scientist is how is it that public health officials know they’re doing their job efficiently? I see many of my friends going to public health school wanting to help with the education arm of public health issues. How do we know if the methods they use are effective? What is a quantifiable measure for us to obtain a level of effectiveness?

Figure 1. An example of paired participant studies. In the case of the SUNY Upstate article I looked at the comparison group would be a group of similar age and income in an adjacent community compared with a group of interest receiving the intervention

Dually, I’ve enjoyed spending the semester reading about statistical tests we haven’t gone over in class. One of those tests is known as the McNemar’s test. A quick interwebs definition says McNemar’s test is a “statistical test on paired nominal data,” or basically assigning a binomial outcome to paired data (see Figure 1 for paired data example). When I first read about this, I thought of vaccines. A good signal to public health educators that their programs are working are whether populations are vaccinated or not, specifically communities that face traditional barriers to quality healthcare.

In a community health paper published in 2013, public health professionals at SUNY Upstate tested their hypothetical vaccination intervention, which involved partnering with community organizations such as the Salvation Army, allowing patients a Q&A session prior to vaccination, and connect to vaccination specialists through community liasions. The authors of the study paired their subjects based on age and household income across 10 different community sites, separating them by intervention positive or intervention negative status, and measuring proof of influenza vaccination in the presence or absence of the intervention. They wanted to compare if the intervention had successfully raised the vaccination levels across age cohorts and overall. The group then used McNemar’s test to construct their 95% confidence intervals to illustrate the nearly 17% increase (95% CI 15.5-19.5) in influenza vaccination levels (see Figure 2) to compared to state and county level alternative interventions. Impressive! Although the authors don’t report a p-value, with the right null hypothesis, McNemar can calculate one for you. It’s so handy.

Figure 2. The contigency table used to calculate McNemar's test. As shown in Figure 3, McNemar's test relies on reporting those not receiving vaccinations but are enrolled (not explicitly stated in the chart).

One limitation to McNemar’s test is that it’s meant for large groups. However, based on the population scope of public health data, this doesn’t seem to be an issue – in fact it is an advantage for novice public health professionals to know this fact, especially if they’ve never done statistical analysis.   

Figure 3. A screen grab of the McNemar test calculator found on GraphPad. Motulsky recommends readers use this for calculation of confidence intervals and p-values in his book. 

 All this time, I thought public health professionals went off magnitudes of numbers alone, perhaps testing averages across populations in an ANOVA test. As it turns out, they use statistical tests, specifically McNemar’s test when employing tired-and-true case-control designs.

Optimizing the BOT

Prior to taking this class, I had a lengthy conversation with my PI about statistics.

We debated over what statistical methods were appropriate to use for our experiments. She opted for the classic t test and I opted for anything but that.

During this debate she would often throw out statements like,

“We don’t base our conclusions solely on whether or not something is significant.”

“We should be able to tell if a result is significant or not just by looking at the data.”

“We can’t publish without stats.”

“Even if a result is significant, it doesn’t matter if it doesn’t have any biological relevance.”

Looking back on this debate now, I realize my PI was/is a follower of BOT.

BOT standing for the “Bloody Obvious Test” coined back in 1987 by Ian Kitchen. Kitchen noted that there was pressure from journals to use statistics and that p-hacking was a problem. 

                                                “but it does seem that too often we

labour over their (statistics) use unnecessarily

and indeed on other occasions we

manipulate them to prove a very

thin point.” –Ian Kitchen

Because of these issues, Kitchen proposed the use of the “Bloody Obvious Test. ”

The protocol for the BOT is as follows:

                Question #1: “Is it bloody obvious that the values are different?”

                  Answer: Yes.  The test is positive, proceed to “Go” and collect $200.

                  Answer: No.   Proceed to question number 2.

                Question #2: “Am I making a mountain out of a molehill?”

Kitchen really wanted to drive home the point that statistics were being abused to appease “the gods of statistics” who happened to frequently sit on journal review boards. He wanted to remind scientists that sometimes the easiest and most obvious answer is the right answer. Lastly, he wanted scientists to recognize that statistical significance doesn’t always equal scientific significance.

Sadly, Kitchen didn’t stop these issues from persisting in science today. Scientists are still appeasing “the gods of statistics” because to be successful in science, you have to publish.

As the reality of science publishing seems unlikely to change and the pressure to include stats continues, I propose we optimize the BOT with confidence intervals.

Confidence intervals are a form of statistics that provides a range in which the true population value may lie. Traditionally, we set confidence intervals at 95%. A 95% confidence interval tells us that there is a 95% percent chance that confidence interval contains the true population parameter of interest.

The addition of CIs would add a statistical robustness to the BOT, that would perhaps appease “the gods of statistics.” Also, the addition of confidence intervals wouldn’t detract from the initial step of the BOT. We could still ask Question #1 without a pesky p-value getting in the way of our conclusion. Instead, confidence intervals would be to the BOT “as a drunk uses a lamp-post; for support rather than illumination.”

Fitting the right model to data

When it comes to fitting models in data, we need to be careful avoiding fancy mistakes. The regression functions can get pretty complicated, which matches any of our wish for letting the data be more explanatory. At this point, we could over-interpret the data, and neglect that we can replace the models with other analysis that make more sense in our research context.

I just finished my honor thesis and I had to persuade myself for not playing around what I have learn from this advanced statistic course for an undergraduate student. The research that I worked on is about whether a plant, GBL, can inhibit growth of a bacteria, ATCC6919. We are interested in this bioactivity because it can be an alternative cure to infection caused by this bacterium.

Extractions of this plant were made from 3 different parts of the plant, leaves, branches, and seed, and two different extract solvent was used: ethanol and water. The question that my data analysis need to answer is not only whether the GBL extracts is active against ATCC6919 growth (% inhibition> 50%), but also whether the tree parts and the extract solvent contribute to effectiveness of the extracts.

The ATCC6919 culture was treated with GBL extracts at a range of concentration. So the result of the antibacterial investigation will generate many dose response curves, like the one shown below. Since the trend of the plot is clear that at the percentage inhibition is higher at higher extract dose, there is a pretty good chance that we can find a regression model which fit most of the data well. I could build a “dose response regression model” for the extracts. However, I recalled that we were interested in finding the extracts that were active (% inhibition > 50%). Therefore, a regression model could be a statistically perfect fit, but it is scientifically non-sense. I have to discard the idea of nonlinear regression curve model.

Then I thought about, could I compare the difference in inhibition result from the two extraction solvents by comparing the fit of the data to two models. The best-fit slope of the regression line should be the differences between two group means. Thus, I set the variable defines extraction method X, and assigned X=1 arbitrarily to aqueous extracts and X=2 arbitrarily to ethanolic extracts. Y axis was the percentage inhibition of the extracts at same concentration. It would look like the linear regression graph shown below. However, if so, I neglected the other factor, which is the tree parts, which can also contribute to the difference in inhibition. I could meet a problem opposite to over-fitting the data, which is over-simplify it.

If we replace the regression models for two-way ANOVA, it is easier to see whether plant parts, or the extraction method, or the interaction of them make inhibition of the extracts differ. If you want it to be more basic, multiple student t-tests would work together, too. 

To sum up, when we try to fit the models to data, before thinking about which certain type of regression model fit better, check if other (and simpler) method fits more.