Take that Jenny McCarthy! Statistical Tests Show Improvement in Vaccination Completeness

It hasn’t been too long since celebrity Jenny McCarthy let it be known that she is vehemently opposed to our current vaccines. She was one of the first celebrities to support the pseudoscientific view that vaccinations cause autism, and recently she has made it clear that she thinks anyone carrying a virus is deathly “sick,” in her comments made about former co-star Charlie Sheen, who has HIV. Uh, that’s not exactly how the fields of virology and immunology have deduced the process, Jenny.

Nonetheless, public health officials will soldier on because they recognize the benefits of vaccinating people, especially children, and that such vaccination prevents sickness even in the case of contracting the virus. One question I’ve always had as a bench scientist is how is it that public health officials know they’re doing their job efficiently? I see many of my friends going to public health school wanting to help with the education arm of public health issues. How do we know if the methods they use are effective? What is a quantifiable measure for us to obtain a level of effectiveness?

Figure 1. An example of paired participant studies. In the case of the SUNY Upstate article I looked at the comparison group would be a group of similar age and income in an adjacent community compared with a group of interest receiving the intervention

Dually, I’ve enjoyed spending the semester reading about statistical tests we haven’t gone over in class. One of those tests is known as the McNemar’s test. A quick interwebs definition says McNemar’s test is a “statistical test on paired nominal data,” or basically assigning a binomial outcome to paired data (see Figure 1 for paired data example). When I first read about this, I thought of vaccines. A good signal to public health educators that their programs are working are whether populations are vaccinated or not, specifically communities that face traditional barriers to quality healthcare.

In a community health paper published in 2013, public health professionals at SUNY Upstate tested their hypothetical vaccination intervention, which involved partnering with community organizations such as the Salvation Army, allowing patients a Q&A session prior to vaccination, and connect to vaccination specialists through community liasions. The authors of the study paired their subjects based on age and household income across 10 different community sites, separating them by intervention positive or intervention negative status, and measuring proof of influenza vaccination in the presence or absence of the intervention. They wanted to compare if the intervention had successfully raised the vaccination levels across age cohorts and overall. The group then used McNemar’s test to construct their 95% confidence intervals to illustrate the nearly 17% increase (95% CI 15.5-19.5) in influenza vaccination levels (see Figure 2) to compared to state and county level alternative interventions. Impressive! Although the authors don’t report a p-value, with the right null hypothesis, McNemar can calculate one for you. It’s so handy.

Figure 2. The contigency table used to calculate McNemar's test. As shown in Figure 3, McNemar's test relies on reporting those not receiving vaccinations but are enrolled (not explicitly stated in the chart).

One limitation to McNemar’s test is that it’s meant for large groups. However, based on the population scope of public health data, this doesn’t seem to be an issue – in fact it is an advantage for novice public health professionals to know this fact, especially if they’ve never done statistical analysis.   

Figure 3. A screen grab of the McNemar test calculator found on GraphPad. Motulsky recommends readers use this for calculation of confidence intervals and p-values in his book. 

 All this time, I thought public health professionals went off magnitudes of numbers alone, perhaps testing averages across populations in an ANOVA test. As it turns out, they use statistical tests, specifically McNemar’s test when employing tired-and-true case-control designs.