Let’s say
you’ve invented a weight loss pill and you’re ready to show the world what that
this pill is the real deal. You start to
think up some experiments that will showcase your work with human patient
trials and you’re deciding between unpaired or paired designs. Which should you choose? Historically, in these situations, the
majority of experimenters would probably lean towards a paired design. The ability to detect an effect of treatment
within an individual over time or in a new context has usually been considered a
“more impressive” result. It makes for a
better story when you’re presenting the data.
Luckily, it’s hard to convince an audience that a study uses a paired
design when that’s untrue, however sometimes experimenters might miss the fact
that their own data is paired and opt for unpaired tests with confusing
results.
Figure 1: Unpaired
comparison in ER chaperone activity with and without sevoflurane.
In the paper “The Effect of
Endoplasmic Reticulum Stress on Neurotoxicity Caused by Inhaled Anesthetics”,
Komita et al. investigate a possible calcium-dependent mechanism for neuronal
death following exposure to sevoflurane anesthetic. Sevoflurane exposure causes calcium to be
released from the endoplasmic reticulum (ER) of neurons into the cytosol. The rapid change in calcium dynamics in the
cell is thought to cause ER stress which may lead to apoptosis and underlie the
neurotoxic effects that have been reported after anesthetic exposure. To test their hypotheses about the effects of
inhaled anesthetics, they chose to expose cultured neurons to sevoflurane while
monitoring markers of ER function. To
analyze this data, the authors used immunofluorescence microscopy to compare ER
chaperone levels with inert gas and sevoflurane gas. Given that these experiments are done in cell
culture, the authors could have chosen to use a paired design, but instead this
test is run as an unpaired design (figure 1).
This may seem
to be a simple, inconsequential mistake in comparison to the paper’s
findings. The effect is rather large and
would have been noticed given either test, but the authors go on to make
further mistakes regarding the treatment of their data. The worst being their improper analysis of
the data in Figure 2 as a 1-way ANOVA instead of a 2-way analysis (figure 2).
For the following figure, different genotypes of rats had their ER
chaperone response tested in both control and sevoflurane conditions. The study design is clearly deserving of a
2-way ANOVA since comparisons are being made between control and sevoflurane
and all the genotypes. However, the
experimenters instead decide to treat all of these groups as unpaired and
unrelated. It’s unclear whether they
ignored the difference in the genotype or treatment in order to justify
analysis with one-way ANOVA, but they do it anyway. It’s improper experimental design here, and
leads to confusing results that seemed to be cherry-picked.
Komita M,
Jin H, Aoe T. (2013). “The Effect of Endoplasmic Reticulum Stress on
Neurotoxicity Caused by Inhaled Anesthetics”. 117(5): 1197-1204.
You say the effect is very large, how would the results have changed with the correct tests? Would that have seen even better results, or would they have had lower significance?
ReplyDeleteThe three versions of Student's t-test were designed for different scenarios. Using independent Student's t-test to test paired data brings us less power than the paired t-test. Similarly, using independent t-test without equal variance to test data with equal variance decreases the power as well. When the sample size is fixed, less power implies harder to get significant result.
ReplyDeleteConversely, using t-test with equal variance to test data with unequal variance increases the false-positive rate. That means we are easier to get significant result.