Let’s say you’ve invented a weight loss pill and you’re ready to show the world what that this pill is the real deal. You start to think up some experiments that will showcase your work with human patient trials and you’re deciding between unpaired or paired designs. Which should you choose? Historically, in these situations, the majority of experimenters would probably lean towards a paired design. The ability to detect an effect of treatment within an individual over time or in a new context has usually been considered a “more impressive” result. It makes for a better story when you’re presenting the data. Luckily, it’s hard to convince an audience that a study uses a paired design when that’s untrue, however sometimes experimenters might miss the fact that their own data is paired and opt for unpaired tests with confusing results.
Figure 1: Unpaired comparison in ER chaperone activity with and without sevoflurane.
In the paper “The Effect of Endoplasmic Reticulum Stress on Neurotoxicity Caused by Inhaled Anesthetics”, Komita et al. investigate a possible calcium-dependent mechanism for neuronal death following exposure to sevoflurane anesthetic. Sevoflurane exposure causes calcium to be released from the endoplasmic reticulum (ER) of neurons into the cytosol. The rapid change in calcium dynamics in the cell is thought to cause ER stress which may lead to apoptosis and underlie the neurotoxic effects that have been reported after anesthetic exposure. To test their hypotheses about the effects of inhaled anesthetics, they chose to expose cultured neurons to sevoflurane while monitoring markers of ER function. To analyze this data, the authors used immunofluorescence microscopy to compare ER chaperone levels with inert gas and sevoflurane gas. Given that these experiments are done in cell culture, the authors could have chosen to use a paired design, but instead this test is run as an unpaired design (figure 1).
This may seem to be a simple, inconsequential mistake in comparison to the paper’s findings. The effect is rather large and would have been noticed given either test, but the authors go on to make further mistakes regarding the treatment of their data. The worst being their improper analysis of the data in Figure 2 as a 1-way ANOVA instead of a 2-way analysis (figure 2). For the following figure, different genotypes of rats had their ER chaperone response tested in both control and sevoflurane conditions. The study design is clearly deserving of a 2-way ANOVA since comparisons are being made between control and sevoflurane and all the genotypes. However, the experimenters instead decide to treat all of these groups as unpaired and unrelated. It’s unclear whether they ignored the difference in the genotype or treatment in order to justify analysis with one-way ANOVA, but they do it anyway. It’s improper experimental design here, and leads to confusing results that seemed to be cherry-picked.
Komita M, Jin H, Aoe T. (2013). “The Effect of Endoplasmic Reticulum Stress on Neurotoxicity Caused by Inhaled Anesthetics”. 117(5): 1197-1204.