Reading through the chapters of Motulsky’s Intuitive Biostatistics, my attention was caught by the section on meta-analyses in Chapter 43. Given the problem of reproducibility in scientific research, the meta-analysis seems to be one of the great tools for addressing the problem of reproducibility; confirming or refuting prevailing scientific theories and advancing humanity’s body of scientific knowledge.
Looking at the “Assumptions of Meta-Analysis”, I was intrigued by the idea that meta-analyses fall into two general categories, each based on a different assumption. Either
(A) All subjects are sampled from one large population; each scientific study is estimating the same effect. Measurement error comes from random selection of subjects, or
(B) Each study population is unique, and the differences in population and random selection of subjects both contribute to the error.
Sound familiar? To me, these two assumptions seemed derived from the two philosophies of scientific realism we discussed at the beginning of class. One being that the truth (the large population) exists somewhere, and that science’s job is to uncover it, the other being that unobservable truth is irrelevant, and that utility of knowledge is paramount as is relates to advancement of medicine or technology. The idea that a population’s true response to a therapeutic exists is reflected in assumption (A), whereas (B) describes the anti-realist philosophy that there is no “global” population response, only individual subset responses as described in each sub-study of the meta-analysis.
The value of the meta-analysis under the framework of (B), then, would be to predict the efficacy of a given therapeutic in the next population of patients, given all those who have been tested before. Motulsky goes on to described this second model as the more commonly used one underlying most meta-analyses. The anti-realist philosophy is commonly associated with being applied and utilitarian, though I’m wondering if there’s a fundamental application of the anti-realism paradigm. Specifically, what does the idea that each sub-population in a meta-analysis is inherently disconnected from the others mean for drawing scientific conclusions from a meta-analysis? Is there a connection to the scientific philosophies of confirmation and falsification inherent in the above assumptions? Do the answers to these questions even affect the conclusions we can draw from meta-analyses, or are they irrelevant exercises in navel-gazing?