Beating a Dead Horse: Reproducibility and Rigor Mortis

Frankly, I think there has been a lot of undue panic concerning the credibility and reproducibility of modern scientific findings. With the rare exception of unethical/highly stressed-out researchers being guilty of falsifying data, everyone I know working in scientific research is doing the best work they possibly can with the tools and funding they have available. Nowadays, scientific methodologies have become so sensitive, specific, and technically advanced that it really shouldn't shock anybody when deep RNA sequencing data from a triple-transgenic mouse model of autism differs between two laboratories on other sides of the country; the quantity and quality of data being harvested here is literally unprecedented, meaning that “significant” differences may emerge in “identical populations” based on confounding variables that we are not even aware of or simply cannot control. Because the data could not be reproduced, we think it must be rejected out of hand. But that's stupid, and also dangerously corrosive to the credibility of science in the eyes of the public. It worries me that scientists put so much stock in the homogeneity of “similar populations of clinical subjects” or “identical strains of inbred mice” or “early passages of the same cell line” and so little stock in the internal consistency of reagents, protocols, “hands,” laboratory conditions, etc. Very “robust” effects are typically borne out in many models, across many experiments, using many techniques, in many laboratories. But what about subtle or trending effects? What if a drug could work for some small subset of a supposedly “homogenous” clinical population but not for the majority? What if the supposedly “identical” samples of the same mouse strain were more stressed on their journey to California than they were on their journey to New York, causing irreversible but essentially “spurious” differences in the results generated by two laboratories in two states? Part of being a productive and appropriately skeptical scientist is knowing which variables can be controlled and which ones cannot. No experiment is perfect, and a failure to reproduce expected results can lead to important exceptions to rules, or clinical breakthroughs, or major paradigm shifts....