Truth and Reproducibility Not Always the Same Thing

The concept of reproducibility of experimental findings is important in research, and it is certainly given great weight in several of the assigned articles. One line in particular in the article “Trouble at the lab” struck me as particularly interesting.

The idea that the same experiments always get the same results, no matter who performs them, is one of the cornerstones of science’s claim to objective truth. If a systematic campaign of replication does not lead to the same results, then either the original research is flawed (as the replicators claim) or the replications are (as many of the original researchers on priming contend).

While this absolute ideal of reproducibility may be true for much of the research in some disciplines and for extremely fundamental processes in all fields, this statement is an oversimplification, particularly in the life sciences. Most physiological processes, particularly more complex ones that involve input from multiple different systems, can be influenced by a host of variables that may be difficult to fully control. This can mean that efforts to replicate published experiments are not always successful. For instance, as many of us have likely experienced, findings can change dramatically between even genetically identical mice in different facilities. 

Such observations of irreproducibility could be viewed as evidence of sloppiness or even deliberate deception, as many of these articles seem to suggest, and certainly irreproducible results that stem from irresponsibility in research are reprehensible. However, irreproducibility in well-conducted studies can call attention to new biological phenomena – such as the powerful influence of the microbiota in the mouse example above – that had not been appreciated previously. Different findings do not have to indicate that anyone is wrong. 

So how would one actually test reproducibility? It could be argued that someone unrelated to the original investigation should be able to use the same methods in the same location using the same materials and produce the same results. However, anyone who has done behavioral testing knows that changing the experimenter can change the results obtained in a study. I think that we have to be more discerning in our consideration of replicability. If a finding is reported that should be broadly applicable, and multiple attempts by other groups cannot duplicate it at all, then it can be considered that the original study might have flaws or perhaps that the interpretation of the findings was incorrect. But how concerned should we as scientists be if some experiments do not always produce the same results? We can lament the sorry state of research and start assigning blame, as some of these articles do, or, once we have verified the rigor of our methodology, we can expand our thinking and consider what unknowns in the system might be preventing it from fitting into our own limited understanding.