The Necessity of Reproducibility and Two Challenges to the Scientific Research

        The Merrian-Webster Online Dictionary defines ‘research’ as ‘the investigation or experimentation aimed at the discovery and interpretation of facts, revision of accepted theories or laws in the light of new facts, or practical application of such new or revised theories or laws’. As such, we, the scientists, usually cannot predict what the results will be (because they are probably not seen before), neither can we know if the observations reflect the facts. Therefore, logic and methodology form the cornerstones of scientific inferences. Ideally, once we follow the well-established methodology to design study, and we interpret and present the observations in a logic manner, we can claim and convince others that the results reflect the reality.

        In the modern scientific society, people starts to question if this paradigm works. The first challenge comes from the validity of methodology, which were discussed in Pannucci, C. J. and E. G. Wilkins (2010) and ‘Trouble at the Lab’ from The Economists (2013). Briefly, two types of error may be introduced to a study. The systematic errors (biases), including selection bias, information bias, and confounding, dampen the validity of a study and cannot be alleviated by increasing the sample size. In contrast, the random error compromises the precision of the measure of association, and it can be controlled by large sample size and appropriate statistical methods. The validity of methodology itself is one kind of research, which will be improved as it goes on.

        The second challenge is more about the research ethics, discussed in a TED talk from Dan Ariely (2012), Julia Belluz (Sep 2015), and Julia Belluz (Oct 2015). Science itself is simply the pursuit of knowledge about the nature or the human society, but scientific research involves politics — the pursuit of benefit, and the competition of resource. Working in such intense publish-or-perish culture, scientific misconduct or violation of regulation may be used to get seemly ‘astonishing’ results, which would earn the investigators much more funding — temporarily. We should have learned lessons from the scandal of unethical source of oocyte (W. S. Hwang, 2005), the STAP stem cell fiasco (O. Haruko, 2014), and the ‘peer review and citation ring’ (P. Chen, 2014), that the mechanism of peer review is not complete enough to prevent some researches from being overvalued. Besides the improvement of ethical education, PubPeer proposed a new paradigm of peer review — the peer review that involves all members in the same filed. Although currently this mechanism works after the research is published, it remains a good and worth trying.

        Perhaps the most essential and informative debate is the reproducibility. In the article of Jeremy Berg (2013), he stated the low reproducibility in modern scientific researches is due to the low prevalence of ‘correct hypothesis’. While I agree with his analysis in general, I want to point that first, the percentage of correct hypothesis can be increased by carefully designing and interpreting the pioneer studies. Second, the specificity can be improved by replicating the experiments by other lab members or even other labs. A significant result occurring by chance would not be reproduced, a phenomenon called ‘regression to the mean’.

        Above all, I still argue that the reproducibility is an essential part of scientific research. Especially for the natural science, the necessity of reproducibility is embedded in the mechanism of knowledge formation and application. What we should work on is keep improving scientific and statistical methods to avoid biases, and educate the scientific society on research ethics.