Irreproducible complexity

A recent article in The Economist explored the "replication crisis" in scientific research and how many recent papers have pointed out how research often cannot be reproduced by outside groups, and in many cases, even by the original labs. They then extrapolate this "crisis" as being a huge blow to the validity of science as a means of assessing claims. I find this to be a bit of an overreach from what I think many would agree is a real issue in the community.  

Yes, we are all taught as young scientists that replication is a key part of scientific exploration. Any result I obtain in my lab should be able to be replicated within my lab and by other researchers. While this is a key part of science, it is just that, a part of science. A single experimental design isn't definitive support of a hypothesis by itself, it should be part of a larger body of experiments all designed to assess the plausibility of given hypothesis by probing the idea with different approaches and looking for results that help to direct us toward accepting or rejecting a hypothesis. While it is an issue if only one lab is able to get a given result, or worse yet, that they cannot consistently obtain the same result, the scientific community shouldn't be taking this one experiment/paper as an absolute validation of a given claim. Science advances through a winnowing process whereby the next set of experiments inspired by a given claim will either further or weaken a given hypothesis. 

As a young scientist I tend think of several things when I read a new paper making a novel claim:

1 - How does this paper fit in with the larger body of research, is it incremental progress from previous work, or is it a massive shift in understanding? To quote Carl Sagan, "Extraordinary claims require extraordinary evidence."

2 - Bigger sample sizes tend to be better.  While not foolproof, if all else is equal, a bigger sample is probably better than a small one. An example of this that comes to mind is the now retracted Andrew Wakefield paper on the Measles, Mumps, and Rubella vaccine and its link to autism. Among it's many issues, they drew conclusions from an n=12.

3 - What are the effect sizes? In my own work I am quite wary of chasing small effects, and we should be similarly wary of those in the work of others, even if it gets into a major journal.

4 - This is one I hope to improve upon in this class, are the statistics used valid to support the claims and limit bias? Does a statistically significant result mean that a given result is of biological significance?

Frauds, scams, cheats in science. What will it take to overcome it?

As I’ve gotten into my research project, I’ve begun to notice the lack of accountability that presides within the scientific field. Research projects typically have one or a few people working on them, and the temptation to make data look better is high. As evidenced by the many papers retracted each publishing cycle, there is major fraud going on within research, and I think it stems from a lack of accountability and responsibility felt by those carrying out fraudulent practices. It’s not a crazy thing to say, as well, that none of us are completely immune to it. As Alex Chen enlighted us to the fraud that occurred at Duke University, it is perhaps too easy to manipulate data, or worse, invent subjects and data that never even existed.

In 2015, New York University bioethicist Arthur Caplan said, “The currency of science is fragile, and allowing counterfeiters, fraudsters, bunko artists, scammers, and cheats to continue to operate with abandon in the publishing realm is unacceptable.” Fraud in science is on the rise, and not only is it a threat to those individuals’ careers, but it is a threat to all research. Science is designed to enlighten, provide methods to answer sought out questions, and in the end to save lives, and when fraud enters into the science, none of that can occur.  Probably one of the main arguments for those that have falsified data is the pressure to obtain funding and publish in a funding environment that has become very strict over the past few years. What they don’t see is how big of a threat fraudulent data is to public funding. The public doesn’t like being lied to, especially when it has to do with potential medical breakthroughs and their own health, and as fraud continues to come in front of the public eye, funding will most likely decrease. Trust from the public is necessary for funding, and maintaining honesty and integrity in research is necessary for that trust.  

An added complexity to this already complex mess of fraudulent research is the idea that it might not be the wanted success and fame that pushes them to be fraudulent, but it could be the necessity to put food on the table for their family.  Not only are scientists striving to provide meaningful research to further medicine (hopefully), but they are also striving to feed their families. Would having a system in which accountability and responsibility were incumbent on the researchers (by some type of education or university program) decrease the temptation to falsify results? Would having a system in which the publish or perish mentality was somehow alleviated, and there was security of an income, decrease the temptation to falsify results? Would journals that began publishing negative, nonsignificant results with repeated experiments from other papers help decrease the temptation to falsify results? And can such a system exist, let alone be put in place?

I believe it is necessary for the sustainability of scientific research to report data correctly. This might only happen if there is some sense of accountability and responsibility. Scientists are humans. We need to eat. But we are expected, like most other humans are expected, to be upright, have integrity, and not lie. Is there a way we can do good science, remain upright, and eat?