Boost scientific data reproducibility with quality standards, honor code

There is indeed a problem in the realm of scientific research when it comes to replicating data, as we learn from an article published by The Economist: Trouble at the Lab. The article discusses several phenomena that contribute to this problem, including bad statistics on the part of the scientists, poor research methodologies, peer reviewing that is inadequate, and lack of access to researchers’ methodological data and software. As a result, scientific research finds vastly challenged its reducibility, a quality which is a pillar of its ascribed objectivity.

There is a striking complementarity between this article and the assigned TED talk by Dan Ariely: Our Buggy Moral Code. Ariely discusses factors that encourage or discourage cheating in people. He discusses that people indeed do choose to cheat, and they choose to do so only a little. Additionally, one of his main points was that when reminded of morality people tend to cheat less. Let us replace Ariely’s “cheating” with the article’s phenomena that contribute to irreproducibility in research. If we do this, “cheating” is essentially not following proper standards of scientific research. From this perspective, we may posit that scientists at least aren’t too improper with their research methodologies, so there must be hope for us! As for reminding scientists of morality, the NIH could create an honor code for scientists.

That may sound funny, but at the same time I envision this: that a future generation will not only have an honor code for scientists but also an enforced formal quality standards for reporting scientific data. Firstly, that future generation with a regulated approach to reporting data will look back to ours and find it absurd that we did not have an enforced formal quality of standards. And I would not blame them, based on data such as that in the article being discussed which indicates that there is a prevalent degree of improper methodology when it comes to reporting data. I conceive that the same generation will look back at our lack of honor code with similar estrangement, for if they were taught to respect a set of morals in research, it would generally move them toward abiding by those morals, shifting the scientific culture. This would likely improve scientists’ desires to implement more proper methodologies in reporting their data, despite any cost to themselves since carrying such costs would not be taboo. Rather, it would be widespread and done in the cause of preserving the integrity of science’s reproducibility and thus objectivity. Oh, and the NIH would make sure these scientists don’t go out of business.

Different types of unreliable research call for different solutions

Our readings this week looked at multiple facets of unreliable research: Deception/cheating, unreplicability, and un-generalizability.  

 

Dan Ariely discusses cheating in the context of Wall Street, but it is easy to see how the behaviors he teases out in a laboratory setting with undergraduates could be relevant for data faking in science as well. Ariely discusses the finding that the propensity to cheat increases dramatically when one believes members of one’s “in-group” (in this case, Carnegie Melon students who see a student with a Carnegie Melon hoodie cheating). Ariely also discusses how under most circumstances, people would only cheat a little bit. He described this as a “personal fudge factor”, related to an individual’s desire to see his or herself as a good, moral person. You can easily imagine how this might translate into a research setting; the belief that your idea is correct (even if the data don’t support that) could lead one to believe that they are only cheating a little bit rather than a lot (which, to be clear, they are.)

Unreliable research can also result from from sloppy, improper use of statistics or poor data collection processes which find “significance” where there is none. An article for the Economist summarizes this problem quite well. This is in many ways the most frustrating area of unreliable research because it is the one that feels most correctable. We can disincentivize cheating, though realistically there will still be people who cheat. But rigorous and appropriate statistical analysis should be achievable for every published paper, if journals are willing to have every paper reviewed by a statistician (and realistically, every University is willing to foot the bill via increased paper submission fees).

Finally, un-generalizable research is research that was properly collected and analyzed, but is sensitive to small changes in experimental conditions. I would argue that these studies are the “features” of unreliable research, as discussed in Jared Horvath’s piece for Scientific American. While research that doesn’t generalize may be disappointing to those who thought they were studying a widespread phenomenon, this research still ultimately adds to our knowledge base.

These three types of unreliability are different in both their causes and solutions, and thus should not be lumped together. Cheating, or faking data, is completely unacceptable and should have serious professional consequences. Poor use of statistics can be corrected with education, and through structural mechanisms such as professional statisticians being employed by journals specifically to review the data analysis methodology of the paper. Un-generalizable research should be revealed through replications specifically designed to test the generalizability of the phenomenon: Using a different mouse line, a different age range, data collected at a different time of day. And unlike the previous two types of unreliable research, un-generalizable research shouldn’t reflect poorly on the researchers who conducted the experiments; instead it’s a familiar call back to the drawing board.

Selfish Behavior Can Limit Bias

We all like to cheat. We all should be cheating. Cheating is expected, because people should be selfish. The Dan Ariely TED Talk discusses the different scenarios in which cheating is more likely to occur. While he seeks different explanations rooted in societal behaviors, it boils down into a simple fact. Those who do not cheat see a benefit to themselves greater than what they would gain from cheating. Individual concern with your public persona are valued more than the physical money that is available to you. The degree to which people are prone to cheat depends on the risk to reward relationship of the scenario, we will always make the most selfish choice.

With selfishness in mind, I am not surprised that the majority of new “cures for cancer” are overhyped as described in "Half of the cancer drugs journalists called "miracles" and "cures" were not approved by the FDA". News outlets seek to boost their profile by giving the public what they want. Doctors promising cures are actually laughable, my interactions with doctors have demonstrated that they have a poor understanding of biochemistry and biophysics, and rarely understand the scope of science they would require to even begin to be viable sources of credibility. They promise hope because they can then feel good about themselves for potentially easing a patient's mind. Selfishness drives their motives, and pushes forward a bias to relay positive results.

Good scientists are trained to treat results with skepticism, to doubt everything, and to avoid excitement without mountains of supporting data. We are taught that we can only ever disprove a theory; we can never conclusively state something as proven. Despite this, we always want to present our data with a specific angle in mind. We want to cheat the system by showing the data we choose, because then we can communicate our story and our beliefs. We selfishly push our own agendas and seek the greatest impact for our work, so that we then become more important.

So who ends up at fault for cancer drugs being biased towards miracle cures? The only people at fault are those who believe it. We are responsible for our own conclusions, and for adhering to the most selfish practices possible. If the public holds everyone to a higher standard, perhaps bias can then be reduced.