Irreproducible complexity

A recent article in The Economist explored the "replication crisis" in scientific research and how many recent papers have pointed out how research often cannot be reproduced by outside groups, and in many cases, even by the original labs. They then extrapolate this "crisis" as being a huge blow to the validity of science as a means of assessing claims. I find this to be a bit of an overreach from what I think many would agree is a real issue in the community.  

Yes, we are all taught as young scientists that replication is a key part of scientific exploration. Any result I obtain in my lab should be able to be replicated within my lab and by other researchers. While this is a key part of science, it is just that, a part of science. A single experimental design isn't definitive support of a hypothesis by itself, it should be part of a larger body of experiments all designed to assess the plausibility of given hypothesis by probing the idea with different approaches and looking for results that help to direct us toward accepting or rejecting a hypothesis. While it is an issue if only one lab is able to get a given result, or worse yet, that they cannot consistently obtain the same result, the scientific community shouldn't be taking this one experiment/paper as an absolute validation of a given claim. Science advances through a winnowing process whereby the next set of experiments inspired by a given claim will either further or weaken a given hypothesis. 

As a young scientist I tend think of several things when I read a new paper making a novel claim:

1 - How does this paper fit in with the larger body of research, is it incremental progress from previous work, or is it a massive shift in understanding? To quote Carl Sagan, "Extraordinary claims require extraordinary evidence."

2 - Bigger sample sizes tend to be better.  While not foolproof, if all else is equal, a bigger sample is probably better than a small one. An example of this that comes to mind is the now retracted Andrew Wakefield paper on the Measles, Mumps, and Rubella vaccine and its link to autism. Among it's many issues, they drew conclusions from an n=12.

3 - What are the effect sizes? In my own work I am quite wary of chasing small effects, and we should be similarly wary of those in the work of others, even if it gets into a major journal.

4 - This is one I hope to improve upon in this class, are the statistics used valid to support the claims and limit bias? Does a statistically significant result mean that a given result is of biological significance?

Is a cure a problem?

Is a cure a problem?
          Vox media published an article titled “Half of the cancer drugs journalists called “miracles”and “cures” were not approved by the FDA,” in October 2015 stating that the use of attractive adjectives in drug advertisements is biasing the consumer by marketing drugs that are not available in clinic. Belluz, author of the article, states this “overselling” of a drug is creating false hope for patients as well as, helping to form misguided policies. While the claims are not false, this form of bias is not harmful but hopeful.

Would we get the average person to read science articles about the latest drug discoveries if these adjectives were not included in the title? Probably not. If adding a word to the title biases a person towards reading a piece of science it may be a necessary evil for the public. Also, if policies are being formed by this public, why do these policies have to be misguided? Are the titles so profound that they neglect the rest of the article? Are they truly so biased by the title that they don’t feel it necessary to make an informed and educated opinion? The argument that these words are misguiding policy makers is poor without proof of what these policy makers read prior to decision making.

Break-through, ground breaking, and cure may be deceptive words but they are also providing optimism for patients and caregivers. Optimism bias clouds our perception of reality. This optimism bias can provide comfort to the patient without options. Maybe these drugs being described aren’t the real cure for cancer but, they are inherently providing hope. Belluz states this hope is bad but hope is one of the necessary ingredients for fighting cancer. It is why there are colored ribbons, fundraising events, and millions of dollars flowing into cancer research. Cancer research is being done and that means the cure is closer. The adjectives describing these discoveries are motivating lay people to spend their time learning and donating to a worthy cause. A biased consumer is a hopeful consumer in the profession of drug discovery.

Connecting mutations to disease

While meeting with a visiting speaker last week, the subject came up of studying biological pathways that only account for <1% of a given disease. For example, schizophrenia is a very heterogeneous disease with multiple proposed risk factors. Genome-wide association studies have been conducted to identify specific risk factors. Although these studies have been successful in identifying candidate molecules, such as microRNA-137 and then gene C4, how helpful can they be if they only contribute to a small percentage of schizophrenia cases? This made me think of a question that keeps coming up in this course: when is clinical significance more important than statistical significance? Though the studies have statistically significant results, how significant are they to the health field if they cannot help patients with schizophrenia?
I believe this problem somewhat originates from the drive to get funding from the NIH. When applying for grants, your project becomes much more appealing when a disease is attached to it. Thus, a protein that a large group of the scientific community didn't care about before becomes much more appealing when it's tied to some disease, no matter how small the percentage of risk is. As soon as you have a study showing a statistically significant result for a pathway implicated in a disease, your research gets some attention. But how much does that matter in the scheme of the actual disease when the mutation you are observing accounts for <5% of a disease?
While this may not be considered a form of "bias", I believe that attempting to put a health-related spin on research is severely affecting the way that we conduct research. Making overreaching conclusions can mislead readers and misdirect future research projects. These studies may have the right numbers and statistics, but do they have the right scientific approach of questioning?