Cancer: The Disease of Irreproducible Results?

The U.S. Government spends $5 billion every year on cancer research yet it is no secret that this return on investment has been quite disappointing. What lies at the crux of this attitude, is the growing mistrust that permeates lab findings - corrupt data due to sloppy analysis, unstable results, poor experimental design, and most of all, a replication crisis. When a cancer study spirals into a wrong conclusion, individuals suffer, a multibillion-dollar industry of treatment loses money, and biologists get jaded. This only propagates the cycle of researchers favoring efficiency in publication over the validity of results. As a testament to this, in 2011, a team from Bayer pharmaceuticals reported that only 20-25% of studies they attempted to reproduce generated results "completely in line" with those of original publications.

It's not only a problem of results, repeated findings, etc. It's also a problem of consistent methodology. We cannot solely focus on standardizing analyses of data outputs if a given experiment cannot even be performed in another lab. A comprehensive review of current research literature suggested that essential steps in a protocol are frequently omitted from published papers. Specifically a 2013 survey of several hundred journal articles that referenced >1,700 different laboratory materials revealed that only about 50% of such materials could be identified by reading the original papers.

Ultimately, this flawed cycle of irreproducibility has shaken the grounds of public trust for research science and the advances it offers society, however science's substantive progress is highly dependent on winning back this trust. The publication and dissemination of not only provocative yet imprecise studies, but also of findings that cannot be re-performed for whatever the reason, is only further contributing to the public's lack of confidence in the veracity of a branch of science that has the potential of being revolutionary for biomedicine. And we will never elucidate the inherent problems in such findings until we've figured out a way to make them reproducible.

Statistics helps explain challenges in cancer prevention research

Last Friday in Winship Cancer Institute auditorium, there was the speaker Dr. Yang talking about tea in cancer prevention. Below is one of his slides which illustrates that tea might help prevent skin, oral, esophagus, lung cancers etc..

It could be very wrong to say that tea definitely prevents these cancers in "potential" human patients based on what have been taught in biostats class.  Thankfully, Dr. Yang did not make that statement in his talk but instead listed findings or research that has been done across the globe to test ingredients in the tea that might help prevent cancers in lab settings or clinical trials.  Even at the very end of Professor's Yang's talk, he did not jump to the conclusion of cancer prevention effects of tea.  This might sound frustrating to young researchers who has the ambition to prevent cancers someday in the future.  But it is a good talk to me if I combine biostats concepts and what Dr. Yang talked about or what others in this field has been pursuing for many years.  

The first thing I learnt was to keep both scientific hypothesis and statistical hypothesis in mind before you actually perform an experiment or start a research project.  It's easy to get lost when you gradually learn more about your research subject but without a clear science question to answer.  For example, in cancer prevention research of a lab setting, if you want to test if ingredient A has the effects of lessing prostate tumor burden in mouse model, you should stick to it even later in research you found that ingredient A somehow magically decreased the mouse weight and may have effects in preventing obesity.  Then you suddenly changed your hypothesis in order to get your research published as soon as possible.  This is the so-called HARKing (hypothesizing after results are known).  In this way, there is greater possibility that you are biased to jump to the false positive results without careful statistical decisions beforehand.  

Also, it's also critical to keep effect size in mind rather than just rely on p-value when you want to extrapolate a lab finding to the clinical trials.  Nowadays, we've seen so many failures in clinical trials where the drug used has been shown to have "significant" effects in lab research.  Part of the reason is that we consider p value to have magic power to draw the conclusion but ignoring the effect size, especially in the case of cancer prevention.  We've seen that 8 out of 10 mice in our lab setting responded to the drug and showed a delay in tumor formation (treated group showed five days delay of developing the same size of tumor).  Statistical test was run and it had low p value.  However, irrespective of the fact that mice are different from humans, would the clinical data capture the difference which corresponds to difference of five days delay of tumor in mice?  Not quite. Confounders in clinical trials are difficult to control, which make the prevention study even more challenging. 

It's still questionable to persuade individuals to have certain supplement or drink tea even if you've seen the significant benefits in clinical trials.  The study in population may not apply to every individual.  In one research presented by Dr. Yang, the clinical trials showed that the benefits could only been seen in females but not males and they proposed that smoking might be the confounder.  If smoking is really the problem here, then drinking tea might not do any good to a smoking individual. We need to treat the clinical data with caution before we make any conclusions or approve any supplements that's said to prevent cancer.

It needs both statistics and sound judgement to do good science, especially in cancer research.            

The Anatomy of Deceit

Like many great scientists before him, Dan Ariely was inspired to answer questions surrounding what he called “deceitful behavior” from a real life experience he had in the burn ward of a hospital. From his laboratory experiments on pain and reward at MIT and CMU, he concluded a few key points, which will serve as a framework for my reaction to the accompanying articles. These key points are as followed (paraphrased from Mr. Ariely’s own words):

1. Many people engage in deceitful behavior, but only do so a little bit at a time.
2. When people are reminded of their own morality, deceitful behavior goes down.
3. If someone is out-performing the rest of the group and part of the in-group, deceitful behavior increases.
4. When there is distance from a tangible end-point, deceitful behavior increases.
5. People have a hard time doing difficult tasks to prove they are engaging in deceitful behavior. 

Some of these points, when contrasted against the accompanying articles, brought up interesting questions for me on how and why dishonest science happens. For instance, according to the article written by Julia Belluz on supposed “miracle” drugs, there appears to be both the in- and out-groups who perform some level of deceitful behavior. That is, it is not only the doctors who use exuberant language to describe the results of certain cancer drugs, but also the journalists who are responsible for reporting on them.

                                                         

Can one then make the argument that journalists, like medical practitioners, occupy the same in-group? Or is it that the in-group and out-group have a symbiotic relationship where the in-group (doctors) can influence the out-group (journalists) and vice versa? In addition, Belluz cites immunotherapies, “the vanguard of cancer research,” as being the most frequently hyped cancer therapies. This addresses Ariely’s 4^th conclusion above: that is, a cure for cancer is far off in the distance, but scientific publications exist as an immediate means of professional currency. However, it exposes an interesting question for me: would cancer researchers not working in cancer’s “hottest field” feel the need to engage in describing their therapies with such hyperbolic rhetoric?

My gut tells me the answer to this question is no, especially considering the implications of curing cancer. I believe no matter where you end up, these overreaching descriptions of therapeutic results serve as a way to move the field forward, albeit not in a very honest way. This grandiose language exposes dishonest behavior by putting a proverbial red flag to heed attention to potential results. In Jared Horvath’s article, he suggests that these mistruths are simply a consequence of science, and that reproducibility, whether it can be achieved or not, must be fully disclosed. Furthermore, the inability to reproduce serves as a helpful caveat to moving the body of research forward. In many ways, Hovarth seeks to engage more researchers in Ariely’s 5^th conclusion: he hopes that researchers will undertake the difficult tasks of proving their deceitful behavior for the common good of science. This, I believe, is the future of scientific research -- engaging with our human errors.