The U.S. Government spends $5 billion every year on cancer research yet it is no secret that this return on investment has been quite disappointing. What lies at the crux of this attitude, is the growing mistrust that permeates lab findings - corrupt data due to sloppy analysis, unstable results, poor experimental design, and most of all, a replication crisis. When a cancer study spirals into a wrong conclusion, individuals suffer, a multibillion-dollar industry of treatment loses money, and biologists get jaded. This only propagates the cycle of researchers favoring efficiency in publication over the validity of results. As a testament to this, in 2011, a team from Bayer pharmaceuticals reported that only 20-25% of studies they attempted to reproduce generated results "completely in line" with those of original publications.
It's not only a problem of results, repeated findings, etc. It's also a problem of consistent methodology. We cannot solely focus on standardizing analyses of data outputs if a given experiment cannot even be performed in another lab. A comprehensive review of current research literature suggested that essential steps in a protocol are frequently omitted from published papers. Specifically a 2013 survey of several hundred journal articles that referenced >1,700 different laboratory materials revealed that only about 50% of such materials could be identified by reading the original papers.
Ultimately, this flawed cycle of irreproducibility has shaken the grounds of public trust for research science and the advances it offers society, however science's substantive progress is highly dependent on winning back this trust. The publication and dissemination of not only provocative yet imprecise studies, but also of findings that cannot be re-performed for whatever the reason, is only further contributing to the public's lack of confidence in the veracity of a branch of science that has the potential of being revolutionary for biomedicine. And we will never elucidate the inherent problems in such findings until we've figured out a way to make them reproducible.
It's not only a problem of results, repeated findings, etc. It's also a problem of consistent methodology. We cannot solely focus on standardizing analyses of data outputs if a given experiment cannot even be performed in another lab. A comprehensive review of current research literature suggested that essential steps in a protocol are frequently omitted from published papers. Specifically a 2013 survey of several hundred journal articles that referenced >1,700 different laboratory materials revealed that only about 50% of such materials could be identified by reading the original papers.
Ultimately, this flawed cycle of irreproducibility has shaken the grounds of public trust for research science and the advances it offers society, however science's substantive progress is highly dependent on winning back this trust. The publication and dissemination of not only provocative yet imprecise studies, but also of findings that cannot be re-performed for whatever the reason, is only further contributing to the public's lack of confidence in the veracity of a branch of science that has the potential of being revolutionary for biomedicine. And we will never elucidate the inherent problems in such findings until we've figured out a way to make them reproducible.
Sanjana, an interesting inquiry but I'm somewhat confused in the relationship between your post title and the two articles that you reference. The 2011 paper that argues only 20-25% of in-house (Nature) papers in the field of womens health, cardiovascular disease, and oncology are fully reproducible is sampling papers from multiple disease foci not cancer research specifically. Furthermore, the 2013 paper that explores the reliability of materials/methods section again is not unique to cancer research, so it trouble me that you've inferred such a narrow viewpoint of what is really two very broad studies. I would certainly agree with the general conclusion that many studies are not reproducible and are quite vague in their experimental guidance at points, but this isn't a field-specific issue, at least not from the commentary supplied. As an alternative, I would wager that late stage clinical trials of novel cancer therapeutics more likely fail due to the intratumoral heterogeneity e.g. lung cancer in individual A is likely unresponsive to therapy that is useful to individual B. This type of therapeutic difference is only one degree of the heterogeneity between different cancer types and different types of the same cancer among different individuals.
ReplyDeleteWhile I agree that by no means this is limited to cancer research, I think that the lack of reproducibility is ubiquitous in the biomedical sciences, and not much is being done about it. One aspect that we must consider is that perhaps irreproducibility is inherent to the way we conduct research and publish our results, and not necessarily due to sloppy scientists (although that may very well be the case). The materials and methods section of papers tend to be short and some don't go into the necessary detail in order to successfully recreate experiments. Another thing that is usually absent from papers is raw data. I believe originally it was not included in publications since there were limitations of space and length in printed articles. However, now that online articles are more the norm rather than the exception, I think there could be some benefits to including raw data to improve transparency and reproducibility.
ReplyDeleteWhat about cancer specifically drives corrupt data and lack of reproducibility? I am curious to learn more about the social drivers on the research industry in cancer compared to other diseases. While my research is not in cancer, I'm guessing that there an expectation that positive cancer treatment results should to be published as quickly as possible, at the expense of potential validity. Is this because there is so much funding for cancer research? Is it because the non-scientific public is more anxious for positive results in cancer than for other diseases, driving disproportionate investment in cancer studies? Is it to hurry potential treatments, with any demonstrated positive effect (no matter how small), into fast gear so that terminal cancer patients can be given SOMETHING that's better than nothing? What makes cancer research more prone to corrupt data and false positives than, say, HIV or Ebola research? I am curious what you think.
ReplyDeleteThanks for posting this. Cancer is my field of study, so maybe I can shed some light on why many studies are hard to reproduce. It's true there may be poor experimental design contributing to irreproducible findings, but as the comments above pointed out, this is true across the board in biology. There's an additional confound we deal with when studying cancer- it is an adaptable disease that responds quickly and often unpredictably to drug treatment. As Fadi pointed out, tumors are inherently heterogeneous, and clonal populations of cancer cells can be selected for in an unpredictable manner depending on the mouse/ cell culture sample, etc. that is treated with a drug. Genomic instability, another identifying property of cancer, also leads to random mutations occurring in cancer cells, which allows tumors to evolve in unpredictable ways. This is something the field is still very much grappling with. When you treat a tumor with a drug, tumor evolution and clonal expansion allows that tumor to find a way to resist that drug and grow. Triple drug combinations are now being proposed, which is a risky idea because giving a patient too many drugs is extremely toxic. Additionally, determining optimal order of drugs/dosing/schedule/etc is often little more than guesswork, and clinical trials are very expensive to conduct. Luckily, the rise of genomic technologies will hopefully soon make it possible to treat tumors from a personalized medicine standpoint, but the technology has a long way to go. It's sad that cancer research is getting a bad reputation for its limited findings, but it's a more complex issue than most people realize.
ReplyDelete