Seeking Integrity on the Shoulders of Giants

One of the books that has most shaped my view of the world is Mother Night by Kurt  Vonnegut. In it, the narrator describes his philosophy of how people’s minds are like clockwork, with every fact we know serving as a tooth in a cog. In many ways, though, we are missing teeth. Either we are unintentionally blinded to truth because of the environment in which we were raised, or we purposely deny truths because they do not fit nicely with our mental constructs. He says, “I… will say that I have never tampered with a single tooth in my thought machine, such as it is. There are teeth missing, God knows -- some I was born without, teeth that will never grow. And other teeth have been stripped by the clutchless shifts of history -- But never have I willfully destroyed a tooth on a gear of my thinking machine. Never have I said to myself, 'This fact I can do without.’” As a scientist, my greatest quest is to find truth and to try to fill the gaps in our understanding of the world. Yet, one of my greatest fears is that, knowingly or unknowingly, I am missing teeth in my gears.

Our first lecture with TJ Murphy struck home with me not only because it brought to light the “reproducibility crisis,” but it also reminded me of my part in it. I see that we today are standing upon the shoulders of giants. Everything that we know and believe to be truth is based upon the years of work of those before us. Though I am astounded by the brilliance, creativity, and passion of our predecessors, a part of me is also frightened. How much of what we take for granted is actually truth? In what ways do I contribute to the reproducibility crisis, and where does my own blindness come into light? How can I help promote an environment of scientific integrity? Christopher Pannucci and Edwin Wilkins cite the many different forms of biases that can distort an investigator’s ability to assess their findings, so I must confront and address my biases head-on before starting a project. As stated in The Economist, science is not necessarily self-correcting, so my fellow researchers and I must begin the movement to be more open about science as a powerful, yet fallible, tool for approaching our world’s many questions.

What can we learn from the Duke University Scientific Fraud Scandal?

I would like to bring to your attention to one of the most notorious cases of scientific misconduct in recent memory. The case involved a prominent cancer researcher at Duke University, Dr. Anil Potti. His research claimed that he could use genomic technology to generate a “fingerprintunique to the individual patient” and use it to predict up to “90 percent accuracy “which early stage lung cancer patients were likely to have reoccurrenceand benefit from chemotherapy" and more importantly, predict which chemotherapy drugs would work best for that particular patient. Scientists have called it the “holy grail” of individualized cancer therapy. Dr Anil Potti’s findings have been published in a number of prestigious journals including the New England Journal of Medicine and Nature Medicine. Many companies were formed and clinical trials were started on the basis of this research. It was heralded and thought to change the way we treat lung cancer, and that would be true if the research was in fact real. It was not.

            Dr. John Minna, a lung cancer researcher at University of Texas Southwestern Medical Center, wanted to use a similar approach as outlined by Dr. Anil Potti to treat his own patients and collaborated with two statisticians at M. D. Anderson, Keith Baggerly and Kevin Coombes to understand how to conduct the genomic tests needed to determine which chemotherapy drug to give patients. However, when Dr. Baggerly and Dr.Coombe started to analyze Dr. Potti’s data, they found a number of mistakesranging from potential careless mistakes to “unexplicable” errors. At this same time, Duke University started three clinical trials. Dr. Coombe and Dr. Baggerly tried to warn the scientific community about the factualness of Dr. Potti’s data. Nevertheless, Duke University continued their trial. It was not until a trade article published in Cancer Letters which showed that Dr. Potti “falsified parts of his resume” and reported that he was a Rhodes Scholar on a number of grant applications, which was a lie. The scientific community finally checked Dr. Potti’s data and realized that the entire result was fabricated. However, some patients died during the clinical trails and were given the false hope that this technology could improve their clinical outcome.

            There are a number of things to be frustrated about in this case. At the top of that extensive list is the idea that a researcher can knowingly publish false data that has a real impact on the lives of cancer patients, many of whom are agreeing to participate in this trial because they have run out of options. Secondly, that the scientific community did nothing about the findings presented by Dr. Coombe and Dr. Baggerly, and it took a forged resume to open the eyes of the scientific community. I am a strong believer that scientific research is self-correcting. Given enough time, especially in the case of potentially groundbreaking research, scientists can separate out the false from the real data. However, the case at Duke University indicates that sometimes it just takes too long to unmask the fraud. In the case of cancer patients, they do not have the luxury of time. There needs to stronger safeguards in place to detect instances of fraud. Maybe that is easier said than done. Prestigious scientific journals favor the once in a life time finding, especially if it contradicts an established theory. Perhaps, we need to seriously think about what changes can be made to better the peer-review system to make sure such heinous cases as the Duke University scandal do not happen again.       

How to think about Statistics and Confidence Intervals (for a p-value-centric scientist)

Introducing Statistics and Confidence Intervals

Statistics is, to me, man’s way of recognizing that we are imperfect and doing our best to control for it. We try to reduce bias at every level of experimentation, from study design to statistical analyses, but because this is a man-made technique of reducing man’s impact on the work that we do as scientists, it is only as effective as we are. It is the same as a computer- a computer is only as powerful and smart as the person who is running it. As such, we need to make ourselves as unbiased and as well-educated as possible in order to trust the conclusions that we draw. It is easy (and only human) to overlook many of the possible variables and situations that can cause our data to look a certain way that have nothing to do with the experimental treatment that we wish to test (and many times, that which we think we are successfully testing!).

The problem with statistics is that many times, we think we know more than we do. We are overconfident in our hypotheses and in our conclusions, and we yell on top of the data (with asterisks) instead of letting the data speak for itself. It is not enough to execute a well-designed experiment. It must be interpreted correctly as well in order to make inferences about the world around us, which is the ultimate goal of experimentation. For example, the p value is touted as the “end-all-be-all” of scientific (statistical) significance. If p<0.05, then we conclude that our treatment is working and we should get a Nature paper. However, in many cases, these small p values still beg the question, WHO CARES? If something is statistically significant, it does not mean that it is clinically relevant. Additionally, the scientific community receives (or should receive) a lot of flak for the weight they give to p values, when in fact what we should be reporting most of the time is a confidence interval. The confidence interval is intimately related to the p value, but it gives far more information and is a more accurate and informative description of the data. People do not understand p values and many times, they do not stop to think closely enough about confidence intervals either. Below are two graphs I have selected from a biostatistics lecture by Patrick Breheny illustrating the differences that result from your choice of confidence level and how they are intuitively very simple, if one takes the time to think about them…

Now, one of these graphs shows a 95% confidence interval, and the other shows an 80% confidence interval. If you think about just the values, you would (wrongly) assume that an 80% confidence interval is “worse” than a 95% confidence interval because 80 is less than 95. However, the definition of a confidence interval is that there is a X% chance that your interval contains the population mean. So, in order for you to be more sure that your interval will contain the true population value, you must widen the interval. Therefore, a 95% confidence interval is actually larger than an 80% confidence interval, but you are more confident that it contains the true population mean. Understanding this somewhat simple but very important concept is essential to generate and interpret scientific data. This course has illustrated this concept and the importance of statistics very well and I will make sure to keep this in the back of my mind throughout my career.

Boost scientific data reproducibility with quality standards, honor code

There is indeed a problem in the realm of scientific research when it comes to replicating data, as we learn from an article published by The Economist: Trouble at the Lab. The article discusses several phenomena that contribute to this problem, including bad statistics on the part of the scientists, poor research methodologies, peer reviewing that is inadequate, and lack of access to researchers’ methodological data and software. As a result, scientific research finds vastly challenged its reducibility, a quality which is a pillar of its ascribed objectivity.

There is a striking complementarity between this article and the assigned TED talk by Dan Ariely: Our Buggy Moral Code. Ariely discusses factors that encourage or discourage cheating in people. He discusses that people indeed do choose to cheat, and they choose to do so only a little. Additionally, one of his main points was that when reminded of morality people tend to cheat less. Let us replace Ariely’s “cheating” with the article’s phenomena that contribute to irreproducibility in research. If we do this, “cheating” is essentially not following proper standards of scientific research. From this perspective, we may posit that scientists at least aren’t too improper with their research methodologies, so there must be hope for us! As for reminding scientists of morality, the NIH could create an honor code for scientists.

That may sound funny, but at the same time I envision this: that a future generation will not only have an honor code for scientists but also an enforced formal quality standards for reporting scientific data. Firstly, that future generation with a regulated approach to reporting data will look back to ours and find it absurd that we did not have an enforced formal quality of standards. And I would not blame them, based on data such as that in the article being discussed which indicates that there is a prevalent degree of improper methodology when it comes to reporting data. I conceive that the same generation will look back at our lack of honor code with similar estrangement, for if they were taught to respect a set of morals in research, it would generally move them toward abiding by those morals, shifting the scientific culture. This would likely improve scientists’ desires to implement more proper methodologies in reporting their data, despite any cost to themselves since carrying such costs would not be taboo. Rather, it would be widespread and done in the cause of preserving the integrity of science’s reproducibility and thus objectivity. Oh, and the NIH would make sure these scientists don’t go out of business.

Publication Pressure and Scientific Integrity

Research today sits on the cusp of an incredible era. The evolution of technology has created a plethora of tools capable of addressing nearly any scientific question from nearly any angle. Microscopy, nanoparticles, sequencing and hundreds of other fields have advanced wildly while at the same time becoming more affordable. This has made scientists more productive, more effective, and led to many more publications flooding the desks of editors of scientific journals. In order to keep up and to stay relevant, the best journals have become more selective. They expect in vivo observations, a biomedical context, and a complete and compelling story. Which is great. Except that such studies typically take years and years to complete without any publishable findings in the interim. With the current environment in academia, where “publish or perish” is a very real concern, this selectivity for the most compelling or unusual articles is frequently a death sentence for some of the best, brightest, and most principled investigators.

Cue PLoS One; one of many journals of its kind that hopes to allow for “a faster path to publishing in a high-quality peer-reviewed journal.” According to its website, “all work that reaches rigorous technical and ethical standards is published and freely and immediately available to everyone.” In my personal and completely unsubstantiated opinion, I think this is a great mission statement. I believe research happens in very small increments, and the ability to publish those increments gives scientists a way to mark their progress as they march toward a more complete understanding. These types of journals also provide an outlet for less-sensational follow up work or negative results, which at its best would debunk false results that are misguiding the efforts of the scientific community and at worst would save some poor graduate student from sweating over a hopeless project. Supposedly if these papers are deemed to meet “technical and ethical standards” they will be freely disseminated to the scientific community. The extent to which these journals serve this purpose in reality, however, is in dispute.

In an article in The Economist in 2013 titled “Trouble in the Lab,” the authors question the utility of these lower-tier journals. In regards to publishing negative results they write: “Journals, thirsty for novelty, show little interest in it; though minimum-threshold journals could change this, they have yet to do so in a big way.”

But the fault cannot be wholly placed on the shoulders on these publications. It is not exciting to try to replicate previously published results, or to write a manuscript centered on a null hypothesis. The authors acknowledge that “Most academic researchers would rather spend time on work that is more likely to enhance their careers. This is especially true of junior researchers, who are aware that overzealous replication can be seen as an implicit challenge to authority. Often, only people with an axe to grind pursue replications with vigour—a state of affairs which makes people wary of having their work replicated.” This is disconcerting for a profession that derives its relevance from a public perception of dogged adhesion to the search for truth and a strong personal sense of integrity.

With a wealth of technological tools and bright minds, scientific research finds itself in an exciting yet perilous position. Moving forward it will be critical to balance the thrill of scientific discovery with the necessity of healthy skepticism. I believe, unlike the author of the cited article, in the ability of the scientific community to police itself. If academic research is to stay relevant, however, we need to acknowledge the importance of negative and mundane results in a more systematic fashion.

Modern Science requires Modern Guidelines

After viewing Dan Ariely’s TED presentation: The Bugs in our Moral Code, I found myself astounded by the clear parallels between Dan’s social psychology experiments and the current state at which the scientific community finds itself. Briefly, an individual’s propensity to cheat to a certain degree (defined by Dan as personal fudge factor) is controlled through not only the associated risk, but also the perception of others - chiefly those within our own community. This is a common modality that can easily sway the minds of scientists if perceived as beneficial to either the funding of their work or the legitimacy of their beliefs or ideas. This can manifest in something as simple as the lack of a proper experimental control to that of turning a blind eye to confounding data on a “make or break” grant submission.

While instances of direct data fabrication are few and far between according to groups such as Retraction Watch and The Scientist, the inundation of published experiments in highly regarded journals found to be un-reproducible is only beginning to come to light. Editorials on this subject first began to surface from names like Nature in 2012 with the release of: Must try harder. This article argues the arrival of an endemic of scientific “sloppiness” citing the overwhelming number of novel cancer therapies that fail to reach clinical trials due to inadequate pre-clinical data that cannot be reproduced. In recent times, nothing screamed out at me “non-reproducible” quite as much as the story of Dr. Charles Vacanti at Brigham and Women’s Hospital in Boston MA. The publication was first heralded as the greatest advancement in stem cell technology of this century: STAP. Its retraction and verdict of scientific misconduct later resulted in the destruction of the careers of many highly regarded scientists as well as the suicide of one of the Japanese co-authors. This instance could be owned up to the ever present “publish or perish”  mindset  inherent of running a successful lab in today’s funding environment, or simply the presence of one dishonest scientist with an enlarged personal fudge factor. Regardless of the cause, these events demand a proactive advance towards the dissemination of highly reproducible studies assessed through strict guidelines imposed by the leading scientific journals. This tenet is supported by Dan’s social experiments in which an honor code is introduced, reducing the generalized cheating. 

 Other editorials have argued that “Reproducibility will not cure what ails science” stating that open access to data is the only “cure”. With the advent of big data experiments, data mishandling becomes inherent to the nature of the experimental plan. Personally, I could not agree more with the push for more stringent data reporting not only in terms of raw data calculations but also in the final statistical analysis of such studies. PubMed Commons aims to provide a secondary approach in which reproducibility and many other subjects can be discussed in a forum based method inside of the scientific community in regards to specific publications. At the end of the day, while the experimental integrity lies at the hands of the scientists, the affirmation is at the sole discretion of the peer review. 

Efficiently revealing truth

Science is the pursuit of absolute truth. In Dan Ariely’sTED talk on deception, he reveals some of the evidence gathered on how people perceive deception. While not explicitly deception, when people form a preconceived idea, they tend to stick with it and ignore evidence that contradicts their idea. In order to avoid this effect, scientists should enter experiments ready to accept any result that occurs. This can be achieved in part by writing out how each possible outcome of the experiment would be interpreted and the impact it would have on the field, before performing any experiments. Thinking through experiments in this manner would help in identifying flaws in the experimental design as well as prevent the formation of preconceived ideas which would in turn aid the scientists in their pursuit of truth.

Ariely’s research also showed that people are less likely to deceive when they are reminded of moral codes, whether or not they claim to ascribe to those codes. Additionally, the closer an act of deception is to money, the less likely people are to commit them. These two observations can be useful in building integrity into the scientific process. For example yearly lectures and monthly newsletters on scientific ethics and serve to remind scientists of their duty to perform experiments and analyze data in an unbiased manner. Some scientists, especially those in training, may not be cognizant of the amount of money needed to support their experiments and publication of the data generated. Reminding them of the financial resources needed to perform experiments could incentivize them to properly prepare prior to beginning experiments or publishing to ensure that the data generated or presented is as informative as possible. Taken together, these findings of Ariely’s can be used to integrate changes in the scientific process to ensure an optimum level of efficiency and integrity.