Disclosing bias may not be worth the risk

When conducting experiments to expand the knowledge within the sciences, it is important to remove as many biases as possible. Biases can mislead the researchers and can also be a major factor to the irreproducibility of the experiment. But, one can argue that it is impossible to remove all biases. Could this mean that all research is flawed? Jared Horvath pointed out that, “In actuality, unreliable research and irreproducible data have been the status quo since the inception of modern science.” This is an alarming but true statement and can sway people’s faith in the scientific and pharmacological community. After the recent vaccine scare, which was considered as unwarranted, it appears that people have lost trust in the large drug producers. The main drive of the “anti-vaccers” is the possibility that these companies have a financial bias. What would people do if they knew of the many other biases that exist when vaccines (or other drugs) are developed and evaluated? Jared Horvath says that biases should be communicated honestly to the public but I think it will make the situation even worse. Brian Shilhavy points out an interesting conflict of interest of the CDC. Because the CDC has a budget to purchase 4 billion dollars worth of vaccines from companies, he questions whether they should be allowed to oversee the safety of vaccines. Regardless if this claim holds water, it still puts distrust between the public and drug companies. Bias does not only impact the irreproducibility of an experiment but also the faith of the public.

I believe that it would be irresponsible and unscientific for someone to follow a claim without doing proper research. However that philosophy should also apply to people willing to take vaccines. Why should they simply believe that vaccines are safe? The truth is, they shouldn’t. But, how can any of these parties conduct literary research without access to the proper resources. It is up to scientists and physicians to provide access to easy-to-understand [raw] data to educate the public. This is a better solution in my opinion than disclosing all “possible” biases. 

Combating Bias and Irreproducibility in Research

Recent reports by the news media have begun to shine light on a serious issue facing the scientific research community: bias and irreproducibility in research are very real problems that often result in wasted resources. During my time as a research technician in a highly competitive academic environment, I frequently witnessed the ‘forces’ that I believe are responsible for these problems: 1) a ‘publish-or-perish’ culture that encourages academic faculty (particularly junior faculty) to rapidly and continuously produce exciting results, sometimes at the expense of being painstaking in their research efforts; 2) scientists’ lack of substantive knowledge regarding how to use statistics to appropriately design experiments and analyze and interpret data; 3) opaque communication of experimental methodology; and 4) lack of communication of negative experimental results.

Moving forward, concrete steps can and should be taken to avoid bias and minimize irreproducibility wherever possible. While the first ‘force’ mentioned above would be impossible to change quickly because it would require a dramatic shift in the deeply ingrained culture of science, I believe the second, third and fourth ‘forces’ could be readily addressed by 1) enhancing scientists’ training in statistics, 2) raising standards for data transparency, rigor and reproducibility, and 3) encouraging more communication between scientists. Fortunately (and also likely in response to increasing public attention to bias and irreproducibility in research), leading scientific journals and grant-funding institutions have already started to push scientists toward more rigorous statistical treatments of data and greater transparency in methodology. I’m optimistic that these efforts will help to positively shape the scientific research landscape such that current problems with bias and irreproducibility will be increasingly mitigated.

For the specific purpose of increasing transparency in scientific methodology and enhancing communication between scientists, another effort that I think could benefit the scientific community would be to establish an online forum dedicated to each scientific paper. In each forum, the different scientists attempting to reproduce the results of an original scientific paper could directly communicate both with one another and with the authors of the original paper. This forum could be used as a ‘safe space’ to ask for help in trouble-shooting problems that arise, to ask for clarifications in methodology, and, most importantly, to report conflicting or negative results. This approach has already been implemented to great effect by some research groups, e.g. the Zhang group for the development of CRISPR/Cas9 gene editing systems.

In sum, bias and irreproducibility in research are finally receiving their due attention from the public as real obstacles that must be addressed and overcome by the scientific community. I’m hopeful that current and future efforts undertaken by the scientific community to address issues in bias and irreproducibility will continue to enhance the quality of published scientific research.

Cancer: The Disease of Irreproducible Results?

The U.S. Government spends $5 billion every year on cancer research yet it is no secret that this return on investment has been quite disappointing. What lies at the crux of this attitude, is the growing mistrust that permeates lab findings - corrupt data due to sloppy analysis, unstable results, poor experimental design, and most of all, a replication crisis. When a cancer study spirals into a wrong conclusion, individuals suffer, a multibillion-dollar industry of treatment loses money, and biologists get jaded. This only propagates the cycle of researchers favoring efficiency in publication over the validity of results. As a testament to this, in 2011, a team from Bayer pharmaceuticals reported that only 20-25% of studies they attempted to reproduce generated results "completely in line" with those of original publications.

It's not only a problem of results, repeated findings, etc. It's also a problem of consistent methodology. We cannot solely focus on standardizing analyses of data outputs if a given experiment cannot even be performed in another lab. A comprehensive review of current research literature suggested that essential steps in a protocol are frequently omitted from published papers. Specifically a 2013 survey of several hundred journal articles that referenced >1,700 different laboratory materials revealed that only about 50% of such materials could be identified by reading the original papers.

Ultimately, this flawed cycle of irreproducibility has shaken the grounds of public trust for research science and the advances it offers society, however science's substantive progress is highly dependent on winning back this trust. The publication and dissemination of not only provocative yet imprecise studies, but also of findings that cannot be re-performed for whatever the reason, is only further contributing to the public's lack of confidence in the veracity of a branch of science that has the potential of being revolutionary for biomedicine. And we will never elucidate the inherent problems in such findings until we've figured out a way to make them reproducible.

The problem is in the design of experiments

Here's another well-written lay article, this time from The New Yorker, on irreproducible findings, the way science is being conducted, and our inherent biases as human beings.

Money quote:

That’s why Schooler argues that scientists need to become more rigorous about data collection before they publish. “We’re wasting too much time chasing after bad studies and underpowered experiments,” he says. The current “obsession” with replicability distracts from the real problem, which is faulty design. [....] “Every researcher should have to spell out, in advance, how many subjects they’re going to use, and what exactly they’re testing, and what constitutes a sufficient level of proof. We have the tools to be much more transparent about our experiments.”

Indeed, "We have the tools." We've chosen not to use them for the last couple of generations.The design of experiments is a concept that was invented decades ago to minimize the very problem we're grappling with today. We've simply not taught it well or learned to use it or chosen not to use it, perhaps because our great ideas couldn't possibly be wrong.

h/t Ken Liu

Intrinsically Intertwined

After reading the posted articles on irreproducibility and bias in science, I am surprised that there are not further measures in place to combat these issues. The use of anonymous post-publication peer review and Bayesian statistics to justify redoing an experiment seem like common sense measures. Why have these not become standard practice among the scientific community?

As the article “Trouble in the Lab” states, “more than half of positive results could be wrong.” This was revealed by John Ionnidis’ 2005 paper, which proved the cost of a seemingly small number of false positives. When I connect this thought to my own research, I am horrified. What if the claims that helped me to develop my experimental theory are unreliable? Though they were published in peer-reviewed journals, perhaps their results do not reflect actuality. These “discoveries” might have not been discoveries at all, but simply instances in which the data told an incorrect story. Because research builds on previously published results, a false published result could lead to a chain reaction on incorrect assumptions. How can this chain of events be halted?

Statistics proved that irreproducibility is a rampant issue among many life science research investigations. I believe that statistics can similarly be used to combat this problem. Even though “most scientists are not statisticians,” acceptance of the explanation laid out by Ionnidis should be a prerequisite for performing research. It should guide scientists to perform more experiments and to not be fooled by false positives. Perhaps, then, greater care will be made to distinguish discoveries made by statistical anomaly from discoveries that represent the laws of science. Because of the nature of their work, scientists should take it upon themselves to become as versed as possible in statistics. The two are so intrinsically intertwined—this fact can no longer be ignored in the scientific community.

15 Seconds of Significance

As pointed out by many, a reigning mentality in science is “publish-or-perish.” The pressure to publish results, positive ones at that, is high. And as a graduate student, I find myself being taught that this how science is conducted.

To earn my PhD, I have to publish.

The publications generated during my years in graduate school are supposed to be a way for me to demonstrate what I have learned. Sadly, publications are frequently plagued by significance bias.

The bias of significance can mean many things. The straightforward interpretation is that of statistical significance. Any positive data published must reach significance, and often times it is reached unethically. I agree with Victoria Stodden, scientists often do not use appropriate statistical methods to interpret their data. But how a scientist comes to use the wrong method is varied and is normally born of ignorance and choice. Inappropriate methods and fishing for significance is a frequent occurrence in science, and it is often caused and perpetuated by a scientist’s pressure to publish.

Another aspect of the significance bias is the drive for something to be novel. What is often valued in science is something that shakes the foundation of its field. Not so surprisingly, many publications that do this are often untruthful, irreproducible or flawed. Replication of data is important to determine the truth in a situation and no scientist should be afraid of it. Horvath stated that, “If replication were the gold standard of scientific progress, we would still be banging our heads against our benches trying to arrive at the precise values that Galileo reported. Clearly this isn’t the case.” I disagree, replication is a gold standard for the progress of science. Replication by many is simply more evidence that what was observed is real, and we can come to accept it as fact and move on from it. And yet, science as a community rewards those who publish in high impact journals, even if there data is wrong. The rewards of more funding and notoriety just reinforce the “publish-or-perish” mentality.

At present, science is stuck in the mentality of “publish-or-perish.” It is unlikely, that this and our obsession with significance will ever change, until science is no longer approached as a for profit business.

Giving Credit Where Credit is Due, or How We Incentivize Bad Science

The scientific community has recently come under fire for irreproducibility and flawed methodology. Both of these are serious accusations in a community that prides itself on the rigorous and self-correcting system that has been built over the last several hundred years. Numerous articles have detailed the holes in the fabric of the system, including one particularly thorough piece that appeared in the Economist. But few have addressed how and why these holes have come to be, and in order to do that we need to take a hard look at how a good scientist’s career is formed.

            The vast majority of benchwork science is performed by graduate students, specifically PhD students in the early stages of their careers. This is a make it or break it time for them, when good results may mean a job after graduation, so naturally they are vastly overworked and, on average, horrendously underpaid. Why they choose to work under such conditions varies, but if you choose to ask them about their projects, you’ll find that they are remarkably passionate about their work. Their goals seem to center around being able to do the science they love, regardless of whether it pays well or not. This passion seems innocent until you combine it with the lax requirements for training in statistics and rigor found in many graduate programs. The NIH has taken the first steps by requiring that all predoctoral and postdoctoral grants address how reproducibility will be handled in the proposed study. But not everyone actually performs the necessary tests of their data, and it takes a lot to change a community with such an intransient culture. Unchecked, these flaws in mentoring result in hasty and botched data analysis in the short term, but even more disturbing issues down the road.

            After obtaining their PhDs, postdoctoral fellows have many demands on their time. They must run projects independently, write papers on the results of those projects, secure their own funding, and review papers that have been submitted to journals in which they themselves have published. Because their future careers depend on their funding and publishing rates, it is no wonder that these tend to be the focus of their attention. The pile of papers to be reviewed, which often gets added to by their mentoring professors, often gets overlooked in the mad scramble of eighty-hour workweeks and fast-approaching grant deadlines. This results in another hole for bad data to slip through: a hasty reviewer is less likely to catch mistakes in methods or analysis than one that is properly incentivized to perform this holy task of peer review.

            And finally we reach the final stage of a scientist’s career, the tenured professor. Their job is, if possible, more hectic than a postdoc’s. They have to combine the obligations of teaching with that of writing grants, directing multiple research projects, working with the administration, mentoring graduate students, and reviewing papers submitted to various journals. Like postdocs, they are incentivized based on their grants and publications, with some incentives for teaching. As a result, mentoring and peer review can fall to the wayside in a culmination of a career that started with failure of instruction, thus creating a self-perpetuating cycle. Students are taught very little statistics if any, and are not incentivized for participating in the peer review process, so they do not pass on those traits to their future students.

            In the end, the solution is remarkably simple. We need to include peer review, quality mentoring, and good statistical analysis as skills necessary to obtain a professorial position. Once professors are forced to embrace these qualities and pass them on to their graduate students, the system will fix itself relatively quickly. The key is forcing PhD programs to put their money where their mouth is and train their adherents in proper data analysis. This will fix the problem of accidental misrepresentation of data. Properly incentivized peer review will fix the problem of purposeful misrepresentation of data. With these loopholes closed, we can turn to the other matter of fixing field-level standards for significance, methodology, and insignificant results that were brought up in the Economist.