Response to “Science has lost its way, at a big cost to humanity”

A few years ago, the Los Angeles Times published an article titled “Science has lost its way, at a big cost to humanity.” In the article, the author claims that billions of dollars in funds are at risk of being lost due to dishonesty in science. The author cites the studies published by Amgen and a group at Bayer Healthcare which reported that most of the findings of the papers which they were basing their biotech and pharmaceutical research could not be reproduced. The falsehoods produced, the author suggests, are the result of scientists wanted to produce exciting data and the current peer review process.

               Being that we, the scientists, are the targets of this accusation, it is easy to dismiss these claims as the concerns of a layman. We understand that falsehood is inherent in a competitive publish-or-die environment but accept it as a necessary evil for the gears of scientific progress to turn. With results comes funding and with funding comes more results. The results which are dishonest or biased may persist for some time but eventually those studies coming behind those results will not be able to stand on their own and will call the validity of the first study into question. And while the peer-review process is not perfect in ridding dishonest studies from science, it is still valuable in deterring a lot of it.

               However, I feel that concerns of a layman are sometimes telling of problems to which those within a field are blind, apathetic, or complicit. The competitive publishing environment which is the norm for science worldwide has serious issues which do hinder progress, ultimately. On top of that, many scientists do not question this system or consider any alterations to the system which may improve research for everyone.

               Despite this, however, I feel the author does overlook a few things. First, the author seems to cast bad light on scientists for not following up most studies as these studies by the large biotech and pharmaceutical companies revealed so much about the reproducibility of these other major studies. Most scientists do not work for multi-billion dollar companies which can afford to devote time and resources to checking other peoples studies. The world of science is mostly a web of independent researchers. Additionally, the studies they were checking were cancer and blood research which often require study of live animals, studies which can take enormous amounts of time and labor for a single lab to complete. To his credit though, the author did include a quote in his article which stated that research seeking to just check another lab’s work would likely not get funded. Second, the author seems to imply that scientists think that peer review is sufficient to promote honest research which I feel is not the belief of most scientists. Reproduction of results is really one of the best indicators of good science, and I do not believe most scientists ignore it in favor of belief in the infallibility of peer review. As it stands, peer review is one of the most efficient ways to check others’ work before it is presented to the larger community. However, I do believe that some sort of amendment to the peer review process is called for to reduce bias in choosing what does and does not get published (without hindering the speed of publication dramatically).

Overall, I feel this article, while having some misunderstandings of the field, is a good reminder to scientists that our methods can always be improved upon and that there are people outside of our research bubble which do care about and are affected by the work we do.

Can we trust peer review?

Peer review is designed to act as the protector of science. The idea is quite simple; experts anonymously review a submitted body of scientific evidence and evaluate its suitability for publication. However, in reality this process fails. This flaw in science’s quality control mechanism was recently discussed in The Economist's “Trouble in the Lab,” in which the author writes that the peer reviewers who evaluate papers for journals aren’t very good at catching errors.

Many journals are aware of this issue and are starting to take action to address the failings of peer review. For instance, the European Journal of Neuroscience is now publishing peer reviewer’s comments along with their names online. This removes the anonymity from the process, and places considerable pressure on reviewers to critically evaluate manuscripts. I like this idea and I think that it will improve the quality of reviews and the peer review process. Another option to address the peer review bias, but maintain the sacred anonymity is to use a blinded approach in which the reviewers are not provided the authors’ names and institutions. This would remove inherent bias in which a reviewer may alter their evaluation of an article due to an author’s celebrity or affiliations. 

What else can be done? As PhD’s in training we should be taught how to critically evaluate an article’s experimental methods, and it’s statistical techniques. In reality, most scientists are not statisticians; thus, it is difficult to evaluate a paper’s methods without formal training in the use of statistical methods in biomedical research. Including more applied statistical training, in addition to the classical mathematical statistical training, within the biomedical PhD curriculum is needed. Ultimately, we must be taught how to look at science with a skeptic’s eye, or else the peer review process will continue to fail. This includes training in statistics, along with the classic training in experimental techniques. The concept of peer review is great; however, its time to improve the status quo. 

Peer Review Imposes Bias

 Peer review within the scientific community is flawed. Having peers in your field or related fields review your data inherently imposes bias. Understandably, everyone in your field would like for your promising findings to be valid because that leads to advancement for the entire scientific community. However, going into a review hoping the results will be a certain way has already clouded your judgement before you even read the first sentence. From then on, you subconsciously—or consciously—look for data that reinforce what you already “know” to be true, and ignore what doesn’t fit into your theory. In psychology, this phenomenon is known as confirmation bias and it stems from the simple fact that we all want to be right. No one goes into their research already thinking that they are wrong; everyone chases evidence for ideas that they think might be correct.

Having peers review your work could impose a certain amount of bias, but to try to avoid this, other measures can be put into place. PubPeer, for example is a post-publication peer review site where scientists from all fields can anonymously critique data and offer suggestions. This is a good start because other scientific communities can be more objective in their reviews. Their careers aren’t necessarily affected by whether your research is valid or not, but they will still offer a thoughtful and careful critique of the data simply because of their love and respect for science (presumably). One problem I see with PubPeer is that all the critiques are happening after the research has been published. This doesn’t quite make sense to me because the whole point of a peer review is to catch mistakes and flawed data before it’s made public. If PubPeer was utilized prior to publication, then many more mistakes and faulty data could be corrected, or at least debated, before they are considered common knowledge. This is especially important for graduate students who are constantly reading papers to learn about the scientific advancements in our chosen fields. If we read papers that “haven’t had all the kinks worked out yet” then our research could be based on unverified information.

I agree that science benefits significantly from having great minds evaluate the work of other great minds, but more importantly our method for evaluation, well, needs a bit of evaluation itself.