Tuesday, January 19, 2016

The Fudge Factor: Where does science fit in to a world where everyone lies?

In Dan Ariely's Ted Talk, we are introduced to a phenomenon where, despite potential consequences or incentives, people lie. This isn't anything particularly novel-- everyone knows they're surrounded by liars, because everyone lies themselves. Despite this universal knowledge, the majority of people exist in a blissful world where science and scientific publishing is exempt from the “Fudge Factor”, and it's difficult to understand why.

Let’s examine for a second the tiers involved in scientific publishing—we have the scientist and collaborators who design an experiment based on observed and researched data, perform the experiments to the best of their ability, reproduce the experiments, and compile their data for publication. We have the publishing companies that receive and distribute the manuscript to editors and peer reviewers. The peer reviewers that diligently read the manuscript for credibility and relevance, and ultimately return the manuscripts back to their editors for revisions. The last player in the publishing of a manuscript is the reader—the scientist, media, or interested party that consumes the published manuscript.

With all that in mind, I see two glaring reasons why scientific publishing lends itself to deception. One—the demand for new manuscripts, new ideas, and even new “miracles” is so high. Take the findings by JAMA regarding the nomenclature of press releases on new oncotherapies. When every new study is hailed as a potentially lifesaving new breakthrough in the field, it is no wonder that reproducibility in the field of cancer research and others is so dismal. The second problem I see, is the aforementioned structure of scientific publishing itself. As it exists today, the system is made so that faulty methods and non-novel findings do not slip through the cracks into publication. The problem is, with so many people involved in publishing a manuscript, all it takes is for one tier of the publishing process to succumb to the "Fudge Factor"-- lying just a little bit, for the integrity of the process to fail. 

I don't see any easy solution to the flaws that exist in scientific publishing. The one thing that I do believe contributes to the integrity of the scientist, the publishers, and the public, that was not mentioned in any of the recent articles on unreliable research is skepticism. A scientist should be skeptical of their research instead of trying to shape their data to fit their hypotheses. Publishing companies should accept openness and discussion in the post review process. Perhaps most importantly when viewing skepticism, the public should maintain a healthy amount of skepticism and scrutiny when reading new publications and press releases. 

It is easy to get swept into the glamour of a promising new result, whether on the bench or over breakfast while watching the news one morning. The job of responsible science lies on all parties to maintain skepticism while producing and consuming science, and hopefully, this scrutiny will lead to a more honest system of scientific publishing. 

2 comments:

  1. I agree that there are a lot of flaws in the scientific publishing system that allows for a lot of bad science to go unscrutinized or critically assessed prior to publication, and I completely agree that the public should always approach scientific "discoveries" with caution and not immediately believe everything that they believe; however, I also think there's a flip side to this (everything in moderation, right?). It's equally frustrating to deal with an easily impressionable and science-illiterate public who is overly skeptical and has been demurred to real science, logic, or even discussion (think of all the conspiracy theories and false science that people believe simply from reading headlines instead of looking at the paper and/or data: vaccines causing autism, climate change not being real, amongst many other topics). I think that skepticism might not be as important for the public, in order for them to understand or trust science, but that critical thinking and analyses, as well as better science education (and a requirement for statistics class? haha), would be more beneficial to the dissemination of science through the publishing. Of course, this assumes that scientists are publishing reliable and accurate data and studies. In the end, as you said, the responsibility of "science consumerism" falls on everyone involved, from the people who are conducting the experiments, to the journals that are editing manuscripts for publication, to the media outlet that is reporting the scientific study, to the consumer, who should probably always be skeptical within reason.

    P.S. Thanks for your thoughts!

    ReplyDelete
  2. I definitely appreciate your thoughts on this topic. It is quite scary how easy it is for misinformation to get out there.

    Even if a journal decides that a study is not reliable enough to be published, another journal might decide otherwise, and there are plenty of lower tier journals out there that it seems will publish just about anything.

    The good thing is that you and I know how to distinguish between a good study and one that is complete nonsense. And we know that a paper in a respected journal is worth a lot more credibility than one in an obscure journal (for the most part, since there are of course exceptions).

    However, what does a general assignment reporter at a local newspaper care about the difference between Nature and some random journal. If they find an article with an interesting finding that will grab people's attention, they won't be vetting it for it's scientific accuracy and statistical prose, they and their editor will be happily publishing it because they know people are going to read it.

    And what's scary to me is that a local newspaper gets delivered to everybody's doorstep. Nature doesn't.

    ReplyDelete