This is a very interesting article about how the irreproducibility of research does not necessarily mean that the research is flawed. Although most of this article refers to psychology studies, I think the same would hold true in the more objective realm of biological sciences. I say this because there are many assays and things that we do as research scientists, that are perhaps more voodoo and superstition than actual protocol. And this is not necessarily a bad thing. These little tricks that we do to make our own research better are not translated into a replication study, and that's okay. Each lab has there own way of doing things, and so seeing that a result can be reproduced between labs (and hands, and superstitions) increases its validity, but not reproducing it does not completely discount the research as a whole. Additionally, there is the concept that if research is published, it is already selected for having unusually small p values. This is a statistical probability in itself and therefore, seeing a similar trend as the original data is perhaps reproducing the "irreproducible" findings.
I'm starting to think that a lot of articles written about irreproducibility is geared towards psychological studies. And I guess this isn't too surprising, considering how much media attention these studies tend to get (ex.This ridiculous study - http://www.telegraph.co.uk/news/newstopics/howaboutthat/3179332/John-Travolta-style-dancing-is-the-way-to-a-womans-heart.html - which was discussed on an episode of Wait Wait Don't Tell Me.).
Anyways, I was torn about reading this article because a lot of its sentiments were how I feel, but the wording was so harsh... I realize that there are also irreproducibility issues in biological sciences, but what are we to surmise from irreproducible data? Hopefully, we can disregard protocol differences from the mix, and focus more on the differences in the population thetas being studied. These differences will help us glean more information about the topic of interest and drive us forward towards the truth.
However, this particular article quotes the following:
Years ago, someone asked John Maddox how much of what his prestigious science journal Nature printed was wrong. “All of it,” the renowned editor quickly replied. “That’s what science is about — new knowledge constantly arriving to correct the old.” Maddox wasn’t implying that science was bunk; he was saying that it’s only as good as the current available evidence, and as more data pours in, it’s inevitable that our answers change
And generally, this is how I feel but when put so bluntly. "All of it." It makes me feel like all of our efforts are quite Sisyphean; we can keep carrying that rock up the hill and try our hardest, but at the end... We're wrong. But I guess that's what drives us as scientists. We're all a little masochistic. So I guess we should embrace the irreproducibility. And all we can do is try out best and keep struggling up that hill...
I think you're right. Over the long term, science is self-correcting. It is unreasonable to expect perfection.
Maddox' Nature publishes two types of articles: One's with long half-lives and others with short half-lives.
I mean, prestigious journals like Nature have the luxury of taking the risk that some articles will turn out to be pure crap. Because the truly solid ones tend to be real paradigm drivers and cement the journal's status, the ones that fall back down the hill don't seem to create as big of a mess as they should. And Maddox knows that.
This is a very interesting article about how the irreproducibility of research does not necessarily mean that the research is flawed. Although most of this article refers to psychology studies, I think the same would hold true in the more objective realm of biological sciences. I say this because there are many assays and things that we do as research scientists, that are perhaps more voodoo and superstition than actual protocol. And this is not necessarily a bad thing. These little tricks that we do to make our own research better are not translated into a replication study, and that's okay. Each lab has there own way of doing things, and so seeing that a result can be reproduced between labs (and hands, and superstitions) increases its validity, but not reproducing it does not completely discount the research as a whole. Additionally, there is the concept that if research is published, it is already selected for having unusually small p values. This is a statistical probability in itself and therefore, seeing a similar trend as the original data is perhaps reproducing the "irreproducible" findings.
ReplyDeleteI'm starting to think that a lot of articles written about irreproducibility is geared towards psychological studies. And I guess this isn't too surprising, considering how much media attention these studies tend to get (ex.This ridiculous study - http://www.telegraph.co.uk/news/newstopics/howaboutthat/3179332/John-Travolta-style-dancing-is-the-way-to-a-womans-heart.html - which was discussed on an episode of Wait Wait Don't Tell Me.).
ReplyDeleteAnyways, I was torn about reading this article because a lot of its sentiments were how I feel, but the wording was so harsh... I realize that there are also irreproducibility issues in biological sciences, but what are we to surmise from irreproducible data? Hopefully, we can disregard protocol differences from the mix, and focus more on the differences in the population thetas being studied. These differences will help us glean more information about the topic of interest and drive us forward towards the truth.
However, this particular article quotes the following:
Years ago, someone asked John Maddox how much of what his prestigious science journal Nature printed was wrong. “All of it,” the renowned editor quickly replied. “That’s what science is about — new knowledge constantly arriving to correct the old.” Maddox wasn’t implying that science was bunk; he was saying that it’s only as good as the current available evidence, and as more data pours in, it’s inevitable that our answers change
And generally, this is how I feel but when put so bluntly. "All of it." It makes me feel like all of our efforts are quite Sisyphean; we can keep carrying that rock up the hill and try our hardest, but at the end... We're wrong. But I guess that's what drives us as scientists. We're all a little masochistic. So I guess we should embrace the irreproducibility. And all we can do is try out best and keep struggling up that hill...
I think you're right. Over the long term, science is self-correcting. It is unreasonable to expect perfection.
DeleteMaddox' Nature publishes two types of articles: One's with long half-lives and others with short half-lives.
I mean, prestigious journals like Nature have the luxury of taking the risk that some articles will turn out to be pure crap. Because the truly solid ones tend to be real paradigm drivers and cement the journal's status, the ones that fall back down the hill don't seem to create as big of a mess as they should. And Maddox knows that.