Tuesday, April 12, 2016

If it is not significant, why do we care?

From my first statistics class in High School I learned the importance of p<0.05, but what happens if my data is not significant. What if other people do not think my data is as interesting or as groundbreaking because it fails some statistical test? 
I recently learned that no matter how intelligent, perseverant or hard working you are science will always be about good experimental design and a bit of luck. Can you imagine the frustration of getting an unwanted protein in your Western blot over and over again? As you keep thinking it might be a contaminant, but by accident you discover one of the most important proteins of cell membranes. And this is exactly what happened to Agre who discovered aquaporin channels and won the Nobel Prize in Chemistry in 2003. 
Surely, if your research was important or relevant enough to win a Nobel Prize grant readers will fund you. Unfortunately, that is not the case. Last week I was fortunate enough to hear a story that showed some of the realities about the “publish or perish” mentality. Dr. Roger Y. Tsien along with Drs. Osamu Shimomura and MartinChalfie won the Nobel Prize in Chemistry in 2008 for their discovery of green fluorescent protein (GFP). What the official story fails to mention is that their work was based on the previous work of Douglas Prasher who was forced to quit science because his interest in glowing jellyfish was not interesting or significant enough to be funded. All the Nobel Laureates recognized the significant contribution of Douglas Prasher to the discovery and use of GFP and were reluctant to accept the award without proper recognition to all the contributors. Sadly, since he no longer had a lab he was not considered for the award and he remains unrecognized for his accomplishments.
What I learned from this story was that no discovery is too small or insignificant, and even though our research is often driven by what grant reviewers consider important or significant we must share our findings one way or another. Who knows maybe your failed experiment can help win a Nobel Prize.

On a happy note, Douglas Prasher was able to return to his scientific research in June 2010 after Dr. Tsien and Chalfie publicly acknowledged and advocated his significant contribution to this important discovery.

4 comments:

  1. I think that this is a very important point and problem in the daily life of scientists. Everyone is pushing and working hard to get published, the caveat is that you can get publish only if your data tell a story that is "statistically significant". In the past few years researchers have started to create a database for all the "statistically non-significant"experiments/results/papers with the idea to help other scientist in their work. Here it's an example... http://www.jasnh.com/about.html

    ReplyDelete
  2. This post was really interesting and also really really terrifying. The thought that someone could lose funding, and consequently lose their career?? Especially someone with contributions so widespread as has been GFP in research. Yikes.

    It's weird, the first thing I thought about when I read this post was a conversation I had with a postdoc I'm working with now. I was talking to him about our "BadStats" assignment, and how it's crazy that there's enough papers with bad experimental design/statistical analysis to merit a whole project on it, and he said something kind of crazy (albeit maybe joking?).

    He said that people only use statistics when they had crappy (my censorship) data. It made me laugh, but it also gave me pause.

    Obviously statistics is incredibly important. Especially when you don't have the luxury of 'all-or-nothing' phenotypes, but sometimes the "P<0.05" as statistically-significant-in-all-cases-so-don't-even-question-it-give-me-money-NIH, is really unreliable.

    At the end of the day, "publish or perish" sucks. And it's a circular problem when publishing DEPENDS on significance, even though there's potentially groundbreaking research that lies outside of P<0.05

    ReplyDelete
  3. Very cool post, Laura! This reminded me of the story I heard about CRISPR-Cas's discovery. Essentially, a lab focused more on basic science noticed the novel RNA structures in certain bacteria, but were uncertain of the function. Now we know that CRISPR-Cas could have such far-reaching applications in translational science and medicine, but it is a little disheartening to think of the missed opportunities we would of had if this research had been squashed early on by the "publish or perish"/make-it-significant-and-relevant funding environment. 25 years passed between the time CRISPR clustered DNA repeats were described and when the CRISPR-Cas system was first used to edit the genome in human cells! It seems that this "publish or perish" environment is making us very short-sighted in some respects of the potential that "insignificant" research could actually have on our society.

    ReplyDelete
  4. Wow, this post started out upbeat then took a down turn then went upbeat again. For most of our time as scientist will be spent searching for that one elusive p.05 which will not only justify our existence but help continue our career (and hopefully graduate in the 5.5 years they say is totally average and achievable- (side note. to get an average of 5.5 years until graduation when some students don't graduate until 6+ then some population of students should be graduating in less that 5.5 years. Who are these individuals and how are they achieve this miracle?) But this gives hope that even research which doesn't seem to be interesting or leading anywhere at the time could have some profound effect in the future.

    ReplyDelete