The value of the P-value

Scientist tend to place a lot of importance in the P value. PIs loose grants over P values, grad students have nervous break downs over “insignificant data”, undergraduates desperately look for outliers they can redo to correct that p-value. Is the P-value really that important?

As we have learned in class this semester, the p-value is the probability of obtaining a result equal to or more extreme than predicted with the null hypothesis. So a low p-value indicates that it was unlikely that random chance gave you such an outlying result. Many scientist would stop when they see that low number, wipe the sweat off their brows, and submit their grant or paper.

However is a p-value of 0.051 really all that much worse than a p value of 0.049? Is the difference between those p-values really worth a nervous breakdown? The p-value is an arbitrary line drawn by scientists, and often young scientists fail to look beyond it. A p-value is just one statistical number that can be used in conjunction with other statistical values such as the mean, confidence interval, and error, to make a decision about the implications of an experiment.

A scientist needs to use his/her judgment to conclude if an experiment is worth pursuing further or if it is a waste of time; a significant or insignificant p value should not make that decision. An insignificant p-value does not mean the data was worthless, but only a clear minded scientist can determine what it actually means. Likewise, but much less considered, a significant p-value does not mean that the experiment showed a result worth celebrating over. Your significant p-value might indicate that your drug decreased the stress levels of mice 10%, but is a 10% decrease in mice stress really significant for any application?

These sorts of questions are what scientists should be thinking about. In one experiment, a p-value may hold a lot of weight, but in another experiment, the confidence interval might tell much more about the results. Scientist must be able to analyze their data, not just statistically, but critically to know what their statistics are actually telling them.

The Anatomy of Deceit

Like many great scientists before him, Dan Ariely was inspired to answer questions surrounding what he called “deceitful behavior” from a real life experience he had in the burn ward of a hospital. From his laboratory experiments on pain and reward at MIT and CMU, he concluded a few key points, which will serve as a framework for my reaction to the accompanying articles. These key points are as followed (paraphrased from Mr. Ariely’s own words):

1. Many people engage in deceitful behavior, but only do so a little bit at a time.
2. When people are reminded of their own morality, deceitful behavior goes down.
3. If someone is out-performing the rest of the group and part of the in-group, deceitful behavior increases.
4. When there is distance from a tangible end-point, deceitful behavior increases.
5. People have a hard time doing difficult tasks to prove they are engaging in deceitful behavior. 

Some of these points, when contrasted against the accompanying articles, brought up interesting questions for me on how and why dishonest science happens. For instance, according to the article written by Julia Belluz on supposed “miracle” drugs, there appears to be both the in- and out-groups who perform some level of deceitful behavior. That is, it is not only the doctors who use exuberant language to describe the results of certain cancer drugs, but also the journalists who are responsible for reporting on them.

                                                         

Can one then make the argument that journalists, like medical practitioners, occupy the same in-group? Or is it that the in-group and out-group have a symbiotic relationship where the in-group (doctors) can influence the out-group (journalists) and vice versa? In addition, Belluz cites immunotherapies, “the vanguard of cancer research,” as being the most frequently hyped cancer therapies. This addresses Ariely’s 4^th conclusion above: that is, a cure for cancer is far off in the distance, but scientific publications exist as an immediate means of professional currency. However, it exposes an interesting question for me: would cancer researchers not working in cancer’s “hottest field” feel the need to engage in describing their therapies with such hyperbolic rhetoric?

My gut tells me the answer to this question is no, especially considering the implications of curing cancer. I believe no matter where you end up, these overreaching descriptions of therapeutic results serve as a way to move the field forward, albeit not in a very honest way. This grandiose language exposes dishonest behavior by putting a proverbial red flag to heed attention to potential results. In Jared Horvath’s article, he suggests that these mistruths are simply a consequence of science, and that reproducibility, whether it can be achieved or not, must be fully disclosed. Furthermore, the inability to reproduce serves as a helpful caveat to moving the body of research forward. In many ways, Hovarth seeks to engage more researchers in Ariely’s 5^th conclusion: he hopes that researchers will undertake the difficult tasks of proving their deceitful behavior for the common good of science. This, I believe, is the future of scientific research -- engaging with our human errors.