Wednesday, April 19, 2017

Philosophy in Statistics and Statistics in Philosophy


Statistics is the development of specific methods for evaluating hypotheses by utilizing and assessing empirical facts. Statistics evaluates a data set and interprets probability distributions over the possible sets. The philosophy of statistics is the understanding and proper interpretation of statistical methods, their utility, abuse, and meaning. Uncovering the steps that were taken to interpret statistical results is quite important to maintain integrity and honesty. This means that the foundation for the motivation and justification of data analysis and experimental design are considered. David Cox, a prominent British statistician, claims that any kind of interpretation of evidence is in fact a statistical model and should be treated as such. Statistical philosophy also tries to make a clear distinction between induction and logical deduction, which is usually not valued. But one of the best aspect of this approach is the awareness of the abuse of statistics, such as selection of method or transformations of the data. The philosophy of statistics is a great progressive movement which can help improve the quality of science being reported.

Statistics can be used to address questions in philosophy. It is not uncommon for there to be more than one way to solve a statistics problem; but there are times when more than one or no solution is correct. I would say that these kinds of problems intrigue people the most. One unanswered statistic problem of a philosophical nature is the “Sunrise Problem.” The question is “What is the probability that the sun will rise tomorrow?” There two ways to solve this problem is by using the “one sun, many days” or the “one day, many suns” approach. As long as you establish the proper axioms, both approaches come to a solution that is considered statistically correct, yet completely different. Another philosophically relevant statistic problem is the two envelopes problem, also known as the exchange paradox. In this brainteaser, the setup is as follows: you are given two identical envelopes, one envelope has $100, the other has $50; you pick one but are given the option to change your decision. There are many proposed solutions to this problem and none is widely accepted as definitive. It would seem obvious that switching the envelopes is pointless, but one can argue that it is better to switch envelopes. The following equations can be used to argue both sides of the argument. The amount in the envelope is denoted by “A”. The first equation states that by switching, the expected value is greater by 25% (1/4). Whereas the bottom equation (3x=total money so one envelope is 2x, and the other 1x) shows that there is no difference in amount expected if you swap.




Unanswerable questions (especially ones regarding philosophical statistics) challenge people to develop new theories. But most importantly, these questions bring about new ways to approaching problems.

How new technology is being used to monitor statistics


New technologies are being developed to monitor published statistics. One of the newest and also most controversial is a software program that scans through published papers and finds, reports, and corrects inconsistencies. StatCheck, developed by Michèle Nuijten and her colleagues in Tilburg University, can scan an article and identify the type of test, degrees of freedom, test statistic, and P value. It can then find inconsistencies and report it to the user. I believe that this is creating a bright future for statistical monitoring. The first time StatCheck was used on a large conglomerate of published papers, it went through 30,717 papers to identify 16,695 that tested hypotheses using statistics and in half of those, it found at least one potential error. However, StatCheck is not perfect and can make mistakes. The former president of the Association for Psychological Science addressed the issues of StatCheck-like programs and what they are capable of if they make mistakes. He stated that uncurated and improper StatCheck results can damage the reputation of the authors and in some cases, may be seen as harassment. Another problem with StatCheck is that it cannot determine the validity of a paper. Even though there was a mistake, this could be due to a typo or a miscalculation that does not change the interpretation of the result. Thus, StatCheck results should be curated or investigated by individuals. As technology advances, more advanced statistical monitoring programs will be developed. This will only help improve the quality and reproducibility of results. In addition, tighter statistical regulations may help restore faith in those who have lost trust in scientific studies.

PubPeer, a user oriented discussion board, is a website allowing academics to engage in post-publication peer review. According to Retraction Watch, a system in place to report retracted papers, PubPeer has been a great ally in the fight against bad scientific paper publications. StatCheck analysis can now be viewed on PubPeer which has increased the number of papers to be wary of. However, there is a concern that this has only decreased the signal-to-noise ratio. Up until now the discussions on PubPeer were impactful at identifying critical flaws in the scrutinized article. Now, StatCheck is flagging papers with P-values rounded off to the wrong digit. The Red outlined image is a submission on PubPeer before StatCheck. The Unregistered User uncovered western blot images that were copy and pasted, a clear ethical infringement. The blue outlined image is from a StatCheck which points out a round off error. Instead of a p-value of 0.406, it should have been 0.40289. This doesn’t change the results of the hypothesis and the paper. (Images are located below)

Just to be clear, StatCheck is a great program that can be useful to tease out small mistakes. It is probably best used to do last minute checks before the authors submit their manuscripts. But, we must consider if this tool is causing more harm than good when people want to meta-analyze previous results. And on top of that, Having StatCheck score of 0 mistakes (which is also reported on PubPeer) can be misleading by making people think that the statistics are done correctly. Even though StatCheck agrees that their analyses should be curated, many people won’t do that unless they are the authors of the paper. Of all the responses that I have seen to StatCheck, all of them were by the authors of the paper. And finally, the biggest problem with StatCheck is that it can influence people to lose their faith in scientific research. Saying that 50% of the papers published have statistical errors can be quite misleading. The last thing that we need is to convince people that scientific research is inefficient and unreliable. Sure, there are bad papers published and improvements can be made, but looking at the big picture, science works. It is what provides us with knowledge about this world. Without the public’s support and trust in scientific research I would not be able to share this with everyone. In conclusion, StatCheck should be on your checklist to be checked-out, however it may not be the best tool for fact-checking. 













The Infamous P-value: What It Means and Why It Sucks

Nowadays, it is nearly impossible to find a published scientific article that does not report at least one p-value.  Although there is nothing inherently wrong with the p-value statistic, the problem lies in how this statistic is used.  Within the scientific community, there is a fundamental misunderstanding of p-values and what conclusions can be drawn from them.  The statistically illiterate scientist thinks that p-values are an indication of the probability that the null hypothesis is true or, even worse, that their experimental result is true.  In fact, however, the p-value is a statistic used to describe the probability of getting the experimental result (or something more extreme), given that the null hypothesis is true.  Unfortunately, most researchers and journal editors incorrectly link low p-values with solid experimental design.  This incorrect interpretation biases the findings from experiments, making them look more grand and inflated.  Worse of all, some experimenters mistakenly use p-values as a measure of effect size.  For some unknown reason, these scientists equate lower p-values with bigger effect sizes.  For example, a clinical trial for an anti-cancer drug as a p-value of 0.001 for a two-sample t-test.  However, this drug only decreases the probability of cancer remission by 3%.  On the other hand, another anti-cancer drug has a p-value of 0.04 after a two-sample t-test, but it decreases remission probability by 35%.  The statistically challenged scientist would report the drug with the lower p-value has the better drug, even though it only had a 3% effect compared to placebo.  Using this logic, it's not hard to imagine that statistical illiteracy and the demand for 'pretty' p-values from journal editors has likely lead to unnecessary human suffering and even possibly death.

This p-value epidemic is a growing problem.  In a landmark study conducted by Dr. John Ioannidis at Stanford university, his team found that after scanning millions of abstracts in Pubmed, p-value reporting in abstracts has increased from 7.3% in 1990 to 15.6% in 2015.  Additionally, they found that nearly 55% of abstracts about randomized clinical trials contained at least one p-value.  These are staggering and alarming numbers considering the scientific community's tenuous grasp of p-values.  Perhaps the most shocking discovery from this article was that 96% of abstracts that reported a p-value, reported a 'statistically significant' p-value of less than 0.05.  This means that of the millions and millions of hypotheses tested, 96% of them were significant.  That seems impossible to me.

In order to avoid bias and report sound science, Dr. Ioannidis suggests using Bayes factors or false-discovery rate statistics.  In my opinion, these two statistics are too foreign for most researchers and will likely be misused out of confusion.  Instead, I think the easiest way out of the p-value crisis is to start reporting confidence intervals (CIs).  The CI is a way to measure uncertainty about the magnitude of the effect.  Essentially, if the 95% CIs for treatment versus control do not overlap, the experimental design is sound and the results of the experiment look promising.  By using CIs, good science can be weeded out from crappy science very quickly, hopefully restoring some validity to our work. 

Tuesday, April 18, 2017

Pre-hoc vs Post-hoc Analysis: Whats the difference?


One of the points of emphasis for this semester has been the use of pre-hoc statistical analysis instead of post-hoc analysis when analyzing experimental data. We have been told over and over again that good statisticians plan statistical analyses before they start their experiment and never wait until after the experiment. But why is this the case? What makes pre-hoc analysis better than post-hoc analysis? The major answer to this question is that pre-hoc analyses allows for testing of a specific hypothesis while post-hoc analysis fits a hypothesis to an observed result. This difference may seem small but it has a large effect when conclusions are made from post-hoc analyses instead of pre-hoc analyses. When conclusions are made from post-hoc analyses, there is an inherent bias, as we are able to test the data in any way that produces a favorable result. In many cases, this leads to data dredging or in the worst cases, p-hacking. Pre-hoc analysis is incorporated into the experimental design and therefore the only testable hypotheses are ones specified by the experimental design. This allows us to make conclusions based on the null and alternative hypotheses, which is the basis of the scientific method. It can then be seen how most scientists should preferentially use pre-hoc analysis, however, all is not bad with post-hoc analyses. The one real benefit of post-hoc analyses is that they have the capacity to show patterns in the data that were not the primary objective of the study. This can be especially beneficial for exploratory science. The only caveat is that conclusions should not be made from post-hoc analyses, only hypotheses which can be further tested in a separate experiment. 

Monday, February 6, 2017

Bias in science

By reviewing a random subset of thoughts that occur to me throughout the day, it is clear to me that the arguments that I make and conclusions that I reach on a regular basis are much more likely to be biased in some way than to be accurate, truthful reflections of the state of the things. This is likely to be caused by both innate biases of the human brain, struggling to build operational frameworks of the world based on limited input while minimizing time and effort (i.e. an evolutionary adaptation), and a certain alignment of incentives in our lives that makes it rewarding enough and/or not too damaging enough for us to warp our interpretations of reality, intentionally or subconsciously, to fit a certain mold (see Dan Ariely’s TED talk [here]).
Nowhere are there higher stakes for recognizing and minimizing bias than in scientific research, which is built on the foundation of seeing the universe as it is, rather than as we want it to be. Still, internal and external biases abound, from large scale publication bias and research “trends”, to reports that unexpectedly high percentages of published research is poorly executed or flat-out wrong (see “Trouble at the Lab” [here]). The incentive structure around science, which funds the grander claims, rewards prolific publication of novel results, and undervalues quality assurance and scientific “due process”, is not helping either. Every time research funding is quickly cut at the first sign of economic uncertainty, every time the media rushes to report unconfirmed results, creating extra unneeded incentives, I can’t help but feel that science is still trying to prove itself to society, to justify the increasing funds coming its way.

The hope of those pondering the fallibility of science, including myself, is that it is a self-correcting process. That amid the chaos, or precisely because of it, the truth will prevail while houses of cards built on wrong hypotheses will come crashing down sooner or later, however high they may rise. Perhaps by slowing the process down a little bit, and investing a little more in the “uncool” science of checking and verifying, we can make science a more efficient and less expensive endeavor.