Thursday, April 14, 2016

Null hypothesis testing, philosophy style

Leave it to those pesky philosophers to find meaning behind it all.
I knew Freud was messed up, but now I can justify my belief with a statistical argument!


  1. I don't think Popper has an accurate understanding of how science works, and I particular disagree with his assertion that science disconfirms or, put another way, that scientific theories separate themselves from ordinary theories by virtue of their falsifiability.

    Take this example: Suppose you are a Newtonian physicist and you have a theory that will predict the orbit of planets. You use your telescope to chart the orbit of Mercury and Mars and your theory does a good job of predicting their orbit. However, when you use this theory to predict the revolution of Venus, you notice a wobble in its orbit that deviates from the predicted orbit. What do you do now? Popper would have you falsify the entire theory since it does a poor job of predicting an accurate orbit. However, I think most scientists would look for other ways to explain the wobble. Perhaps there was another planet that came close enough to Venus on that particular night to cause the wobble in its orbit? In this case, the theory does hold up, and it is an even stronger theory since it is used to explain another phenomena.

    Unlike Popper's assertion that science is "science" because it is falsifiable, I think that predictive value is what gives science it's validity. A given theory might come under fire because it does not predict a given event, but ad-hoc hypotheses can be added to strengthen a theory and give it more predictive power.

    So the, what makes science different then pseudoscience? It's not falsifiability, and both "pseudoscience" and science can be used to make predictions about the future. We need another resource to explain this distinction. Perhaps its just bias.

  2. I can't say I agree. With your argument, the role of any causative factor in any system would be disregarded if there are additional independent variables influencing the system. In your example of the revolution of Venus, a 'wobble' is not sufficient to falsify the theory, because of the overwhelming data from other planets and of Venus minus the wobble. It is important to distinguish between scientific theories and a single cause --> single effect assertion. A passing comet does not falsify a theory, in the same way that variation in body fat does not falsify the relationship between height and weight.
    What I heard in your argument was the need of a perfectly fit model. As we've discussed, models are perfect, but data aren't. A wobble, whether literal (with venus) or figurative, can be anything from random variability to the presence of a third variable. Scientific falsifiability does not and should not rest on overfitting models to the data.

  3. Right. I'm not arguing for perfect model fit and I don't at all think that a perfect fit is necessary for a scientific theory. A good model needs some wiggle room. Rather, I was critiquing Popper and the above video's assertion that science is "science" because of hard falsifiability. Many contemporaries of Popper criticized him because he maintained that science progresses via hard falsification. In the example I gave, Popper might suggest that we should discard the orbital theory because of the wobble in Venus. Instead, I think science progresses and models change because we add on ad-hoc hypotheses to explain additional independent variables beyond the scope of the original model. This muddles the video's assertion that science separates itself from pseudoscience because science disconfirms. Scientists spend a lot of time explaining away data that seems to disconfirm a given model. Barry Marshall's and Robin Warren's discovery that H. Pylori causes gastric ulcers and the ensuing controversies surrounding this claim is a prefect example of this.

  4. Both of you raise very interesting points. I can see what both of you are getting at, and agree that the system that Popper proposes may not be complete. While it is certainly the accepted ideal that all science is designed to test and disprove hypotheses, I'm not quite so sure if it is conducted in accordance with this ideal in all cases. For instance, the video compares two very distinct fields - psychology, and physics. Psychology exists as a field that is closely related to the hard sciences, but is at a distinct disadvantage. Because psychology deals with analyzing the delicate human mind, it is unethical for psychologists to design experiments to test their hypotheses. Doing so could cause irrevocable damage to their subjects, and has been condemned by decades of questionable practices. Instead, they must be content with analyzing the results of their patients' real lives. They can never eliminate biases or confounding factors, because they are powerless to properly control their studies. They must draw whatever conclusions they can from the data available to them, often being unable to properly subject their ideas to tests which have the capacity to disprove them. Now, it might be easy to simply dismiss psychology as not being a true science, since it is incapable of passing Popper's requirement. This may well be the case, but I think the line may be a little blurred when it comes to other situations that we do typically consider to be conventional science. Clinical studies, where it is unethical to inflict the patients with a particular disease or a treatment; epidemiology studies, where researchers can only watch an illness spread through the population. Anything that involves humans is uniquely conflicted in that we constrain our scientific practices with the (very necessary) bonds of ethics. Are these studies to be considered pseudo-science too, since they are incapable of testing and disproving hypotheses? While Popper very eloquently lays out the ideal, the realm of true, messy science in the real world may be a bit larger than he proposes.