Another Podcast

Hi Everyone.

I'm stealing Arielle's idea and sharing this Freakonomics podcast about how to become a "super-forecaster" that aired last week. It's an interesting take on how important understanding probabilities and statistics can be outside our hard-science world.

I particularly appreciate the comments that what sets apart a bad or overconfident "forecaster" from a "superforcaster" is dogmatism. In the context of this podcast, dogmatism is discussed as a personal desire to come up with reasons to support a preferred prediction with a tendency to disregard reasons that go against the preferred prediction. I think this idea can be extrapolated quite well to the scientific community, and community in general, as a whole. Often, what has been done previously or what is discussed with the most passion steers decision making just as much as-- if not more than-- the direction most evidence is pointing.

Listening to this podcast (for the third time, now) reinforced in my mind that we should embrace open mindedness and flexibility while allowing data-- be it in a scientific context or not-- to drive our opinions and change them throughout time.

The podcast closes by campaigning for more accountability in public debate following up on all predictions rather than simply choosing to discuss those that are convenient to discuss at a later date. I completely agree that this strategy could eliminate that tendency of public personas to make sweeping, dramatic promises and predictions that can rile up the populace with little (or no) basis or consequence. If we are going to continue as (or return to) a civil society it is important to remember that thinking and effort are critical components.

Towards better understanding irreproducibility.

“The idea that the same experiments always get the same results, no matter who performs them, is one of the cornerstones of science’s claim to objective truth. If a systematic campaign of replication does not lead to the same results, then either the original research is flawed…or the replications are…Either way, something is awry.”

Perhaps one of the most prevalent biases occurs when one assumes that doing X will always result in Y. For example, a scientist wanting to confirm previous findings will repeat protocol X based on the assumption that this should replicate result Y. A doctor prescribes treatment X because treatment X is expected to produce outcome Y, alleviating their patient’s ailment. 

Relying on definitive X=Y thinking is comforting in everyday life. We drive route X every morning because it leads us to our work at destination Y. Except, we know that in day-to-day commuting there are a number of events that could arise in which taking route X would not necessarily lead us to destination Y. Tuesday morning, we may find ourselves taking an unexpected detour halfway along route X due to an unpredictable fallen tree across our route. In science, we adhere to randomization, proper controls, blinding ourselves to our samples, p=0.05, etc. as a means to prevent our results from being thwarted by unpredictable fallen trees. Yet, as an article in The Economist details, many studies do not repeatedly end up at result Y, despite adhering to route X. 

The author points to a number of different reasons as to why studies may be failing to consistently replicate: review processes are less meticulous for some journals than others, authors omit methodological details needed for others to accurately reproduce protocols, an increasing pressure to quickly publish findings may promote quantity over quality of scientific results, etc. In response, I wonder whether fixing these institutional problems would definitely result in marked increases in reproducibility? If so, maybe we could potentially agree with the author’s comment that irreproducibility is the product of either flawed original research and/or replication studies. However, if irreproducibility continued (in instances where proper protocols, assays, sample populations, and analyses are implemented and institutional issues are corrected), then I believe that occasional unforeseen replication outcomes may accurately depict the complexity of the answers we seek, the illnesses we investigate, and the reality in which we live.