Pffft...Luke...I Am Your (Updating) Factor!: A Short Guide to Bayesian Statistics and Bayesian Experimental Design

Bayes Theorem – remember that mentioned way, way back in Lecture 2? No, it isn’t some new age way of predicting who you’ll be romantically involved with this winter, but there is a field of inference that comes from this theorem that plays into Bayesian statistics, and that is a subfield of statistics many scientists should be paying attention to.

Up until this point, we’ve basically been learning more frequentist statistics than Bayesian statistics (i.e., heavy on the linear regression, chi-squares, correlations, less so on multiple comparisons, etc.). This is evident by our HistoryStats projects: we’ve been looking at the lives and work of some of the founders of frequentists’ school of thought like Neyman, Pearson, and Wald. How do we best describe these  frequentist statisticians? Well, let’s take a simple, intuitive analogy described by this StackExchange forum. According to “user28” having a frequentist frame of mind is like hearing the phone go off, referring to a model upon which helps you identify the area of your home that phone is going off to make the inference on where the phone is. Having a Bayesian frame of mind means you may have that model in mind, but you also take into account places where you’ve mistakenly left the phone in the past. Simply, frequentists believe that data is a frequency, or a repeatable random sample, while Bayesians believe that data is observed from a real sample. Furthermore, frequentists believe that parameters are fixed, whereas Bayesians believe the parameters to be unknown but can be described by probabilities. (So…that would make Fisher’s maximum likelihoods a closet Bayesian statistic, wouldn’t it?).

TJ gave us some great examples of Bayes Theorem applied to real life, like the probabilities in clinical trials with cancer treatments. However, we never really got to see how Bayesian inference affects the experiment’s statistics and experimental design.

To understand the experimental design, we need to understand exactly how experimental design is updated or modified by Bayes Theorem, generally. Let’s say you are going to flip a coin 10 times and you suspect a probability distribution to describe these coin flips. Therefore, h would represent the probability of heads, and p(h) would represent the distribution settled on prior to any coin flips. Then the coin is flipped and way more heads come up than usual, say 8 heads. By using Bayesian inference, we need to update our prior belief about the coin – it’s now unfair. So our new beliefs may be modeled like p(h|f) where f is the number of heads experienced in those 10 flips. This abstraction is read as “what is the probability distribution of heads given the number of heads resulting from 10 tosses [in this case 8]?” This seems like a reasonable update as we pare down our hypotheses to fit our experimental data. Mathematically, the update would look like p(h|f) = u(h, f) x p(h) where u(h,f) is an updating factor written out as u(h, f) = (l(f|h))/l(f) where l(f|h) is a likelihood function or the probability we observed 8 heads given the parameters we modeled in the beginning. The denominator of the updating factor is just the likelihood of the data under no conditions. Because Bayesian statistics doesn’t believe parameters are fixed, they can have conditions added to them. Therefore, the likelihood of the data can be written as an integral l(f) = ∫l(f|h)p(h)dh (this is similar to a general expectation value). The denominator turns out to be a weighted average of likelihoods across all possible parameters. Or simply, a ratio that is able to tell you what parameter values are most likely.

Darth Vader: crafty with a lightsaber and some conditional probabilities.

How does this play out in the lab? Let’s take a hypothetical animal trial where dose concentrations of many drugs are tested on large amounts of animals to test their potencies. The lab wants to apply regression analyses to the different drugs based on the specimen they inject the drug into. For experimentation of one drug, the experimental design included six equally spaced doses given to ten mice each; so, 60 animals to test a range of concentrations for one drug. The investigators measured the number of surviving mice one week after drug administration. It turns out that about 90% of mice died at high concentrations of the drug, while 10-20% died at low concentrations of the drug. After each of the experiments, maximum likelihood estimations were used to estimate an LD50 value (or the dose at which the probability of mice dying is 50%). As it turns out, the investigators used results from the first few sets of experiments to predict a distribution for following experiments, in anticipation of constructing an updating factor, as described above. In total, if 50 drugs are tested with similar experimental design, the investigators can use these 50 LD50 values as a sample from a distribution of LD50 values.

Overall, these Bayesian inferences and the statistics are mathematically rooted in Bayes Theorem. This theorem relies on conditional probability. These conditional probabilities make the system easy to update and a noteworthy design for scientists to consider -- because writing grant proposals on frequentist assumptions can be dangerous when we try to predict a model for data without any prior knowledge of the system.  

Does Misuse of Statistical Significance Give Us a Higher Likelihood of Immortality?

This past week, a letter in Nature was published by the Department of Genetics at Albert Einstein College of Medicine in New York. The letter’s title was pithy and shocking “Evidence for a limit to human lifespan.” GASP! “But this can’t be true,” some may say, “with advances in medicine, we’re only continuing to lengthen the human lifespan!” First off, that’s what I think is a bit of a teleological argument – assuming just because humans have certain intelligences about the world, our purpose is to expand life or live forever. There’s some sort of moral purpose that is implied in that answer. Anyways, that’s not the point, but it sets off an interesting point of debate

Maybe we, humans, aren’t supposed to live forever. We forget that we are but specks on the lithograph of time; we’re babies! Only 200,000 years has the species Homo sapiens existed. On a 4.5 billion-year old planet. Many things have lived before us, and many things will live after us. Perhaps this sounds a bit nihilistic, but I was happy to hear these reports from these geneticists. Humans are single-handedly taking the gauntlet to ourselves and our planet, making sure life, in any form, is going to have a hard time existing on a planet where the temperature is rising almost 2 degrees Fahrenheit each year. So yes, I was overjoyed, ecstatic, and relieved to hear we may be shortening the execution of our planet Earth -- until I heard these geneticists used statistics to come to their results. Oh, great, another potential mishap with statistics by those not formally trained in statistics, I thought.

The methods that the study employed were actually pretty simple. Most people who took a basic stats class could probably understand everything up until the cubic smoothing splines. The authors plotted the maximum reported age at death (MRAD) for 534 people over the age of 110 or what they call “supercentenarians” gathered from the International Database on Longevity. These supercentarians came from Japan, France, US and the UK. Two linear regressions were done for subsets of data between 1968-1994 and 1995-2006. The scientists found an increase in MRAD of 0.15 per year before 1995 (with r=0.68 and p=0.0007) and a decrease in MRAD of 0.28 years from 1995-2006. There was a r=-0.35 and a p-value of p=0.27 for the decreasing MRAD. The scientists didn’t expand on the statistical shortcomings of their decreasing MRAD points. The scientists conducted the same procedure with other data points, and found a similar overall trend – that is, a significant increase until breakpoint and an insignificant decrease after breakpoint -- but did not discuss the weak correlation or significance.

The initial criticisms of the work have been predictable, to say the least. Why didn’t the scientists discuss their dismal p-values? Moreover, some say that seeing a significant increase in MRAD then observing a decrease, even if it was insignificant, is still something. This is, however, an amateur cop out, in my opinion. There are so many more reasons to be critical of the work than just the terrible p-values, and it starts with study design.

The main figure in Dong et. al. and the bane of my existence these last four days...

First, it’s really not clear why the scientists decided to use the arbitrary breakpoint between 1994 and 1995. Clarification of this would be helpful, if not crucial, to understand, as the entire crux of the paper’s argument leans on this breakpoint and its subsequent data analysis (Figure 2a). It appears that the breakpoint was chosen arbitrarily to support their initial hypothetical claim that humans have reached a plateau in age advancement, and the linear regression decrease model in MRAD is used in a rhetorical sense, rather than in a strict statistical-sense of proving their ad-hoc null-hypothesis statistical test (NHST). Is it even right to use NHST here? I’d argue no. If their alternative hypothesis was that r=0, it’s increasingly hard to test effect sizes closest to zero – you probably need to have a larger sample size, and with this data, data that exists on the margins of collectability, that’s hard to do.

The authors do make an implied comment on the power of their statistical tests, noting that they probably don’t have a large enough sample size to make case for a more robust statistical model. To alleviate this, they applied a post-hoc sample expansion to include several RADs as the highest point, including the MRAD and the second through fifth RAD; this appears to be more or less a sneaky way of doing away with outliers to me. They concluded that the average RADs had not increased since 1968, and that “all series showed the same pattern as the MRAD.” But instead of applying an ANOVA to test their hypothesis about average RADs not affecting the means, they never reference any supplemental material. It appears that they just eyed it. Another suggestion would be to compare the slopes of the “plateau” regions by doing some sort of t-test.

In the end, the statistical procedures taken to prove their own point would have any data scientist’s head spinning; probably enough so that people just wouldn’t take a hard look at it for too long. This is dangerous, but we can’t say Nature hasn’t committed the crime before: many times, the journal has published funky stats simply because the title of the study was provocative (as demonstrated by our class this semester). Nonetheless, you have to tip your cap at the scientists who wanted to make a headline, they surely did it, then perhaps bonk them on the head with your cap and tell them to do better stats before they make such grand claims.

The color of the dress should give you confidence... intervals

A bit over a year or so ago, an otherwise unremarkable photo of a dress in poor lighting became an internet sensation when it was discovered that different people saw the colors of the fabrics in the dress in dramatically different ways. There were two camps with little middle ground in between: the white and gold camp and the blue and black camp. People, including myself, were adamant that the other camp needed to have their eyes checked. Personally, I saw the dress as white and gold, and thought that people claiming that they saw the dress as blue and black were part of some elaborate hoax set up by ophthalmologists to generate checkup business. To investigate this, I set up a poll and submitted the link to r/SampleSize, which contains a small subpopulation of the popular website Reddit that are interested in taking surveys. I received a total of 45 replies, 27 responses for white and gold and 18 responses for blue and black This indicates that only 60% of responders saw the dress as white and gold, which was certainly not the hallmark of some great conspiracy, unless the secret ophthalmologist societies were in on my small-scale survey. The 95% confidence interval calculated from these results for the proportion of the population that sees the dress as white and gold was 45.45% to 72.98%. Phrased differently, there's a 95% chance that actual value for the percentage of the whole population that sees the dress as white and gold is in the range of 45.45-72.98%.

Another poll regarding the dress colors was conducted on a more visible platform. This survey received 432 respondents, 172 who saw the dress as white and gold. This is a much smaller percentage (39.81%) than that observed in my poll. The 95% confidence interval from this survey came out to 35.30-44.50%. Comparing my results to the results from this survey, there are two different 95% confidence intervals that do not overlap. So what does this mean? One possible reason for the disparity could be due to response validity, or lack thereof. I had created my poll well before taking this statistics class, thus it fell victim to a few different flaws in design. For one, my sample population was not random nor representative of the total population. Posting the poll on Reddit alone limited my sample population to the part of the population that uses Reddit, which is demographically weighted towards young males that spend a lot of time in front of a computer. The poll on Survata included a much larger age range of participants, though data on gender and time spent in front of screens was not recorded. Additionally, I did not limit the poll to one response per person, so responses may not have been independent. I also did not provide a choice other than white and gold or blue and black, which the Survata poll did include a choice for "neither". This may have skewed responses, as perhaps respondents who didn't see either color scheme chose to select the color scheme closest to what they saw, or just selected the first option on the survey, which was white and gold. These errors within my poll violated several assumptions that are necessary when analyzing confidence intervals, and thus weaken the meaningfulness of confidence intervals in the analysis of the data from the poll. However, it should be noted that it's entirely possible that confidence intervals from two different completely valid data sets do not overlap, or even do not contain the true population value of the measured parameter, due to the nature of what the confidence interval truly means.

The infamous dress

Is bias a necessary evil?

Biases are ubiquitous. And because they are ubiquitous, we must embrace them for we cannot escape. This is especially true to science. I see biases as vital and necessary components of science. I often see bias portrayed as being bad--rightfully so as bias can really screw us over as portrayed by the articles I read. But from reading these articles, I wanted to find reasons for how bias can be good. Perhaps I am misunderstanding the connotation of the word "bias" or applying the word subjectively? No matter what, here are some reasons why I think bias could be a good thing:

1. Biases, aka hypotheses, are the impetuses for projects. Scientist must have a central belief, which are biased by our expertise, past life experiences, our colleagues, mentors, and etc, to which we frame our scientific questions. I believe that these biases provides the momentum for the creation of projects and propel discovery. Without our constantly changing biases science would not be in perpetual motion. 
2. Biases allow us to be more critical, allowing for the advancement of science. We are taught as scientist to always question what we see, what we read, and what we hear. We would not have such critical minds if we did not have a bias that something published is not always true or causative. I think that such skepticism pushes science forward. 
3. Biases force us to do better science. The point of publishing is share your discoveries with supportive evidences that try to minimize biases. Because we have these biases and want our results to be as objective as possible, we design "controlled" experiments. Thus, bias forces us perform scientifically valid experiments and analyze data that can best confirm our hypotheses.

Here are my thoughts on how bias is a necessary evil in science. Without them it may be hard to pose a scientific question, make it impossible to be more critical, and most importantly, perform and analyze truly honest and objective experiments. Since there are many examples of how bias can be detrimental to science, I just wanted to be a devil's advocate and provide some reflections about how bias can actually be good for science.

The problem is in the design of experiments

Here's another well-written lay article, this time from The New Yorker, on irreproducible findings, the way science is being conducted, and our inherent biases as human beings.

Money quote:

That’s why Schooler argues that scientists need to become more rigorous about data collection before they publish. “We’re wasting too much time chasing after bad studies and underpowered experiments,” he says. The current “obsession” with replicability distracts from the real problem, which is faulty design. [....] “Every researcher should have to spell out, in advance, how many subjects they’re going to use, and what exactly they’re testing, and what constitutes a sufficient level of proof. We have the tools to be much more transparent about our experiments.”

Indeed, "We have the tools." We've chosen not to use them for the last couple of generations.The design of experiments is a concept that was invented decades ago to minimize the very problem we're grappling with today. We've simply not taught it well or learned to use it or chosen not to use it, perhaps because our great ideas couldn't possibly be wrong.

h/t Ken Liu

How to become a qualified future Scientist

After viewing the video of deception and reading articles about unreality in science, I feel more pressured of becoming a qualified future scientist than ever.  This is good as it makes me, a graduate student in biomedical science, to think deeply about what's important to carry on along our way of pursuing a scientific career.

As mentioned in Dan Ariely's video, there might be a "fudge factor" which underlines our cheating behaviour.  If people signed the honor code before the test, their cheating behaviour was greatly reduced.  It signifies the importance of awareness.  The awareness that biased research might cause very "bad impact" should be beared in our minds before we design our research, during our data collection and at pre-publication status.  "Bad impact" might be the failure of a drug, the waste of time and cost into the research, or the damage to our reputation in the field. As a student in cancer research, attempting to find a "cure" to pediatric brain cancer, one "bad impact" might be creating the "false hope" to those young patients and their parents or family.

Other than being aware of possible bad impact, it is also important for us biologists to be armed with basic concepts of statistical knowledge. In Jeremy Berg's blog and the article of "Trouble at the lab", they mentioned that scientists tend to produce false positive data and barely publish false negative data, which is more reliable from the perspective of statistics.  This may explain for the most biomedical non- reproducible experiments.  If we knows better about statistics, we would at least not overstate our conclusions considering the possibility of producing false positive result is not that low.  Though no statement is 100 percent true in science as mentioned by Jared Horvath, there must be ways for us to improve our methodology in conducting and communicating science, for example, better interpreting our data.  Nowadays, peer reviews call more attention on statistical explanation, which might attribute to more involving of statistics in biological research.

Now realizing that we are under the age of "publish or perish", we should avoid being pushed to nowhere by the tides of publication stress.  A good start to change might be the awareness of possible bad impact on our society by uncareful research and the integrating of biostatistics in biomedical science.