Bias: an unavoidable word related to Science and Research

Research bias, defined as the process where the scientific researcher influences results for a specific outcome, is a loaded phrase that has become an important topic to address in scientific studies. Unfortunately, the cyclic process necessary to become a successful, funded scientist leaves many opportunities for the scientist to introduce bias into their research. This includes, but is not limited to, research design, hypothesis development, previous research results, choice of analysis, and getting a grant to keep a job.

A relatively new field in science, mainly because of advanced technology and resources, is big data research. Big data research, regardless of subject, relies on the scientist either to process data with previously generated algorithms/codes or to create their own processing method. When picking or creating the “correct” algorithm or code to use, it is easy to unconsciously choose a method that best agrees with one’s proposed hypothesis or desired results. For example, it is hard to decipher whether the data has no effect because there was indeed no effect or if the wrong algorithm/methods of analysis were used.

There needs to be accountability. One way to do this, specifically with big data research, is by sharing raw data, coding algorithms, and specific data analyses methods through open source websites such as GitHub: Science. Not only will this encourage other scientists to reproduce results, it will also create an environment that encourages collaboration. If raw data and codes are made available through open source, anyone can provide feedback or ways to better the analysis process. Another important aspect is allowing people to upload or provide data on negative results. In most cases, negative results are not publishable. This would give a research group an opportunity to get feedback from other people in the community on their negative results and perhaps even help others that are pursuing the same research question. 

Now, more than ever, it is necessary to expect experimenters to produce unbiased and reproducible results. We have the resources to do this. It is up to the science community to create new standards that allow for accountability and reproducible results while encouraging collaboration. 

Heteroskedasticity is the Statistical Concept We All Took for Granted

Heteroskedasticity. Hard to say, and often forgotten, but something that should be at least considered by many data scientists and people who work with data. Data is neither good, bad nor ugly, but it can take on such qualities when we analyze that data. Newly minted-PhDs can get their first R01 grants (good), scientists can waste a new grant (bad), or scientists can gain funding fraudulently (ugly). Therefore, how we think about and interpret our data is important step to determining the data’s fate. Heteroskedasticity is one of the concepts that can determine whether your model for your data is good or bad. However, before we explore heteroskedasticity, we need to look at statistical modeling, study design and analysis principles that precede it.

From the beginning, one imperative that seems to be eluding most scientists with data these days is Hyman’s Categorical Imperative. Never met or read any of Hyman’s stuff? Well, it’s just a fancy rule for stating what most data skeptics try to follow. Hyman’s Categorical Imperative -- a maxim coined by Professor Emeritus of Psychology at the University of Oregon, Ray Hyman – states that before you choose to investigate if a phenomenon is true, you should first determine if the phenomenon is real. Essentially, violating this imperative is the scientific equivalent of flippantly adding a trend line to data based on what the data appears to behave as. This is usually manifested in scientific studies as a linear trend line, often after least squares regression analysis, and can be seen in the following examples (with captions).

Figure 1 from AJP Endo Journal. Change in prolactin secretion vs. change in leptin secretion over 24hrs. The model’s fitting of least squares linear regression states that as leptin decreases/increases over 24hrs, prolactin secretion acts similarly. However, a quick scan of the data shows points in the plot populating mostly the -20 to 0 range in both axes. It consistently under-predicts positive and some negative changes in the hormones.

Figure 2 prepared by data scientists Ng and Blanchard. Deaths per state as a linear function of obesity rate. The trend line predicts a linear relationship between obesity rates and deaths per state. However, the model under-predicts some states with high death rates and low obesity rates. The data seems to be plotting an explanatory variable with no firm correlation to the dependent variable (more on this in the blog post).

Figure 3 from data scientists at TIKD app. Tickets per capita as a negative linear function of income. The trend line predicts that as income goes up, ticket issuance goes down. However, the model under-predicts all high incomes (>~$82,000) and even under-predicts low incomes (<$40,000), showing the influence that data point density can have on linear least squares regression models.

As we have discussed in class, least squares can be a good model for some data, but not all data (see above figures). The problem with least squares regression analysis is that it happens to be unusually influenced by where large amounts of data points accumulate or where unusual points scatter in Cartesian space, i.e., the variation in variance of the data points. The figures above are all good examples of this.

Generally, when data scientists want to look at how well a model works, they look at the residuals of the observations and the model’s predictions (data point value-model prediction value). If there is a pattern that appears due to the model’s application, then the model probably isn’t the best fit for the data. In other words, the model could be consistently over-predicting or under-predicting past or future points of data.

So what exactly is this unusually long word and what does it have to do with what we mentioned? Well, as described by Stats Make Me Cry, heteroskedasticity “refers to the circumstance in which the variability of a variable is unequal across the range of values of a second variable that predicts it.” Or, more simply, when the variance of data is conditioned on some other variable not shown. For the figures above, that could be some other variable like another hormone (Figure 1), an environmental source negatively affecting health in the state (Figure 2), the uneven policing of certain communities (Figure 3), or the cautiousness of drivers in older ages who just so happen to have higher incomes (Figure 3).

Why is it that heteroskedasticity gets no love in the classroom then? Well, it’s something that can typically be ignored when scientists analyze data. That’s because the presence of heteroskedasticity doesn’t truly bias the fitting of least squares regression, but it will lead to problems when analyzing variance, like in ANOVA tests. You can’t assume homogeneity of data without first analyzing heteroskedasticity of the data. Finally, heteroskedasticity might not get a lot of love from the life sciences, but it is a huge deal in the social sciences, especially in economics, and the subfield of econometrics, or the branch of economics that uses math to study the behavior of economic systems. Economists rely on homogeneity of data to give reliable confidence intervals and hypothesis tests. Maybe us life scientists should start looking for this then, too. So remember, the next time you assume homogeneity of data, make sure you use a logarithmic transformation to account for heteroskedasticity, or use these other cool tricks found on page 6 of Richard Williams’s lecture from Notre Dame.