Showing posts with label model fitting. Show all posts
Showing posts with label model fitting. Show all posts

Saturday, October 8, 2016

Heteroskedasticity is the Statistical Concept We All Took for Granted

Heteroskedasticity. Hard to say, and often forgotten, but something that should be at least considered by many data scientists and people who work with data. Data is neither good, bad nor ugly, but it can take on such qualities when we analyze that data. Newly minted-PhDs can get their first R01 grants (good), scientists can waste a new grant (bad), or scientists can gain funding fraudulently (ugly). Therefore, how we think about and interpret our data is important step to determining the data’s fate. Heteroskedasticity is one of the concepts that can determine whether your model for your data is good or bad. However, before we explore heteroskedasticity, we need to look at statistical modeling, study design and analysis principles that precede it.

From the beginning, one imperative that seems to be eluding most scientists with data these days is Hyman’s Categorical Imperative. Never met or read any of Hyman’s stuff? Well, it’s just a fancy rule for stating what most data skeptics try to follow. Hyman’s Categorical Imperative -- a maxim coined by Professor Emeritus of Psychology at the University of Oregon, Ray Hyman – states that before you choose to investigate if a phenomenon is true, you should first determine if the phenomenon is real. Essentially, violating this imperative is the scientific equivalent of flippantly adding a trend line to data based on what the data appears to behave as. This is usually manifested in scientific studies as a linear trend line, often after least squares regression analysis, and can be seen in the following examples (with captions).

Figure 1 from AJP Endo Journal. Change in prolactin secretion vs. change in leptin secretion over 24hrs. The model’s fitting of least squares linear regression states that as leptin decreases/increases over 24hrs, prolactin secretion acts similarly. However, a quick scan of the data shows points in the plot populating mostly the -20 to 0 range in both axes. It consistently under-predicts positive and some negative changes in the hormones.

Figure 2 prepared by data scientists Ng and Blanchard. Deaths per state as a linear function of obesity rate. The trend line predicts a linear relationship between obesity rates and deaths per state. However, the model under-predicts some states with high death rates and low obesity rates. The data seems to be plotting an explanatory variable with no firm correlation to the dependent variable (more on this in the blog post).

Figure 3 from data scientists at TIKD app. Tickets per capita as a negative linear function of income. The trend line predicts that as income goes up, ticket issuance goes down. However, the model under-predicts all high incomes (>~$82,000) and even under-predicts low incomes (<$40,000), showing the influence that data point density can have on linear least squares regression models.

As we have discussed in class, least squares can be a good model for some data, but not all data (see above figures). The problem with least squares regression analysis is that it happens to be unusually influenced by where large amounts of data points accumulate or where unusual points scatter in Cartesian space, i.e., the variation in variance of the data points. The figures above are all good examples of this.

Generally, when data scientists want to look at how well a model works, they look at the residuals of the observations and the model’s predictions (data point value-model prediction value). If there is a pattern that appears due to the model’s application, then the model probably isn’t the best fit for the data. In other words, the model could be consistently over-predicting or under-predicting past or future points of data.

So what exactly is this unusually long word and what does it have to do with what we mentioned? Well, as described by Stats Make Me Cry, heteroskedasticity “refers to the circumstance in which the variability of a variable is unequal across the range of values of a second variable that predicts it.” Or, more simply, when the variance of data is conditioned on some other variable not shown. For the figures above, that could be some other variable like another hormone (Figure 1), an environmental source negatively affecting health in the state (Figure 2), the uneven policing of certain communities (Figure 3), or the cautiousness of drivers in older ages who just so happen to have higher incomes (Figure 3).

Why is it that heteroskedasticity gets no love in the classroom then? Well, it’s something that can typically be ignored when scientists analyze data. That’s because the presence of heteroskedasticity doesn’t truly bias the fitting of least squares regression, but it will lead to problems when analyzing variance, like in ANOVA tests. You can’t assume homogeneity of data without first analyzing heteroskedasticity of the data. Finally, heteroskedasticity might not get a lot of love from the life sciences, but it is a huge deal in the social sciences, especially in economics, and the subfield of econometrics, or the branch of economics that uses math to study the behavior of economic systems. Economists rely on homogeneity of data to give reliable confidence intervals and hypothesis tests. Maybe us life scientists should start looking for this then, too. So remember, the next time you assume homogeneity of data, make sure you use a logarithmic transformation to account for heteroskedasticity, or use these other cool tricks found on page 6 of Richard Williams’s lecture from Notre Dame. 

Thursday, March 31, 2016

Use Prism sample data to understand model fitting

Having trouble fitting a model to your data? Don’t worry because Prism is there to help. The program makes it easy to walk you through the basics of model fitting, and even includes sample data sets for you to practice on. Prism offers the following analyses that can fit lines and curves:
  • • Linear regression
  • • Deming linear regression (use when both X and Y variables are subject to error)
  • • Nonlinear regression
  • • Spline and Lowess (for curve fitting without selecting a model)
  • • Interpolating from a standard curve

For simplicity’s sake I’ll walk you through how to take advantage of linear and nonlinear regression model fitting using Prism. 

Linear Regression
Linear regression is used when you can describe the data using the equation y=mx+b. Knowing x, you can predict y using the goodness-of-fit line predicted by Prism. To start you can use the sample data Linear Regression – predicting the slope. You should get a data table that looks like this:


MinutesControlTreated
1.034.29.28.31.29.44.
2.038.49.53.61.89.
3.057.55.78.99.77.
4.065.65.50.93.111.109.
5.076.91.84.109.141.
6.079.93.98.134.145.129.
7.0100.107.89.156.134.167.
8.0105.123.119.167.180.
9.0121.143.134.178.192.175.
10.0135.156.198.203.234.

Next it is time to analyze the data: click on the analyze button and then select linear regression.

ControlTreated
Best-fit values
Slope12.42 ± 0.567917.96 ± 0.8684
Y-intercept when X=0.017.42 ± 3.47128.48 ± 5.446
X-intercept when Y=0.0-1.402-1.585
1/slope0.080500.05568
95% Confidence Intervals
Slope11.26 to 13.5916.17 to 19.75
Y-intercept when X=0.010.28 to 24.5517.26 to 39.69
X-intercept when Y=0.0-2.163 to -0.7630-2.432 to -0.8818
Goodness of Fit
R square0.94850.9448
Sy.x8.44013.13
Is slope significantly non-zero?
F478.5427.8
DFn, DFd1.000, 26.001.000, 25.00
P value< 0.0001< 0.0001
Deviation from zero?SignificantSignificant
Data
Number of X values1010
Maximum number of Y replicates33
Total number of values2827
Number of missing values23







When determining whether or not the model fits your data, take a look at the R2 value. This is called the coefficient of determination and provides you with an idea of how well the best-fit line fits the data. The closer the R2 value is to 1, the better the fit.  In this example, we see that R2 is equal to .9485 and .9448 for the control and treatment group respectfully. The high fit of this model is further more confirmed by the graph in which there is little deviation of the actual from the expected values of Y according to the model.

Nonlinear Regression

Nonlinear regression is used when the predicted relationship between x and y is not as simple as a linear line. These models use different functions to derive a goodness-of-fit line and may depend on multiple independent variables. We’ll use an enzyme kinetics model to see how nonlinear regression models can be used to fit your data. Use the sample data provided by Prism titles Enzyme Kinetics – Michaelis-Menten. The data table should look like this:


[Substrate]Enzyme Activity
2.265.241.195.
4.521.487.505.
6.662.805.754.
8.885.901.898.
10.884.850.
12.852.914.
14.932.1110.851.
16.987.954.999.
18.984.961.1105.
20.954.1021.987.


When analyzing the data, use the Michaelis-Menton model. The results of the Prism analysis are as follows:



Enzyme Activity
Michaelis-Menten
Best-fit values
Vmax1353
Km5.886
Std. Error
Vmax75.93
Km0.9498
95% Confidence Intervals
Vmax1197 to 1509
Km3.933 to 7.839
Goodness of Fit
Degrees of Freedom26
R square0.9041
Absolute Sum of Squares170343
Sy.x80.94
Constraints
KmKm > 0.0
Number of points
Analyzed28




Notice, again that the R2 value is close to 1, and that when you look at the generated graph that the actual values deviate very little from the expected. To show you that this model fits the data best, let’s see what happens if we had tried to fit a linear regression model to the data.

Best-fit values
Slope35.79 ± 4.456
Y-intercept when X=0.0408.6 ± 55.70
X-intercept when Y=0.0-11.42
1/slope0.02794
95% Confidence Intervals
Slope26.63 to 44.95
Y-intercept when X=0.0294.1 to 523.1
X-intercept when Y=0.0-19.32 to -6.649
Goodness of Fit
R square0.7128
Sy.x140.1
Is slope significantly non-zero?
F64.53
DFn, DFd1.000, 26.00
P value< 0.0001
Deviation from zero?Significant
Data
Number of X values10
Maximum number of Y replicates3
Total number of values28
Number of missing values2

As you can see, the R2 value significantly decreases, and when you compare the linear to the nonlinear goodness of fit line, you see that there is much more deviation from the actual observed values from those that are predicted. That is why fitting the data to the model is so important, because if we use a model with poor fitness we are unlikely to make accurate predictions about the dependent variables from independent variables.

A more thorough walk through, as well as model fitting examples using the other analyzes provided by Prism, click here.