Tiny bird headphones and testing model fit: An example use of the extra sum of squares F test

One of the main avenues of research in our lab is quantitative analysis of behavior. One behavior we are particularly interested in is sensorimotor error correction in songbirds. “Sensorimotor error correction” refers to the process by which sensory feedback (such as the auditory feedback of hearing oneself sing) is used to correct a motor behavior (such as singing).

To induce a sensory “error”, our lab fits birds with sets of miniature headphones. While the bird sings, a microphone in the cage records them, and sound processing software will artificially “shift” the pitch of the song up by a couple semitones. This pitch-shifted version song is then played back to the bird through the headphones, virtually in real time. To compensate for the “error” it hears in the auditory feedback, the bird will start singing at a lower pitch (note that if you artificially shift the pitch down, the bird will shift its pitch up).

Previous work had shown that birds will learn to compensate for pitch shift at a faster rate if the shift was small. For very large pitch shifts, they barely learn at all. It’s important to note here that while each bird has its own individual song, they don’t sing it exactly the same way every time. The pitch of a particular note will vary from rendition to rendition, in a normally distributed manner. A technician in our lab hypothesized that it wasn’t the raw amount of sensory error from the pitch shift that influenced learning rate, it was the overlap between the error and the distribution of pitches the bird typically sang at that mattered most.

To test this, the technician decided to use an extra sum of squares f-test.  He created two models (not discussed in detail here to avoid going too egregiously over the word limit, see the paper for more), one of which included parameters for both error size and overlap with the prior distribution, and one that just included error. Then, he took birds with a range of pitch distributions (young adult birds have more variability, and thus a wider distribution of pitches they sing at, while older birds have a narrower distribution). He then tested those birds with a variety of different error sizes via the headphones pitch shift.

The extra sum of squares F test is a way of comparing the fit of two nested models. “Nested” models are models which are identical, but one has additional parameter(s). The extra sum of squares test asks whether the additional parameters reduce the residual error or not. In the case of my labmate, he wanted to know whether there would be less residual error in the error + prior distribution model than the error-only model.

GraphPad’s help page offers another great example of nesting, which may be more intuitive to most biologists:

“If you asked Prism to test whether parameters are different between treatments, then the models are nested. You are comparing a model where Prism finds separate best-fit values for some parameters vs. a model where those parameters are shared among data sets. The second case (sharing) is a simpler version (fewer parameters) than the first case (individual parameters).”

The change in residual sum of squares is divided by the additional degrees of freedom for the extra variables, giving us a mean square. The mean square is then compared to the residual mean square from the full model. An F-test allows us to determine the likelihood of our result, assuming the null is true.

In my labmate’s experiment, the more complex model that included prior distribution overlap significantly reduced the residual error. Check out the full paper here. Here is a longer explanation of extra sum of squares F tests.