Lies, damned lies and statistics

I was randomly browsing the TED Talks website the other day when a trending video caught my eye: Lies, damned lies and statistics (on TEDTalks). It's a short video, only 6 minutes, so I highly recommend you watch it.

In case you're short on time or lacking in headphones, I'll summarize for you. The speaker, Sebastian Wernicke, analyzed the information contained in all of the TEDTalks on the website (c. 2011) which at the time amounted to more than a week's worth of video, or over 1.3 million words when transcribed.

His goal in this analysis was to discern if you could reverse engineer the best possible TEDTalk based on three things: the topic, delivery, and the visuals. This included things like the words most commonly used in highly rated talks, the colors people wore during their talks, if the speakers had glasses or not, the length of the talks, etc, etc.

First of all, the topic you speak on is highly important for whether your talk will be successful or not. Success in this case was ranked based on comments, favorites, and shares. For instance, "you" "happiness" "brain" "French" and "coffee" were the words most often repeated between top talks. Wernicke therefor concludes that if you were to present a talk on those five things, it would be a real winner.

Additionally, he advises that if you want to have a popular TEDTalk, you should speak for the maximum amount of time you're allowed. Typically TEDTalks are quite short, but the most favorited talks tend to be longer in length in most of the categories that the website divides them into: except if you want a talk highly ranked as beautiful, funny, or ingenious.

On the visual side of the presentation, if you want to have a successful talk, statistics on the visuals of the presenter suggest that you should have longer than average hair, wear glasses, and be slightly more dressed up than the average presenter.

Tying into that, Wernicke suggests that "setting the mood" on stage through color highly correlates with the rankings of talks in different categories. Blues, grays, and often greens are favored in most categories, unless you're trying to be persuasive, in which case red is the top color. 

Obviously, this kind of analysis is pretty bogus. Sure, length and topic certainly relate to the popularity of a talk, but do you really think the color someone wears matters significantly? From the title (and the lack of description of his methods) I'm almost certain all of these "statistics" are made up, but that's not why I wanted to discuss this TEDTalk. What struck me when I watched this video was how ludicrous it seemed to scour a database, pull bits and pieces of information from it, and compile it into a cohesive measure of an output (ie success, in this case). There are so many more reasons that influence why a TEDTalk is successful, or isn't, that aren't accounted for in this "study". 

Translating that to science, I put it into the context of hypothesis driven research. We begin a project by asking a question. We set out to test the possible answers to that question, using statistics to guide the design of the experiment and tell us if we were successful in answering it. What we can't do is gather data willy-nilly, sit down and pull bits and pieces together until it tells us a story that we think we want to hear! Some of the best scientific inventions have been accidents, so I'm not saying exploration, creativity and invention don't have a place in the lab, because I really believe they do. But when it comes time to actually answer an important question, we have a responsibility to do it the right way and not make ridiculous inferences.

You don't have to wear a blue shirt, glasses, and give an 18 minute long talk on "making your brain happy with French coffee" to have a successful TEDTalk. Hopefully it's equally obvious that you can't jump to conclusions about the cause of an outcome in science.