Sunday, April 24, 2016

Lies, damned lies and statistics

I was randomly browsing the TED Talks website the other day when a trending video caught my eye: Lies, damned lies and statistics (on TEDTalks). It's a short video, only 6 minutes, so I highly recommend you watch it.

In case you're short on time or lacking in headphones, I'll summarize for you. The speaker, Sebastian Wernicke, analyzed the information contained in all of the TEDTalks on the website (c. 2011) which at the time amounted to more than a week's worth of video, or over 1.3 million words when transcribed.

His goal in this analysis was to discern if you could reverse engineer the best possible TEDTalk based on three things: the topic, delivery, and the visuals. This included things like the words most commonly used in highly rated talks, the colors people wore during their talks, if the speakers had glasses or not, the length of the talks, etc, etc.

First of all, the topic you speak on is highly important for whether your talk will be successful or not. Success in this case was ranked based on comments, favorites, and shares. For instance, "you" "happiness" "brain" "French" and "coffee" were the words most often repeated between top talks. Wernicke therefor concludes that if you were to present a talk on those five things, it would be a real winner.

Additionally, he advises that if you want to have a popular TEDTalk, you should speak for the maximum amount of time you're allowed. Typically TEDTalks are quite short, but the most favorited talks tend to be longer in length in most of the categories that the website divides them into: except if you want a talk highly ranked as beautiful, funny, or ingenious.


On the visual side of the presentation, if you want to have a successful talk, statistics on the visuals of the presenter suggest that you should have longer than average hair, wear glasses, and be slightly more dressed up than the average presenter.


Tying into that, Wernicke suggests that "setting the mood" on stage through color highly correlates with the rankings of talks in different categories. Blues, grays, and often greens are favored in most categories, unless you're trying to be persuasive, in which case red is the top color. 

Obviously, this kind of analysis is pretty bogus. Sure, length and topic certainly relate to the popularity of a talk, but do you really think the color someone wears matters significantly? From the title (and the lack of description of his methods) I'm almost certain all of these "statistics" are made up, but that's not why I wanted to discuss this TEDTalk. What struck me when I watched this video was how ludicrous it seemed to scour a database, pull bits and pieces of information from it, and compile it into a cohesive measure of an output (ie success, in this case). There are so many more reasons that influence why a TEDTalk is successful, or isn't, that aren't accounted for in this "study". 

Translating that to science, I put it into the context of hypothesis driven research. We begin a project by asking a question. We set out to test the possible answers to that question, using statistics to guide the design of the experiment and tell us if we were successful in answering it. What we can't do is gather data willy-nilly, sit down and pull bits and pieces together until it tells us a story that we think we want to hear! Some of the best scientific inventions have been accidents, so I'm not saying exploration, creativity and invention don't have a place in the lab, because I really believe they do. But when it comes time to actually answer an important question, we have a responsibility to do it the right way and not make ridiculous inferences.

You don't have to wear a blue shirt, glasses, and give an 18 minute long talk on "making your brain happy with French coffee" to have a successful TEDTalk. Hopefully it's equally obvious that you can't jump to conclusions about the cause of an outcome in science. 

7 comments:

  1. I wonder if he's related to the dude who characterize the Wernicke's area? Anyways...

    I always wonder what makes a presentation good from bad. Is there a formula for good presentation? But I agree, there is definitely a lot of confounding variables here. I think what he should have done was control for the reputation of the speaker: i.e. what institution they work for (Harvard compared to other schools), how many degrees do they have, how many books they have published, etc? I think if he had done that, this "study" would have been very interesting. But I have heard anecdotally that red is an appealing color for most people and it exudes "power". So maybe there is something there? But essentially, he should have controlled for these variables that I have mentioned. I should watch this TED talk at some point. Thanks for the recommendation.

    ReplyDelete
  2. I wonder if I could reverse-engineer a successful dissertation...

    In a way, this reminds me of the dreaded grant writing process, where we try to guess what the NIH or other money-granting institution will want to hear. We try to use the right buzz words and make our proposal exude relevance. For the most part, I think we all have good intentions and fully believe in the importance of our research, but there are times when I think we begin to lose focus and cater too much to what these institutions want (much like one might "cater" to the TED community).

    I also enjoyed what you wrote about the research design process. We can be more successful scientists if we methodically test our hypotheses instead of working backwards from the results. Nice post, Emily!

    ReplyDelete
  3. Reading this reminded me of the "subreddit simulator" feature on reddit.com, where bots use word associations of the top posts from the day's current events to generate their own posts. All of these posts seem to be based on what words commonly follow other words in top posts, but the resulting sentences are often comically nonsensical (from today: Bill Clinton become Bernie Sander's loses to Hillary, Trump should choose Martin Shkreli for Securities Fraud, Seize Wu-Tang Clan up as Chewbacca and see if that helps).

    I think analyses of this type, though not hypothesis-driven in the scientific sense as we're used to, reflect a broader trend in the development of "AI." Correlation-based insight has led to a lot of advances in voice-recognition and human-computer interactions, but whether that can be considered artificial intelligence is still up for debate..

    ReplyDelete
  4. Reading this reminded me of the "subreddit simulator" feature on reddit.com, where bots use word associations of the top posts from the day's current events to generate their own posts. All of these posts seem to be based on what words commonly follow other words in top posts, but the resulting sentences are often comically nonsensical (from today: Bill Clinton become Bernie Sander's loses to Hillary, Trump should choose Martin Shkreli for Securities Fraud, Seize Wu-Tang Clan up as Chewbacca and see if that helps).

    I think analyses of this type, though not hypothesis-driven in the scientific sense as we're used to, reflect a broader trend in the development of "AI." Correlation-based insight has led to a lot of advances in voice-recognition and human-computer interactions, but whether that can be considered artificial intelligence is still up for debate..

    ReplyDelete
  5. While I agree that the premise of this TED Talk is mainly a statistical fishing trip, I don't think that we should necessarily discount all the data out of hand. Surely, the study was designed for the amusement of the audience, but he seemed to know his audience well enough to understand that they would not be amused by a poorly constructed flight of fancy. Since the video was short and the platform was ill-suited to explaining every detailed of his potentially detailed statistical analysis, I'm willing to give him the benefit of the doubt that he made an effort to apply genuine properties of statistical analysis to his whimsical problem. As you point out, properly conducted controls and replicates are surely important for a real experiment, but we'll just call this little venture a pilot study to take a peak at anything of note.

    That being said, I found some of his results provocative. Human perception is incredibly powerful, especially color perception. While I too have heard that people react differently to various hues, I wanted to see if this urban myth has a leg to stand on. Just by running a quick search through PubMed, it seems that there has been a lot of work done on how people relate to and integrate color in their lives. Humans seem to perceive color differently based on context of their immediate surroundings, associate color preferences closely with the context of different objects (your all-time favorite color, your favorite t-shirt color, and favorite wall color are rarely the same), and use color in attracting (or dissuading) a potential mate (Just a note: the urls for these articles are listed at the bottom, since I cannot seem to integrate hyperlinks into comments). Our relationship to color is incredibly dynamic, powerful, and apparently hard to pin down in absolute completeness. I would not be surprised if we have distinct associations of particular hues with 'inspiring' or 'informative'.

    Since the TED speaker took a moment to point out the association between red hues and 'persuasive', I would like to take a moment to point it out too. In the second link provided below, the study found that red consistently scored as people's favorite color across all categories. This may not be surprising given that apes - as distinct from most other mammals - evolved cones to detect red wavelengths, which gave them an advantage in identifying fully ripened, nutritionally optimal fruits. Red in particular has strong significance in sexual display, and can be used to make a mate more or less attractive.

    Red, as distinct from all other hues that we can see, is particularly charged to us. It may signify something of great importance, or great appeal, on a subconscious level. And by utilizing this color in a powerpoint presentation, you might be able to instill these feelings in your audience. By employing the power of the color red, you may be able to convince people that your talk is in fact all the more persuasive. Of course, this is extrapolation and speculation, and detours away from the realm of statistics. But as someone mentioned, exploring new and unconventional fields is part of the excitement of science. And just because the premise may be unconventional and the findings unexpected, such preliminary studies may be just the thing to lay the groundwork to open up new (properly conducted) fields of research.


    Links to the articles referenced above, for your convenience:
    http://www-ncbi-nlm-nih-gov.proxy.library.emory.edu/pubmed/27060181
    http://www-ncbi-nlm-nih-gov.proxy.library.emory.edu/pubmed/27022909
    http://www-ncbi-nlm-nih-gov.proxy.library.emory.edu/pubmed/26960135



    ReplyDelete
  6. While I completely agree with your analysis that his analysis is completely bogus, this type of "what is needed to be successful" analysis could definitely draw some interesting "conclusions" on a lot of other fields -- off the top of my head, successful grants and published papers would be interesting to "analyze." I'm sure the words used -- significant, novel, important, etc -- would show a very clear pattern, and it would be interesting to analyze this pattern based on sources of grants and journals. Does Nature prefer some words in their articles more so than Science does? While not exactly a science, I think the results of this would be incredibly entertaining to look at -- and potentially useful when we go to publish in the future.

    ReplyDelete
  7. Thanks for sharing this TED talk! I really enjoyed the talk and your blog as well. I would agree that his analysis is bogus and the inference he made to become a good or bad speaker is not applicable. Two possible problems as I can see for the inference. One lies in how to define "good" vs. "bad" talks. Does the most "popular" talk the best ones? I'm afraid not. Some people are just famous and adored by more audiences. Second problem is that he actually meddled too many factors in one "research". If we want to draw any conclusions, the hypothesis we generate is just like "two-sided" coins. We don't have so many choices. The speaker talks in a funny way and he didn't expect us to buy it anyway. He did provide some "clues" and food for thoughts on presentation and statistics. It's cute and I like it.



    ReplyDelete