I'm looking to buy a used car in the very near future, and there are some important attributes I'm looking for: performance, a gorgeous interior, 2015, and, yeah, it should be made on the other side of the pond.
But then I look at my stipend provided by my PhD program
Out of curiosity, I take a peak at Consumer Reports' 2015 Annual Auto Reliability Survey. And I find the following plot:
A very informative plot. What they've done is survey subscribers on the vehicles they own, in total covering more than 740,000 vehicles. For each reported vehicle a reliability score is calculated and associated with that vehicle's model. A mean reliability for the model is calculated, and all the mean reliabilities for a make's models are averaged to generate the yellow dot, indicating the make's mean reliability. The blue spread indicates the range from the make's least reliable model on average to the make's most reliable model. But why didn't they use confidence intervals? After all, they do claim to be determining "predicted reliability scores."
A confidence interval is used when sampling a population for a value of some characteristic in order to estimate the population's mean for that characteristic. Since samples are being taken as opposed to observing every individual in the population, our calculations will not give us the true mean, but it will hopefully be close. A confidence interval informs us of the range of values surrounding our calculated sample mean that must include the true population mean with a certain level of confidence.
All that is needed to calculate the confidence interval is the sample size, sample standard deviation, and the sample mean. But I can see why Consumer Reports decided not to have their plot display that. As a consumer, once I see the average of the all the models' mean reliabilities, I don't need to see a spread telling me about the true mean. The true mean is not the goal of the reliability report nor the consumer who's in the market for a vehicle. The spread I want to see is the range from the make's least to most reliable model, indicating the variability in the reliability of the make's models. For example, if I was comparing Mazda's reliability to Subaru's, I would note that their means are very close, but the spread of Mazda's reliability is constrained and sits to the higher end of Subaru's reliability, which varies much more. This information might give me more confidence in Mazda's reliability than if I had simply compared the means. The confidence interval may resemble the worst-to-best spreads, but it simply does not mean the same thing. And once we have "outliers" that skew a spread in one direction (see Hyundai, BMW, Ford, Cadillac, Jeep), the fixed ± error of confidence intervals begin to have less meaning to the consumer.
Perhaps if I were a chemical engineer or computer scientist, that Maserati pictured above (no reliability data available, but who cares when a car is that beautiful!) wouldn't be such a distant possibility. But for now, I'm going to look at Toyota. And it probably won't be 2015, either.
Hey Akram, great points here and good luck in your car search. I like the point you are getting at concerning audience. That's something I think we don't talk about that much in biostats because, most of us are scientists and generally we are okay with sifting through a little extra data to be sure we get the full story (although the popularity amongst journal clubs of short papers from "letters" or "communications" journals would imply that we are not entirely invulnerable to settling for a little simplicity.
ReplyDeleteI think if I were a consumer and saw CIs listed or added onto the graph it could make it a bit more complicated and in the end obscure what I was trying to get out of the figure. Perhaps Consumer Reports should have a supplementary methods section online for each study they do?
In speaking about presenting stats and data to the public, we must also think about what the best way to present data simply, but still honestly, is. In this case I feel as though confidence intervals actually decrease the reader's confidence, although they should probably make them trust Consumer Reports even more!
After taking a medium-length look at this graph, I almost wonder if its absolute bullsh*t. As you said, "What they've done is survey subscribers on the vehicles they own, in total covering more than 740,000 vehicles. For each reported vehicle a reliability score is calculated and associated with that vehicle's model. A mean reliability for the model is calculated." But what in the world is a reliability scale? Did they ask for how many times they've needed to take the car to the shop? Did they correct this by where an individual lives, how much they drive, how old the car is, or something as simple and commonly not done such as regular oil changes? What in the world is a reliability quotient? There is no description. This makes me think that maybe they surveyed people to give a reliability score of 0 to 10, completely arbitrary numbers with absolutely no real life applications. What is a reliability of 0? Did your car stand you up on a date? What about a reliability of 10? Does the car just always, always have gas in it, and so is always reliable? So many questions, so little answers. More over, if the data was collected in such as a 0 to 10 scale, than a mean really means next to nothing! And, ugh, what are these numbers in the 100's? I call bullsh*t, and that this was probably funded by Lexus.
ReplyDelete