dickdiamond.com

Why I Continue to Hate Surveys

Following up on my earlier post regarding the evils of polling... MSN has an article that purports to uncover the "bottom 10" worst customer service companies. If you take the list at face value, the only conclusion you can draw is that large banks and large phone and Internet companies have poor customer service. No initial surprises there. But the fact that I'm at least a part-time customer of up to seven of these companies--and that only one has stood out in my mind as having truly bad service (Time Warner)--set off some warning bells. Do they seem particularly bad at customer service simply because they have so many customers to complain about bad customer service?

If you dig into the methodology of the survey, you'll find it's a little more complicated than a straight ranking, and that in fact there's no "top 10" to go with the bottom 10. Here's why:

Right off the bat you have a self-selection bias inherent in asking the initial group only for bad customer service experiences and then picking the top 20. If you asked the same group for their best customer service experiences you'd likely get a substantially similar group, simply because so many people are customers of these same 20 companies.

Zogby tried to clean it up a bit by offering a full scale of response ("excellent," "good," "fair," "poor," "not familiar," "not sure"), but the damage is already done. In fact, by discarding the "not sure"s and "not familiar"s they actually increased the bias against large companies. This sort of downward divination can only serve to exaggerate the response biases in both groups. So this survey ends up telling you very little about what to expect when dealing with these companies, and it gives you no positive information at all, since no data was collected about good customer service experiences.

Bottom line: Even a company with years of experience applying science to customer satisfaction research can find it challenging to come up with a robust methodology. And that's presumably without the influence of the tabloid-style inflammatory agenda that MSN has.

Labels: media, microsoft, rant

posted by Rich @ 4/30/2007 12:40:00 PM