Other than John Miller’s excellent piece in the Journal today, there’s also a piece by Sharon Begley on the problems with opinion polls. It’s a very useful primer on a problem I’ve heard about from pollsters and the like for years and it’s only going to get worse. Alas, the article’s behind their subscription firewall. But two points worth noting:
First, the plus/minus margin of error is the most inconsequential of the problems built into opinion surveys. She explains:
Don’t be fooled. For all its apparent precision, the plus-or-minus statement bears little resemblance to how accurately a poll reflects the opinion of voters. It is error of a completely other kind that trips up polls.
(Math-averse readers are allowed to skip this paragraph.) The sampling error represents the range of possible outcomes from taking a random, representative slice of the population. For practical purposes, it equals one divided by the square root of the number of people surveyed. If you poll 1,600 people, then the sampling error is 1/40, or 2.5%.
Second, the real problem with polls is their non-sampling error. These are the people who don’t get asked questions because they are either unavailable or unwilling. Surly people, for example, are radically under-represented in public opinion polls. If certain attitudes are over-represented or under-represented among the surly, what are pollsters missing? Pollsters randomly dial phone numbers, but since rich people have more phone lines, are they over-represented? Pollsters won’t call cell phones, but young people increasingly have only cell phones. etc etc. Anyway, it’s a useful piece.