... if you leave out certain questions.

I was listening to ABC News on the radio as I was driving home. They were discussing this poll and said something like, "A new poll shows there may be trouble ahead for President Bush."

After that rousing lead-in they cited several issues in the poll in which people, by margins of like 4-8 points, gave opinions which would, indeed, seem to go against the Bush administration. No where in the story was it mentioned that the *very same poll* shows Bush ahead by six points. (The margin of error, apparently, was 3%, so, barring rounding differences, this would appear to be a statistical dead heat.)

Yes, it's true that the results of the poll could be interpreted as showing that some of Bush's support is soft. And yes, from that you could infer, possibly, that Bush is "in trouble." It seems to me though that in a report on what a poll might mean for the candidates, it might be worth mentioning what the results for the question, "Who will you vote for?" were.

Doesn't it?

Update: Here's a related story, along with the best headline of the day.

More: And just to be fair, I'll point out that Drudge's link to the Yahoo News story about the same poll describes Bush as having a "solid lead". That's not exactly true. The poll shows Bush up by 6% among "likely" voters. According to the ABC News story, the poll has a 3% margin of error among likely voters. That means, to simplify greatly, that the pollsters believe they have pegged each number to within 3%, give or take, of the actual values among the general popluation. This poll shows Bush up 51-45. However, with a 3% margin of error, it's possible that the true value is 48-48. In circumstances in which the margin of error allows for the possibility that the two possibilities are actually equal, it's not possible to confirm statisically that either party is actually ahead. Thus, Drudge's headline is sensationalism at best.

However, that doesn't mean that the poll is meaningless. After all, that 3% margin of error also means that it's *possible* the true lead for Bush is 54-39. I seriously doubt that he's doing that well, but when you combine this poll with most of the others I've seen over the last 3 weeks or so that show Bush up by any where from 4-10 points, it provides compelling evidence that Bush is, indeed, ahead in the popular vote.

All this of course leaves aside the issue that nationwide polls are often useless in determining a winner because the "popular vote" has absolutely nothing to do with deciding who wins. In our federalist system, as employed in the Electoral College, it's possible for a candidate to "lose" the "popular vote" by millions and still be elected President. And no, I don't see anything wrong with that.

On a related rant, my college roommate and I were discussing the other day our frustration that stories about political polls rarely include the degree of confidence. If you don't know what this is, I'll try to keep this simple. (Please note that in my discussion of statistics I'm flying by the seat of my pants here. It's been *years* since I've actually done any but the simplest of statistical calculations. There's a plus of not being an auditor!) Basically, when a statistician calculates the margin of error of any poll, they also calculate how likely it is that the true value falls within the margin of error. This is known as the degree of confidence and in commercial polls the degree of confidence used is usually between 95-99%.

What is often not fully appreciated is that every poll has multiple margins of error, depending on how confident you wish to be in the margin. And the higher degree of confidence you use, the wider the margin of error becomes. So for the same poll you might find that you could have a 90% degree of confidence that the true value fell within a 2% swing. You might be 95% sure that it fell within a 5% swing. To get up to 99% you might need a 10% margin of error to reach that degree of confidence. For all practical purposes, to reach 100% your margin of error would have to be large enough to cover the entire possible spread. (If your poll showed 50-50, you'd need a 50% margin of error.)

My understanding is that political polls generally use 95%, but I can't remember ever seeing a story about a political poll that actually came out and said so. Certainly, if someone wanted to try to use a poll to affect public opinion in the short term, they could adjust the margin of error to get the results they wanted (assuming that the initial results were close to what they wanted). Since the degree of confidence is rarely published, who would know?

For instance, if your candidate was ahead, and you wished to demoralize the other side, you could increase the degree of confidence, thus shrinking the margin of error. This would give the impression that your candidate's lead was more solid than it actually was. Likewise, if you were behind, you could decrease the degree of confidence. This would make your candidate appear to be closer, statistically speaking, than he really was. This *might* then be used to help rally the base.

Of course, if you gamed the system too much and started coming up with margins of error way above or below those generally seen in political polls, people would probably notice. But since the degree of confidence is rarely released, subtle attempts to game the system might go unnoticed in the short term.

## Recent Comments