Over the past couple of weeks, Bond Papers has been commenting on public opinion polls and especially on how those polls are described.
Last week, we specifically singled out some of the polls on the New Brunswick election, at least of which was touted as showing Bernard Lord's Progressive Conservatives in a significant lead over the Liberals. The polls were conducted in mid to late August.
Two polls taken in September - one by Corporate Research Associates and another by Bristol Group's Omnifacts - showed the racer as too close to call.
So what happened?
Absolutely nothing.
Both the August polls had margins of error of over four percentage points at the 95th confidence interval. What that means is that if you had conducted one hundred polls at the same time,using the same methodology, 95% of the results would have fallen either four points above or four points below the number shown in the poll actually reported.
It doesn't take a rocket scientist to notice that the total range of error in each instance is well above eight points. Therefore any gap between the respondents that is - say - eight points or less is functionally meaningless if you are trying to figure what is going on in the election.
Bristol/Omnifacts' poll in August showed the Liberals and the Conservatives seven points apart and a margin of error of 4.4%. Now think about that. The difference between the two parties was the better part of two full percentage points within the range of variation. (Range = 8.8%; difference = 7%)
Savvy readers - who are not rocket scientists - likely picked up on that and hopefully, some of the posts here on polling have helped people appreciate that simple thing.
Leaving aside the issues in conducting the polls themselves, we still have to wonder why the research company itself will report things as if this little issue with the margin of error didn't exist.
Like Corporate Research Associates.
In late August, CRA issued a news release saying that "[t]he ruling PCs lead the opposition Liberals in voter support at the beginning of the 2006 provincial election campaign...".
Simply put, that's a claim CRA couldn't make base don their own polling results. The question on Bernard Lord's personal popularity wouldn't really have given them an extra piece of information that helped interpret the party support number as they did. CRA showed a 10% difference between the Liberal and Conservative leaders, with Lord in the lead. But if you stop for a second, you realize that with an error range of 8.8%, a difference of 10% between two leaders isn't very much to go on. After all, drop Lord's figure by the margin of error and goose Liberal Shawn Graham by the same amount and you have the two of them respectively at 35.6% and 33.4%.
Now before the partisans start crawling out of the wood work, realize that the scenario just described is just as likely to describe the situation during the polling period as any other, within the margin of error range for the poll.
Pointing out the wide range of possible outcomes wouldn't support the main purpose of these media polls, though, and that's getting media attention to help market the research company itself. There's nothing unprofessional, unethical or anything else wrong with what's being reported by the companies. We just have to be slightly more sophisticated information consumers to avoid being misled on a simple point.
With all that behind us, it's more interesting to actually take a hard look at the Omnifacts poll results, released last Friday. Omnifacts tells us something that is both accurate - for their results - and also a good headline grabber, namely that going down to polling day, the election is neck and neck.
Omnifacts also tried to probe people who might be leaning in one direction or another. That helps trim down the self-described "undecideds".
Here are the questions, as asked:
1) If a provincial election were held today, which party would you most likely vote for?
2) If undecided or not sure, is there one party you are leaning toward?
Now let's take a look at some specific results. For the first question alone, Omnifacts got these answers (n=1,065):
PC 30%
Lib 29%
NDP 5%
Oth 1%
Und 27%
Ref 8%
It's a dead heat among decided voters. But look more closely. Undecideds are at 27% and another 8% flat-out refused to answer the question. Put another way, that means the single largest category the largest response category was among undecideds and "no answer."
The next table presented by Omnifacts just adds in those who were "undecided" but who claimed to be leaning one way or another. The PC's picked up another 16%, the Liberals 17% and the NDP 2%, although note in the news release, Omnifacts shows the margin of error on this data table (with "leaners" included) runs between 6% and 9%.
Holy gap, Code Man.
A week ago, someone trying to predict an election outcome based on reported poll results would have mistakenly assumed Bernard Lord would sweep back to power.
This week, the world seems to have changed dramatically.
Actually it hasn't. The poll results over the past four weeks or show have showed essentially the race is both very close and that a significant number of New Brunswickers were either genuinely unsure of how they would vote - if they would vote at all - or were unwilling to tip their hand.
Anyone who tried to predict seat counts based on goofy ideas like distributing the "undecideds" like the decideds or trying to blow them off as being people who wouldn't vote likely would have missed what is essentially the real story of the New Brunswick general election:
the voters will decide.