[Update and correction: 1230 hrs. CRA has still not posted the news release and tables for the regional poll as of the update time.]
A new poll from Corporate Research Associates is making the rounds. This time, there's supposedly a report on how Newfoundlanders and Labradorians feel about the Harper Conservatives. The Conservatives and the Liberals are in a dead heat, according to the CRA poll.
You can find a story at vocm.com - although the thing is virtually incomprehensible - as well as coverage in the Telegram (not online) and CBC News. Let's just take the CBC version, though, as typical of what the poll purportedly contains:
Corporate Research Associates reported Thursday that 39 per cent of decided voters in the province would choose the Conservatives, if an election were held now. That's up from 31 per cent recorded in an August poll.
Right off the bat, there's cause to be a little curious if not downright suspicious.
The undecided, refused or won't vote categories combined apparently showed up as being 45%. Basically, even if the results are perfectly accurate, we are talking about relatively small movement in the electorate here. With numbers like that, even a party showing at 40% in a CRA poll is actually dealing with numbers less than 25% of the total electorate. CRA gives it results as a percentage of "decideds", incidentally and not all news reports noted the "undecideds".
But there's something else.
CBC and the Telegram are reporting provincial numbers that are part of the quarterly omnibus survey taken in November, with a sample size of around 400 and a margin of error of plus or minus 4.9%, 19 times out of 20. But vocm.com is reporting a poll of over 1500 people across Atlantic Canada with a margin of error at 2.5%.
The answer to your wonderment is that CRA is reporting both regional and provincial results simultaneously.
But wait a moment, sez you. Which margin of error do we take? For the numbers presented, they'd by the higher one if the numbers are actually for the province alone. For the region, they'd be the lower number.
Go back and look again. There seems to be some confusion among the news outlets over which numbers are which. Both vocm and CBC, for example, purport that Conservative support is on the rise in the province.
Okay.
The numbers - 31% moving to 39% don't add up to the reported poll results.
Neither of the last two CRA polls are available on the company website, but the earlier one for May is. Take a look at the numbers for Newfoundland and Labrador on the party support question The Conservatives sure weren't at 31% in this province according to CRA, that is unless their political fortunes were tied to the proverbial dog's stomach. They were at 17% in May and the poll before that - in February - they were at 49%.
Hmmm!
They reported regional figures as provincial ones. And if the news organizations had access to the tables of results or had taken the time to search them, they see something else.
[Update, my bad and other observations. First, the my bad. The numbers are provincial numbers reported accurately by local media in Nova Scotia and in this province.
Second, though, that means the Conservatives have indeed been up and down radically according to CRA numbers. That's a story in itself.
Third, here's a new point to note: the provincial numbers have a huge margin of error at plus or minus 4.9%. While CRA is reporting party support in this province at a near dead heat, the actual breakout could have the Liberals 11 points ahead of the Conservatives or the Conservatives could be eight points ahead!]
Look at the regional numbers - the ones reported by the media - and you can see the Conservatives are up where they were in May. The Liberals are slightly down. But for the February result, notice that the Liberals have actually gained while the Harper Conservatives? Well, they're up and down.
But wait.
It gets better, still.
Look at those sample sizes again. Quarterly omnibus. Newfoundland and Labrador. Sample size 402.
The quarterly omnibus results for questions on political parties for Newfoundland and Labrador had a sample size of over 800 when it was released last week.
Double hmmm.
That's odd.
Well, it is odd. It looks like those polls of 800 plus are actually two trips to the field with two separate samples, in essence two separate surveys with the results provincially glued together mathematically for the final tally. We'll leave it to the mathematicians in the audience to discuss the usefulness of that.
Triple hmmm.
Basically we have a poll here that news rooms are reporting with a jumble of numbers, mixing results for the province and margins of error. [Correction: as noted above the media are reporting the provincial numbers as provincial numbers.]
Then we have margins of error which, in the provincial results are high enough to make them of dubious value. Face it, even if you take the Newfoundland and Labrador numbers, the range of variation is 10 percentage points. That's a wide enough gap you could completely miss real changes in the electorate over time if there's enough systematic error in the polling.
Then we have - apparently - some question about how big the sample is for the omnibus. Is there one or are there several omnibi?
That gets really important if you're one of those people who follows the news and who accepts the CRA poll results for provincial political support. Tories, Liberals and New Democrat politicians all treat the polls results as gospel. media commentators and party supporters cite the Premier's overwhelming popularity as having something to do with what they are talking about at the time. CRA poll results have achieved an unprecedented level of currency and validity in local political discussion.
The problem is, you see, the CRA polls might not be accurate. The news reports usually describe the poll results accurately in and of themselves, however, they do not delve into whether or not the polls themselves are accurate.
| CRA | Telelink | Poll | CRA | |
| PC | 62 | 42 | 43 | 67 |
| LIB | 13 | 7.6 | 13.6 | 9.8 |
| NDP | 5.7 | 3.7 | 5 | 6.6 |
| UND | 18 | 31.7 | 38 | 18 |
| Ref | - | 14.7 | - | - |
The table above compares the most recent Corporate Research Associates polls on provincial politics with voting results in the October general election and the only other poll publicly available. That poll was conducted by Telelink for NTV during the election campaign.
The results in this table are presented to ensure that, as much as possible, comparisons are being made to the same sorts of numbers. For the polls, the results are given as a percentage of respondents not, as is usually the case, as a percentage of what are often erroneously called "decided voters". For the October election, the results are presented as a percentage of all eligible voters, not of voter turnout.
Notice in the table above that the CRA numbers for Liberal and New Democrat party support are almost dead on the actual results in the October general election. Likewise, the Telelink Progressive Conservative number is one percentage point off the actual result. [Note: as in the original post on the Telelink poll, we've dropped about 4% of the survey, being those who indicated they would not vote, and adjusted the remainder. The PC number from the results as reported by NTV had the PCs at 40.3%, which slightly more than the margin of error reported for that poll.]
As noted here in an October post, the Progressive Conservative voter support in the 2003 general election was 42% of eligible voters.
However, the CRA Progressive Conservative number is 19 percentage points off the actual result. Likewise, the category in the CRA poll that encompasses "undecided", "will not vote", and "no answer/refused" is 18% of those polled while the non-voter population on polling day (roughly analogous) is 38%.
Even if we look at the results as reported, that is, as percentages of decided voters, the results are no more comforting. CRA's August poll results showed the Progressive Conservative Party support at 76%. It came in at 70% during the general election, a variation of more than twice the margin of error reported for the poll.
In pollster terms, that's gigantic. in recent provincial and federal elections, it's not unusual to see polling firms report results of surveys with comparable error margins hitting the real popular vote number within one one percentage point or even one half of a point. Sure, they poll far more close to election day than the CRA or Telelink polls.
But think about it.
There wasn't anything that occurred between early September and early October that might have caused such a dramatic shift in local Progressive Conservative fortunes. If anything, the dismal performance by the Liberals and New Democrats should have boosted the Tory numbers.
Don't think the October vote isn't an anomaly either, compared to the CRA poll. CRA's numbers have been so consistent for so long that it is hard to imagine a CRA poll taken in October would have revealed such a dramatic shift of Tory popularity.
Nor would it be reasonable to think that there is some greater variation in voter choice between polling periods. Frankly, even allowing for the goosing, the CRA results have been too consistent with each other to think they peak up some sorts of freakish super-support for the Tories at the most convenient time.
Nope.
There's something amiss. Obviously.
The most likely explanation are issues in the survey design or in data collection. As good and professional as a firm is, errors do occur.
Given the consistency of the CRA results, it looks like something more than people in the call centre fudging results, guiding answers or falsely filling out forms. Those days are largely gone, given that most call centres are highly automated. Monitoring is strict and even things like question order can be varied by the computer, not the telephone operator.
It's more likely a structural problem in the way the survey is conducted or some aspect of the sample that skews results.
Something's off in the poll results and it isn't just the reporting of them.
-srbp-