08 June 2011

The perils of polling stories

Not surprisingly, local media reports of Corporate Research Associates’ most recent quarterly poll got the news spectacularly wrong.

For example, the Telegram reported that “[s]upport for the provincial Conservatives has dropped to 57 per cent of those polled…”.

That’s not true.  The CRA release tells you that 57% of decided respondents to their poll felt that way.  There’s a difference between the people they polled and the people who they polled who made a choice about which party they’d support.

Meanwhile, CBC correctly described the party support numbers as being about decided respondents but for some reason they felt the need to add these gratuitous and unfounded editorial interpretations into their story:

The poll suggests the NDP could pose a considerable challenge to the Liberals in the Opposition benches, as the Oct. 11 general election draws closer.

Voters remain comparatively cool to Liberal Leader Yvonne Jones, who had an approval rating of 16 per cent, down from 18 per cent in March.

NDP Leader Lorraine Michael, by contrast, saw her approval jump from five per cent to 14 per cent.

So how does this happen? 

Well, for starters, CRA reports numbers as a share of decided responses, not all responses.    CRA doesn’t  explain what that means and reporters and editors typically wouldn’t know if it was fit to eat.  As a result the reports go off into the trees.  Not their fault, by the way.  The obligation is on CRA to accurately present its own information and ensure people know what it means.

Professional pollsters in the Untied States specifically reject this approach because it creates a false impression of what the poll found.  Not only does it artificially inflate some results, it actually discounts entirely a perfectly valid choice. 

People who are undecided have actually made a choice that, for the moment, doesn’t include one of the choices offered.  it’s like saying “none of the above.”

If you want to see this sort of misrepresentation in action, take a gander at the CBC report on CRA’s quarterly poll for New Brunswick.  Almost identical numbers to the ones in Newfoundland and Labrador.  Ruling Tories with a supposedly “commanding lead” of 56% and the Liberals and New Democrats in a “statistical tie” at 20% or thereabouts.

Small problem.

Look at the undecided number in New Brunswick and then look at the poll results as a share of all responses, not just “decided” voters.

Suddenly the Tories aren’t at 56%.

They are at 35%.

And the other two?  Somewhere in the low teens.

More people in New Brunswick are undecided than expressed a choice for the provincial Conservatives.  As for the opposition parties,  you’d have to use your imagination to see anything there other than a pair of parties that voters are barely noticing.

In Newfoundland and Labrador, a look at the unadulterated CRA numbers still tells you that the Tories are in front of the other parties.  You can also see a major league slide going on.  They were at 62% in August 2010, down to 51% in November, back up to 56% in February and now at 44%.

But the other parties?  As in New Brunswick, neither Yvonne nor Lorraine have captured any great attention yet.

But yes, sez you, there’s been this orange surge from eight percent up to 20%.

Yeah, not really.

For starters, the numbers are more like six percent to 15%. 

Still more than double, right?

Maybe.

Consider that the margin of error for this poll is 4.9%.  That means you can shift any number on the page up or down by five percent and still be within the range of possible numbers you’d get if you polled every eligible voter in the province.

So when you hear a reporter talk about a statistical tie or an NDP surge, odds are you have someone who learned everything they know about polls from being spun by masters of the torque.  Either that or they just read a badly distorted news release and relied on their own abilities to surmise what it means.  Doesn’t mean the reporter’s a boob:  just means the reporter doesn’t really understand what is going on.

But still, the NDP surged ahead, right?

Negative, again, there Ghost Rider.

The unadulterated CRA poll number puts the NDP at 15%, give or take five percentage points.  The Dipper machine could be at 10%, up a mere four percent from the February result. 

By the same token, they be at 19-ish%.  You can slide the Liberal or Conservative numbers up and down likewise and still be right. That’s what the margin of error means.  Plus when you consider that CRA results over the past seven years have been notoriously wide of the mark even without using the margin of error, you can pretty well bet these numbers are only vaguely close to what’s actually going on.

To give you a sense of what the results could be – take a breath for this one -  consider that at 20% of all respondents, either the Liberals or the NDP would be close to 30% of decideds in this recent survey. 

With the Tories at 44% of all respondents and 57% of decideds.

Not quite so commanding now, is it?  A pretty easy slide of another 10 percentage points, in other words, the kind of shifts we’ve already seen in the past six months, and the Tories could be looking at the shortest time in government since Confederation for any political party.

When you look at those sorts of possibilities,  it gets pretty clear to see that any  comments about the Tories having a a “commanding lead” are nothing more than total hogwash.  A relatively small shift in votes up or down for either of the parties and you’d be looking at Kathy Dunderdale potentially headed for the opposition benches come the fall.

Then there’s the second bit, namely the editorialising.  There’ll be a few New Democrats who’d love you to believe they are on the verge of a miracle here, but these numbers don’t support any such idea.  The reasons are pretty simple.

For starters, you can’t tell anything with one single point.  You need a couple of points to start making a trend. 

Beyond that, there is absolutely no sign of a gigantic shift of voter support to the NDP anywhere provincially, let alone in the metro region, despite federal electoral success over the past couple of years.  Provincial Conservatives are not happy with their Kathy but there’s a big difference between voting against Stephen Harper and voting against their local Tory candidate. 

That takes a major league piss-off of metro voters the sort you saw in the late 1980s.  St. John’s and the surrounding areas have gone anything but Tory for only about seven or eight years in the past decade.  In 1989, the Liberals took seven of eight St. John’s seats and 11 of 13 metro seats.  They did just about as good in 1993 and 1996 but after that things started to swing back toward the blue guys. Those sorts of changes don’t come easily;  it took the Tories a lot of misery after 1985 to fry townie voters to the point where they picked a bunch of red candidates in huge numbers. 

But let’s suppose for a second that we are seeing exactly what some people are claiming.  Slam the swings into the swing-o-meter and you won’t see any seats changing hands in St. John’s.  That’s because the Tories are still running so far ahead, it would take vote shifts of a magnitude out beyond what we have seen yet in order for the Tories to start sweating bullets about losing their base in town.

Come the fall you might be able to start musing about that sort of thing.  But that would be only if the NDP can somehow keep up momentum in a way they never have before, pull in way more money than they have and get tons of candidates and volunteers like they never have before.  They’d also have to run a provincial campaign using their provincial resources that rivals the federal one.  That’s a mighty tall order.

And even if the Dippers started to make some headway against the Tories in St. John’s, out beyond the overpass, the fights that will shape up would be between the Liberals and the Conservatives except in Labrador West.

Thing is voters don’t appear to be cool to Yvonne and potentially hot for Lorraine as the CBC writer imagined.  Voters haven’t warmed to either of the opposition parties and their leaders.  Lorraine’s at a amazing level of 11% while Yvonne’s at 13%. 

That could change and there’s plenty of room for change.

We just aren’t there yet.

Still, allowing for all the vagueness of these poll results, if you were Kathy Dunderdale, these poll results would be causing you a few sleepless nights in addition to the ones you are already feeling.

That’s the ongoing story in all this, just as it has been for the past six months or more.

- srbp -

1 comment:

Jerry Bannister said...

By point of comparison, the CRA poll results for NS peg support for the governing NDP at 42% of decided voters and Premier Dexter's approval rating at 28%.

According to the first line in today's Chronicle Herald story, this means that Dexter is "basking in popularity."

Later on in the story, they report that undecided voters "and those with no party preference," comprise 29% of those polled by CRA, while 4% don't plan to vote and 7% wouldn't say which party they prefer.

The story then ends by noting, "About 23 per cent of Nova Scotians don’t care which leader is in, while five per cent don’t like any of them," which is about the same percentage as Premier Dexter's bask-worthy approval rating.