Ye, verily, lo and behold, good citizens:
There are problems with reports coming from Alberta about the Wild Rose party and polling.
Sounds very familiar.
One thing leaped out in particular and this had to do with the reliability of online surveys:
Unbeknownst to most Albertans – even many politically savvy ones and apparently to most political journalists, too – was the import of this statement.
First, this is an on-line survey, based on interviews with a group of Albertans, obviously interested in politics and quite possibly committed to a political party, who selected themselves for the job.
The advantage of on-line panels of this type is that they’re inexpensive to conduct. The disadvantage is that their results cannot be called scientific and they are not particularly credible.
Also unknown to most Albertans following this story is the fact that the phrase “margin of error is 3 per cent” is highly controversial in professional polling circles when applied to this kind of survey.
The implication is that the survey was based on sound scientific methodology and can be counted on to be accurate within a margin of 3 per cent.
The fact is the survey is based on the opinions of people who selected themselves to join the panel and answered questions on-line. In other words, this is not necessarily a particularly trustworthy poll.
That probably refers to issues like the ones raised in this businessweek.com article in 2008.
And it points out again why reporters need to ask more questions about polls and pollsters before they report the results.
Lie say this little gem from an American association of polling firms giving 20 questions journalists should ask about polls:
6. Are the results based on the answers of all the people interviewed?
One of the easiest ways to misrepresent the results of a poll is to report the answers of only a subgroup. For example, there is usually a substantial difference between the opinions of Democrats and Republicans on campaign-related matters. Reporting the opinions of only Democrats in a poll purported to be of all adults would substantially misrepresent the results.
Poll results based on Democrats must be identified as such and should be reported as representing only Democratic opinions.
Of course, reporting on just one subgroup can be exactly the right course. In polling on a primary contest, it is the opinions of those who can vote in the primary that count – not those who cannot vote in that contest. Primary polls should include only eligible primary voters.
Yes, ladies and gentlemen, reporting poll results as a percentage of “decideds“ is “one of the easiest ways to misrepresent the results of a poll…”. Some people won’t be shocked by that nugget.
If you scroll down you’ll also find a good section that points out methodological problems with online poll results.
And that’s before people misread news reports that seemed to suggest Danny Williams scored a 70% approval rating with all Canadians surveyed by Angus Reid last fall during the local poll goosing month of November.
At some points, local news organizations will either stop reporting obviously unreliable poll results or hold off until they get answers to some tough questions from the people flogging the polling fodder.
-srbp-