The Canadian Payroll Association is getting a chunk of media attention for a poll they just released.
The stories are running something like this one from CBC:
Almost 60 per cent of Canadians live paycheque to paycheque and say they'd be in financial difficulty if their paycheque were a week late.
A new survey from the Canadian Payroll Association released Monday showed some troubling signs about Canadians' personal finances.
To its credit, CPA released a detailed research report. That’s where you will find some useful information when it comes to the survey.
For one thing, there’s no indication of how CPA or its research partner selected or contacted the respondents. The report does give some demographic data but how CPA found these 2766-odd people could affect results. If CPA depended on respondents to contact CPA then that might tend to skew the results toward people who were motivated for some reason. Since CPA represents the people who look after payrolls across the country, the responses might be skewed if the respondents knew the person from payroll was asking questions about their job security and personal finances.
These sorts of details matter when it comes to the results.
If you don’t think so, notice that one of the questions was about confidence that the person preparing the regular paycheque got it right. Forty-eight percent were extremely confident, 38% were very confident and 10% were confident. That’s 96% incidentally.
And if you think that might not be important, notice that in most provinces the total number of respondents was very small. Just 16 people answered the survey in Newfoundland and Labrador.
That might not matter in a national poll designed to sample the opinion of all Canadians on a few general questions. But on a political poll, it could matter quite a bit.
Even on a regional or provincial basis, you can get plenty of skewed results based on who gets surveyed. A recent opinion article in Campaigns and Elections predicted that 2012 will be the last presidential election in which pollsters rely on landline telephones for collecting data. Already 25% of Americans use cellular phones as their primary means of making telephone calls. The demographic that goes with that – young, African-American or Hispanic – is also an important one for some races or aspects of the bigger picture.
The situation is much the same in Canada. Researchers are having a harder and harder time getting people to spend the 15 or 20 minutes to do a telephone survey. A sample of 400 respondents might mean trying to contact over 6,000 people. And while cellular telephone usage might not be as significant in Canada as in the Untied States, other issues can affect results. Even in a province like Newfoundland and Labrador, language, comprehension (literacy) levels and social and cultural factors can all have an impact on results.
So when you look at a news story about another poll, don’t just take the results at face value. Go looking for more information about how the poll was conducted. Sometimes you might turn up some details that affect your perception of the poll and the news story based on it.
- srbp -
A-ha! Update: An excellent story in the Tuesday Telegram (not online) raises questions about this poll. According to the article, the survey went to local payroll officers who sent it out to employees for response.
That accounts for the varied response rates (16 in NL, 193 in QC and 500 in ON) and, as the article acknowledges, likely would affected the ability to take attitudes about the economy and extrapolate them to regions.
Of course, it also explains why 86% think their payroll officer is doing a fine job, but that they just need to keep those cheques coming to avoid personal financial disaster. Even if it wasn’t true, what else would you tell the person who makes sure you get paid?