The Answer is in the Questions

Rasmussen Reports released a poll on the Wisconsin public employees standoff that seems more intended to generate a predetermined result than real insight.  The headline poll result was 48% Back GOP Governor in Wisconsin Spat, 38% Side With Unions

But as always, it it best to look at the actual questions.  Rasmussen asks five questions:

1* How closely have you followed news reports about the Wisconsin governor’s effort to limit collective bargaining rights for most state employees?

This is a pretty typical first question for Rasmussen.  It isn’t necessarily a bad question.  It serves to frame the subject of the discussion.

2* Does the average public employee in your state earn more than the average private sector worker in your state, less than the average private sector worker in your state or do they earn about the same amount?

The question jumps from Wisconsin to ‘your state’ (it was a nationwide poll, not restricted to Wisconsin).   Putting aside the difficulty of making comparisons for differences in skill levels and other factors that appropriately impact compensation, it is silly to seek an opinion about a factually calculable number – particularly a detailed one like this.  How can a typical respondent know factually off the top of his head about the wages in public vs private sector jobs?  The question has the undesired impact of introducing bias into the results.

3* Should teachers, firemen and policemen be allowed to go on strike?

Public safety employees are exempted from the proposed Wisconsin bill.  So the two most emotionally charged of the three examples in the question are irrelevant to the issue.  Consequently, the question is not properly focused on the result that Rasmussen ultimately reaches.

4* In the dispute between the governor and the union workers, do you agree more with the governor or the union for teachers and other state employees?

This is the ‘headline’ question, the one that Rasmussen uses for its poll result.  But, at this point, the poll has been affected by what preceded it.

5* Would you favor or oppose reducing your state government payroll 1% a year for 10 years, either by reducing the number of state employees or by cutting the pay of state workers?

This question is too vague to understand whatever result may come from it.  First, the question gives multiple options for how to get the reduction.  Additionally, since respondents undoubtedly favor some programs over others, the question fails by making no mention of what program area might receive the cuts.  The questions is easy say ‘yes’ to, since who would not want to save money?  But one person may respond ‘yes’ for wage reductions, another person may respond ‘yes’ for cutbacks in department X, and yet another person may respond ‘yes’ for cutbacks in everyplace other than than department X (that the second person wanted to cut).

So what would be a better question?  I submit the robo-poll conducted at the same time by We Ask America.  Following basic demographic questions it asked the following question:

As you may know, Gov. Scott Walker has proposed a plan to limit the pay of government workers and teachers, increase their share of the cost of benefits, and strip some public-employee unions of much of their power. We’d like to know if you APPROVE or DISAPPROVE of Gov. Walker’s plan.

We Ask America’s question is good in all the ways that Rasmussen’s was bad.  It was direct, does not distract, provides enough background without using jargon, and does not try to pull emotional strings.

Unfortunately direct comparison of the two polls’ results is not accurate because Rasmussen conducted a nationwide sample while We Ask America called only Wisconsin residents.  We Ask America’s results were  43.05 percent approve 51.9 percent disapprove, and 5.05 percent uncertain. The 5.05 percent is low for undecided in almost any poll, particularly one in which nuanced responses (such as ‘mostly agree’ or ‘mostly disagree’) were not available.

(I must offer a hat-tip toMark Blumenthall for bringing this to my attention, including the following clip from the British Series Yes Prime Minister that makes the same point with more humor and better snappy accents)

To learn how to word poll questions and ensure unbiased results, see a Primer on Creating and Using Surveys.

Permanent link to this article: https://betweenthenumbers.net/2011/03/the-answer-is-in-the-questions-2/

Leave a Reply

Your email address will not be published.

*