Pollsters have been asking rather fuzzy question about trusting the “government in Washington,” according to a number of reports this past year.
Is the “federal government” to be trusted or not?
A cardinal assumption in asking any survey question is that it should mean the same thing to all respondents.
And the question should also mean the same thing to respondents next year as it did this year or last year.
That’s why pollsters take so much trouble to make the wording of their questions identical for all respondents and to keep their wording consistent when they repeat the same question for comparisons over time.
The problem is that pollsters all too often violate this cardinal principle by asking fuzzy questions.
An excellent ex
So Pew told us nearly 80% of respondents surveyed had lost their faith in the federal government.
| Pew Research reported that only about 22% of Americans trust the government in Washington. (Credit: Pew Research) |
Using the exact same question, a recent NYT/CBS poll of registered voters also found that only about one in five (21%) of those surveyed felt the government in Washington could be trusted “to do what is right: just about always or most of the time,” leaving nearly 80% in the non-trusting category.
A CNN/ORC poll of U.S. adults in early September, using an identical, fuzzy question cited much the same lack of faith.
Just 25% trust “the government in Washington” [CNN/Opinion Research Corporation Poll, Sept. 1-2, 2010].
| See above a comparison of how media polls measured the public’s trust in government. |
What Do the Poll Questions Mean?
As a poll skeptic, one always wants to see what’s inside the survey sausage. How was the question worded? Was it clear and precise, or vague and ambiguous? Is the trust glass half empty or half full?
The standard Pew, NYT/CBS, and CNN/ORC question (s)—over and again a prime example of fuzziness and ambiguity in public opinion polls reads.
“How much of the time do you think you can trust the government in Washington to do what is right? Just about always, most of the time, or only some of the time?” [Emphases mine]
But a more precise pollster would ask while formulating their survey question, what does the “government in Washington” mean to different respondents, not only this past year and just before the midterm election, but also last year and 5 or 10 years ago?
What will it mean, now, after the Republicans have gained power in “the government in Washington,” and two or three years from now?
We know that there are always large partisan gaps on this question, depending on who occupies the White House.
The wording of the question may be kept the same; however, even this does not ensure that the meaning-and-interpretation of the survey question will be the same across respondents or over time.
Notice that respondents have to select either “always” or “most of the time”—essentially perfect or nearly perfect ratings—for government to be ranked as being trustful. But if a person answers that they trust the government “some of the time,” such a person is counted as not being trustful of the government. The polling measurement of this individual, like the water glass, moves to half empty from half full ebooks.
Not only that, one respondent’s “most of the time” might well be another’s “some of the time.”
Examples of Fuzziness
Despite the glaring fuzziness of this question, it has been the basis for the trust in government charts that Pew and other pollsters have been using for years to show how faith in the “government in Washington” has fluctuated up and down over time.
Here are some examples from the same Pew poll about trust in government. In their recent probe of what Americans think about the “federal government,” they also asked respondents a whole block of unclear and imprecise questions [Emphases mine] in their March 2010 telephone survey.
Pew asked:
“Imagine a scale from one to six where ONE represents someone who generally believes that federal government programs should be cut back greatly to reduce the power of government, and SIX represents someone who feels that federal government programs should be maintained to deal with important problems. Where on the scale of 1 to 6 would you place yourself?”
So I have to think of a scale where the number one is my desire to greatly reduce power of government, whereas six is keeping programs to deal with problems? How is reduction of government power the opposite of keeping government programs to deal with problems? It just does not scan.
And another example:
“All in all, how good a job does the federal government do running its programs? An excellent job, a good job, only a fair job, or a poor job?”
Which programs? It doesn’t say. Some people’s experiences will fall in the federal Veterans Administration in one case and another, the village or county sanitation service. The poll question fuzziness does not seem to stop.
Here’s another:
“Some people think [criticism of the way the federal government does its job is justified], while others think the [government often does a better job than it is given credit for]. Which comes closer to your view?”
So if I think criticism of some [specific] things the government does is justified—like not adequately screening air and ship cargo from other countries or protecting consumers from unsafe medications or products—that’s one thing.
But if I feel the government does a better job than it is given credit for on such things as providing Medicare benefits to people like me, how am I supposed to answer this question. It’s a little bit of both!
One last fuzzy one from the same Pew poll:
“And which of these statements comes closest to your view…
-
- The federal government needs very major reform
- The federal government is basically sound and needs only some reform
- The federal government doesn’t need much change at all.”
Which part of the “federal government” are you talking about? That’s quite a big net you’re responsible for casting in a survey.
You may think there are some things the government does well, such as defense spending but that others, such as Social Security, needs some reform.
And then there are things like tax credits for home improvements to save energy that some like.
Now compare the stock fuzzy question above about trusting the “government in Washington” with those actually used recently by Gallup, in which respondents were asked more specifically about their views:
| See above, USA Today/Gallup’s September poll asking what the public thought the government should be responsible for. (Credit: USA Today/Gallup) |
Another difficulty seen in these examples above is the number of possible answers under each question. Respondents have to pick a number on the scale from 1 to 5, but with unspecified middle points 2, 3, and 4.
For example, when asked to rate how much responsibility the government should have for “protecting Americans from foreign threats,” “preventing discrimination,” and “providing a minimum standard of living for all,” only point 1 (no responsibility) and 5 (total responsibility) are defined.
Respondents must infer what points 1, 2, and 3 mean, which can vary from one person to another. My 4 might not be your 4.
Frank Newport, Editor in Chief of the Gallup Poll, concluded: “… Americans are more likely to choose middle-ground responses rather than extremes…and tend to use the middle 2, 3, and 4 points on the five point scale in their answers, avoiding the two extreme endpoints.”
But what is the meaning of “protecting Americans from foreign threats”—first item on Gallup’s list in chart above? It can mean different things to different respondents, everything from international terrorism to homeland security and airport security scanners.
What is the meaning of “preventing discrimination” — second item from the top — to African-Americans, whites, and Hispanics? To men and women? Gays and straights?
“Providing a minimum standard of living”– third from the top? Whose minumum? What minimum?
The most grossly vague and ambiguous phrase— not to mention profoundly relativistic— in Gallup’s list of potential government responsibilities, and probably one of the fuzziest I’ve seen as a survey professional is:
“Upholding moral standards among its citizens”?
Whose moral standards?
Yet over all, Gallup’s way of asking about more specific responsibilities of government, from my polling perspective, will produce far better, accurate opinion measurements than the fuzzy questions often used by Pew, NYT/CBS and CNN/ORC and Gallup too.
So how much do people trust the nebulous “government in Washington”?
Gallup’s questions just don’t go far enough in making the meaning of the question(s) clearer and more precise.
Questions Need To Be Clearer, More Specific
Asking respondents, for example, about protecting Americans from specific foreign threats such as the Taliban in Afghanistan and Pakistan, Al Qaeda in Yemen, or potential nuclear weapons in Iran would likely produce better polling results and is a step in the right direction.
The results produced by the Pew Center, NYT/CBS, and CNN/ORC’s question about trusting the “government in Washington” suggests that the vast majority of Americans don’t trust the government.
Yet, even so, AT THE SAME TIME the Gallup poll results show that the vast majority also think the government should have lots of responsibility for all sorts of activities anyway!
Such bizarre-world survey outcomes are born from fuzzy questions.
Bottom line: If so many people don’t trust the government, it just does not make sense that these same people want the government responsible for so many things.
This glaring contradiction underscores the fundamental fuzziness of the “trust in government” question that has been appearing on the questionnaires used by polling organizations over the years, starting during the late 1950’s in the University of Michigan’s election surveys.
If different respondents interpret the meaning of “the federal government” or the “government in Washington” in different ways, there is no standardization of measurement, and all question meanings become relative to time and place.
How Questions Are Interpreted
Far better I say to start with open-ended questions about how they interpret questions about the “federal government” or “government programs.”
For example, in a pretest interview, respondents could be asked: “When you think about the ‘government in Washington’ [federal government, government programs], what comes to mind? Could you be a little more specific, please? How do you feel about it?”
With respondents’ verbatim replies to such questions, pollsters can create more useful, concrete, and specific questions about “the government.”
Or when they ask respondents closed-ended, or fixed choice, questions, such as the standard fuzzy trust question—“How much of the time do you think you can trust the government in Washington to do what is right? Just about always, most of the time, or only some of the time?” [Emphases mine]—they can use follow-up probes to understand how the question is being interpreted.
For example, when a respondent selects “some of the time” [most of the time or just about always], he or she can be asked: “What do you mean when you say you trust the government in Washington some of the time?” “Can you be a little more specific?”
But this greater precision produced by better poll design questions means higher costs for pollsters to categorize spontaneous verbatim responses to such open-ended probes, not to mention it takes more time.
It’s far easier and cheaper to stick to stock questions and fixed response alternatives that can be rapidly data processed and analyzed, making good quick news copy for the mass media.
With the Republican Party’s huge victory in this year’s midterm election in mind, the pollsters, politicians, and pundits will now be telling us about what it all meant.
Among the highly likely explanations of the election results will be various assertions about the revolt of the electorate against the too big and powerful “government in Washington.”
As Speaker-To-Be, John Boehner, put it the day after the election: he referred to the results as “a repudiation of big government and a repudiation of politicians who refuse to listen to the people.”
Senator Mitch McConnell, Republican minority leader spoke passionately about cutting spending and shrinking “the size and scope of government…”
But what cuts? What part of the “big government in Washington?” That’s about as clear as some of the questions about trusting “the government in Washington to do what is right?” What’s that about?
For this report, New York Times’ Janet Elder, CNN’s Keating Holland, Pew’s Vidya Krishnamurthy, and Gallup’s Frank Newport were contacted for comment. By publication time, none has responded. We will update with any responses.
George F. Bishop is Professor of Political Science and Director of the Graduate Certificate Program in Public Opinion & Survey Research at the University of Cincinnati. His most recent book, The Illusion of Public Opinion: Fact and Artifact in American Public Opinion Polls (Rowman & Littlefield, 2005) was included in Choice Magazine’s list of outstanding academic titles for 2005 (January 2006 issue).
David W. Moore is Senior Fellow with the Carsey Institute at the University of New Hampshire. He is a former Vice President of the Gallup Organization and was a senior editor with the Gallup Poll for thirteen years. He is author of The Opinion Makers: An Insider Exposes the Truth Behind the Polls (Beacon, 2008; trade paperback edition, 2009). Publishers’ Weekly refers to it as a “succinct and damning critique…Keen and witty throughout.”