See above a screenshot from Mark Mellman's post for The Hill on polling. (Credit: The Hill, screenshot, highlight added)

Recent polls by Gallup and Pew show conflicting results over the public’s views about the government’s efforts to save the automobile industry.  Gallup reports a majority of Americans expressing negative views; Pew shows a larger majority expressing positive views.

Compare above Gallup’s findings to Pew’s above.

Mark Mellman, writing in The Hill, notes that the question wording used by the two pollsters is quite different:

Gallup Poll:

“Now, thinking back to one of the major actions taken by the federal government in the last four years, would you say you approve or disapprove of the financial bailout for U.S. automakers that were in danger of failing?” (Approve 44%; Disapprove, 51%; No opinion, 5%)

Pew Poll:

“The government also gave loans to General Motors and Chrysler during this period. Do you think this was mostly good or mostly bad for the economy?” (Good, 56%; Bad, 38%; No opinion, 6%)

Mellman quotes Gallup’s Frank Newport, who offers this explanation for the different results:

“One reason for the different responses may be the Pew focus on the impact of the actions on the economy, while we at Gallup asked for a broader approve/disapprove response.”

If we are to follow this reasoning, Newport is saying that a lot of people disapprove of actions they nevertheless feel were good for the economy.  Hmm…I don’t think so!

As Mellman points out,

“Another difference in the question wording, unremarked upon in Gallup’s analysis, is its use of the word ‘bailout.’ Pew talks about ‘loans,’ technically more accurate and certainly less inflammatory, though ‘bailout’ is clearly common parlance. It should surprise no one that Americans prefer ‘loans’ to ‘bailouts.’

Anyway, as Mellman notes, Pew asked a similar version of its question back in 2009, when a majority said the “loans” were “bad” for the economy. Compared with the results of the Pew poll that year, Americans today are 19 points more positive and 16 points less negative about the loans, a 35-point turnaround. As Mellman notes: “Whatever the actual level of support for rescuing the auto companies, that 35-point net shift reflects strong positive momentum.”

Compare above Pew’s 2009 and 2012 polls.

Mellman’s final paragraph is an important caveat about polls:

“There’s a broader lesson here. The next time someone attempts to interpret public opinion based on responses to a single question, at one point in time, recall this example and be suspicious — seemingly small differences in wording, barely noticeable without thinking through alternatives, could be exerting a substantial impact on the answers.”

Mellman is right. But it’s noteworthy he doesn’t mention that neither pollster measured non-opinion, that both pollsters exaggerate how engaged the public is. Do we really think that more than 90% of Americans have a fixed opinion about that one issue? Not very likely.

Both pollsters used forced-choice questions, which means that a lot of people who really had no opinion were nevertheless pressured to express one, by choosing among two options offered in the poll. It is these people who are most likely to be influenced by those “small differences in wording” that Mellman notes.

If pollsters want to produce a realistic measure of what the public is thinking, they will need to measure both opinion and non-opinion – instead of continuing the illusion that the overwhelming majority of the public is fully engaged on all public policy issues.

Mellman is right: Be suspicious of what the media polls tell us.