Every year, poll watchers are confronted with poll results and commentary that defy either logic or science, often raising question about the very utility of polls. Typically, the problems are not with the method of conducting polls, but with the pollsters themselves – as they focus on what they believe is entertaining and appealing to the audience rather than an accurate reflection of public opinion. In the process, pollsters manipulate public opinion or write commentary that makes a mockery of what the public is really thinking.
With this article, veteran pollsters, authors and political scientists, George F. Bishop and David W. Moore launch what will henceforth be their Annual Top Ten “Dubious Polling” Awards. These awards are intended to mark for posterity some of the most risible and outrageous pronouncements by the major media polls during the previous year.
This is an especially exciting year to launch the awards, because in 2008 the polls were wrong so much of the time; thus, the fodder for the awards is virtually limitless. Still, with so many polling gaffes and so much goofy commentary, it has not been an easy task to decide which ones deserve the year’s top honors. So many deserving cases, and so few awards…
Each award is ranked, from a low of one set of crossed fingers to a high of five sets. Pollsters generally know in their hearts when all is not right with their polls, but they (figuratively) cross their fingers and hope that no one notices anything amiss. The five crossed-fingers icon is the ultimate in wishful thinking, perhaps the equivalent of football’s “Hail Mary pass” for the truly untrustworthy poll.
THE YEAR’S TOP AWARD!!
“The Absolutely Most Preposterous, But Entertaining, Myth of the 2008 Election: Rudy Giuliani Was the Early Frontrunner for the Republican Presidential Nomination!”
WINNERS: Unlike our other awards, this one is shared by all the major media polls. (1) Collectively this group, beginning more than one year prior to the beginning of the first statewide electoral contest in Iowa, relentlessly, and without regard for any semblance of political reality, portrayed Rudy Giuliani as the Republican frontrunner in a fictitious national primary.
BACKGROUND: As anyone with even a scintilla of knowledge about the U.S. electoral system recognizes, the presidential nominations of the two major parties are determined by how many delegates each candidate is able to win in the state caucuses and primaries. The winners are not determined by the national popular vote among Republicans, and separately Democrats, for their respective party’s candidates.
In the months leading into the first statewide contest for delegates, Rudy Giuliani was either trailing or barely competitive in the pre-election polls in Iowa, New Hampshire, South Carolina, Nevada and Michigan – the first five state contests for Republican candidates. While early on in the primary campaign Giuliani was leading all Republican candidates in Florida, because of greater name recognition, no candidate in the modern era of primaries has ever skipped the first five contests and still been successful.
The only people in the world who might have been expected to cite Giuliani as the Republican front-runner were the candidate himself and his wife, and maybe Bernard Kerik.
Still, the major media polls – relying on national surveys of people who said they were Republicans – claimed that the former New York city mayor was indeed the Republican to beat. In fact, the dominant question among most media pundits in the fall of 2007 was who would win the presidency – Hillary Clinton or Rudy Giuliani?
Giuliani lost all five contests in which he was not entered, of course, and then got skunked by John McCain and Mitt Romney in Florida. He dropped out of the presidential contest without having won a single delegate to the Republican National Convention.
Still, Giuliani enjoyed an entertaining ride (who can forget the oft-played video of him interrupting his speech before a large audience to answer a cell phone call from his wife?) as the fictional Republican front-runner for more than 12 months.
But shouldn’t the major media pollsters be embarrassed by this hoax? Oddly enough, none has admitted the truth: There was no early Republican front-runner. Really!
OTHER “DUBIOUS POLLING” AWARDS
“Loopiest Poll” Award
WINNER: The Pew Research Poll for its weekly pre-election polls in October, which showed Barack Obama with14- and 15-point leads two weeks and one week, respectively, before the election, but then (alone among the other polls) showed a precipitous 9-point drop in Obama’s lead to just six points in Pew’s final prediction – conveniently snapping the Pew poll back into line with most other polls just before the election.
BACKGROUND: On NPR, Pew’s director, Andrew Kohut, explained the dramatic decline shown by his poll in part by saying, “The other thing – McCain is enjoying the typical boost we get when we narrow the sample from registered voters to likely voters” (our emphasis).
It turns out that the explanation was as loopy as the trend: Pew had been basing its results on likely voters (not registered voters as Kohut asserted) for the whole month of October. The 9-point drop in Obama’s lead right before the election was among Pew’s likely voters, and could not be explained by Pew’s “narrowing the sample from registered voters to likely voters.” Some other mysterious force was at work that caused such loopy results.
Oh! Those inscrutable (likely) voters!
“Shooting Yourself in the Foot” Award
WINNER: The Gallup Poll for publishing two polls on February 25, conducted over the same time period, which contradicted each other. Its poll for USA Today showed Barack Obama leading Hillary Clinton among Democrats nationally by a 12-point margin, while its own tracking poll showed only a 2-point margin.
BACKGROUND: Mark Blumenthal of pollster.com called “the dueling national primary results from Gallup” the “polling mystery de jour.” The mystery has never been solved.
Gallup’s Frank Newport offered the startling suggestion that the poll “differences may well be another indicator of the conflicted nature of the Democratic race this year” (What? Polls can’t accurately measure opinion when people have conflicting views? But isn’t that the whole purpose of polls?!).
Blumenthal was not persuaded (“Dueling Gallups“). Instead, he concluded that “Newport seems to be stumped.” USA Today, of course, was left with egg on its face – its poll results torpedoed by its own polling partner. Whoops!
“Over-the-Top Gloating ” Award
WINNER: ABC’s polling director, Gary Langer, who characterized the erroneous projections by all the major polling organizations in the New Hampshire Democratic Primary as a “polling fiasco.” All of these polls had predicted Obama would win the primary by an average of nine percentage points, though Hillary Clinton won by two points. Langer proclaimed that this “fiasco” demanded an immediate investigation. He joked online: “What I like best about the final New Hampshire pre-election polls is that I didn’t do any of them.”
BACKGROUND: Establishing a pattern that would be followed in other primary states, and eventually mimicked by other major media pollsters, ABC and its polling partner, the Washington Post, conducted a poll in New Hampshire leading up to the primary election, but (craftily) refrained from conducting another poll close to Election Day.
The so-called “acid test” of a polling organization’s accuracy is how closely it can predict an election, because the election provides the real measure of voter intentions.
Polls that are conducted no closer than about two weeks before an election are typically not judged for accuracy, because it’s reasonable to assume that voters could change their minds in the interim. Throughout the primary season, the ABC/Post poll avoided such acid tests.
As it turned out, none of the major media polls that made projections in New Hampshire ever again subjected themselves to such acid tests in the subsequent primary states. Many followed the clever ABC/Post model – poll early on, but never close enough so the final polls can be compared with election results. Why risk embarrassment when lesser known polling organizations are willing to do it? Why indeed?
So much for the national media pollsters’ confidence in their own polls!
“180 Degree” Awards
WINNERS: The CBS News/New York Times Poll and the USA Today/Gallup Poll, for coming to diametrically opposite conclusions in early May about the electoral effect of the controversy over Barack Obama’s pastor, the Reverend Jeremiah Wright.
BACKGROUND: In the wake of criticism over a video showing Obama’s pastor denouncing the United States in inflammatory language, the two polls found quite different reactions among Democrats.
USA Today headlined its poll story, “Flap over pastor pulls Obama down, poll finds.”
By contrast, CBS News headlined its report that “Support for Obama Rebounds” in the wake of the controversy, while its polling partner, the Times headlined the results as “Obama Survives Furor.”
Let’s see now: Was he up? Or was he down? Yes to both questions!
Though this was one of the most important turning points in the election, the pollsters fell asleep at the switch. Gallup’s Frank Newport noted that it may be “a perplexing issue to many readers” when the polls conflict, but it’s simply one of the “facts of life when polling in a churning environment.”
Ahh…maybe polls should come with a cigarette-like warning upon their release, letting readers know that polls shouldn’t be taken seriously if conducted in a “churning” environment.
Next question: When is the environment not churning?
“Waiting for Godot” Award
WINNER: The special American Association for Public Opinion Research (AAPOR) Committee that was organized to investigate why the polls were all wrong in predicting Obama the winner in the New Hampshire Democratic Primary. A year has passed, and there is still no sign of a report.
BACKGROUND: Five days after the “New Hampshire primary fiasco,” Nancy Mathiowetz –president of AAPOR, the foremost organization in the country of scholars, practitioners, and users of survey research – announced the formation of an ad hoc committee “to evaluate pre-election primary methodology and the sponsorship of a public forum on the issue.”
She reassured the public that polls had long been “remarkably accurate,” and added, “Sixty years ago the public opinion profession faced a crisis related to the poll predictions of the Truman-Dewey race. The way survey researchers reacted then – with a quick, public effort to identify the causes – played a key role in restoring public confidence and improving research methodology” (our emphasis).
Quick? Hmm…Six decades ago the polling industry did act with dispatch, but these days, apparently, all pollsters have to do is promise to examine why the polls were wrong and the public is reassured.
Besides, who cares anymore about the New Hampshire primary – it was so last year. On to the next poll!
“Who Knows?” Award
WINNERS: Pew Research, ABC News/Washington Post, and the Los Angeles Times/Bloomberg polls for their enlightening, but wildly contradictory, conclusions about whether the American public supported a federal government bailout of Wall Street – Pew said definitely yes, ABC/Post said maybe, and Times/Bloomberg said definitely no. That about covers the spectrum!
BACKGROUND: On September 24, 2008, Pew and the Los Angeles Times posted the results of their respective polls, both conducted over the previous three-day period, with each showing large majority of Americans on one side of the issue – except on the opposite sides. A Pew Research poll showed a 27-point margin in favor of the Wall Street bailout, the Los Angeles Times/Bloomberg poll a 24-point margin opposed.
Coincidentally, the Washington Post/ABC poll published the same day, and also conducted over the same time period as the other polls, found an evenly divided public (with a two-point margin in favor, but well within the poll’s margin of error).
Pollsters tend to be upbeat about their polls, regardless of the outcomes, and ABC’s Gary Langer rose to the occasion: “Some analysts might say the results are contradictory; I’d suggest instead that we learn more, not less, by comparing and contrasting them.”
“Instead”? (Non sequitur, Gary!) The results are contradictory, and we can also learn by examining their contradictions.
So, what do we learn? What we’ve known for a long time: Pollsters can create different “public opinions” by the way they word the questions.
What do we not learn is the answer to this question: what does the public really think?
“Wake-Me-Up-When-It’s-Over” Award
WINNER: NPR, Kaiser Family Foundation, Harvard School of Public Health Survey, for a vague, 131-word question.
BACKGROUND: Pollsters and their clients cannot resist the temptation to ask detailed questions about complex matters over the phone. We understand the temptation, but seriously, folks – do we think instantaneous responses to these long, complicated questions really measure public opinion? Maybe they interviewed only Harvard grads with Ph.D’s in health policy?
(Pretend a pollster called you in the early evening, and try listening to this question over the phone while you’re fixing dinner and people are talking in the kitchen!)
| “Next, I’m going to read you a description of a plan to make sure everyone has health insurance, and then I’m going to ask you if you support or oppose this plan. This proposal would place requirements on individuals, employers, the government, and insurance companies so that everyone shares in the responsibility. Individuals who don’t already have insurance would be required to buy it or pay a fine, with financial help from the government for people with lower incomes. Employers would be required to cover their workers, or pay money into a pool that helps people buy insurance. Government health insurance programs would be expanded. Insurance plans would be required to take anyone who applies, even if they have a prior illness. Would you support or oppose this kind of plan?” (Feb. 14-24, 2008) |
Would you strongly oppose or only moderately oppose pollsters asking that kind of question?
“Flip-Flop” Award
WINNER: CNN for its two polls, one in early December and one in mid-December, which showed virtually mirror images of the public’s support for the auto bailout.
BACKGROUND: On Dec. 3, CNN announced that the public opposed a federal government bailout of the auto industry by a solid 25-point majority. On Dec. 22, CNN announced the public had flip-flopped and now supported a bailout by a 26-point majority. Other polls, found no change in public attitudes over that same time period.
The difference in the two polls was almost certainly caused by a creative change in the wording of CNN’s question rather than a real change in what the public was thinking.
Here are the two questions. The first implies that auto companies want a free handout (“several billion dollars in assistance”) with no certainty of success. The second suggests that money will be repaid (“thirteen billion dollars in loans”) and that the loans will be successful (the companies will produce “plans by the end of March that show how they would become viable businesses in the long run”).
| 1. “The major U.S. auto companies have asked the government for a program that would provide them with several billion dollars in assistance. The auto companies say they may go into bankruptcy without that assistance. Based on what you have read or heard, do you favor or oppose this program?” Favor 36%; Oppose 61%, no opinion 2%
2. “The federal government will provide some of the major U.S. auto companies with more than thirteen billion dollars in loans in order to prevent them from going into bankruptcy. In exchange for the loans, the government wants the auto companies to produce plans by the end of March that show how they would become viable businesses in the long run. Do you favor or oppose this decision?” Favor 63%, Oppose 37% |
A little tinkering with the wording of a question will go a long way! That’s certainly what General Motors thought (see below).
“Poll For Sale!” Award
WINNER: Peter D. Hart Research Associates, for their General Motors-sponsored poll, which (surprise! surprise!) found the public overwhelmingly in support of a federal government bailout of the auto industry, and fearful of a recession if no bailout occurs.
BACKGROUND: What do you do if you’re going to appear before Congress to ask for money, and you don’t trust what the public polls might tell you? Commission your own poll, of course! You’re more likely to get the results you like. And the more prestigious pollster, the better.
That’s what General Motors did, when it commissioned Peter D. Hart, a veteran and highly respected Washington, D.C. pollster, who is also one of the two pollsters for the NBC News/Wall Street Journal Poll, to find out the public’s attitudes toward an auto industry bailout.
First priming the respondents with questions about the importance of the auto industry to the economy, and the potential harmful results if “the American automobile industry no longer had the resources to produce vehicles” (no mention of “bailout,” of course), the poll then asked whether the federal government should provide loans to American automakers “so they have the money to manufacture vehicles?”
Yes, indeed, said these respondents – by a 25-point margin! (With such a build-up, how could anyone say no?!)
According to the story published by Reuters, “Hart said that behind the support for the automakers is the fear that the failure of GM, Ford, and Chrysler in the next few years could trigger an economic depression.”
What question led to that conclusion? This is the 144-word gem that the Hart poll used to “discover” (create) the public’s calamitous views about the potential failure of the auto industry:
| “Let me read you four facts about the auto industry’s effect on the American economy.
“(1) The U.S. auto industry affects numerous other American industries because it is the largest purchaser of steel, copper, plastics, electronics, and computer chips. “(2) The U.S. auto industry provides pensions to seven hundred and seventy-five thousand Americans and health benefits to two million Americans. “(3) The U.S. auto industry supports approximately five million U.S. jobs in all fifty states. “(4) The U.S. auto industry has fourteen thousand dealers in nearly every town in America employing seven hundred thirty thousand people. “Knowing this, if General Motors, Ford, and Chrysler were to go out of business in the next few months, how likely do you think this would be to trigger an economic depression in the U.S.––extremely likely, very likely, somewhat likely, not likely, or not at all likely?” Extremely likely…………………………… 44 Somewhat likely…………………………… 18 |
It’s a good thing General Motors didn’t wait for the public pollsters. In its November poll, Gallup found the public evenly divided over an auto industry bailout. The ABC/Washington Post poll found Americans opposed by a substantial margin, 57 percent to 35 percent – almost the opposite picture of the one Hart painted for the car company.
General Motors’ money was well spent. Now if only GM could spend money as efficiently on building good cars!
George Bishop is Professor of Political Science and Director of the Graduate Certificate Program in Public Opinion & Survey Research at the University of Cincinnati. His most recent book, The Illusion of Public Opinion: Fact and Artifact in American Public Opinion Polls (Rowman & Littlefield, 2005) was included in Choice Magazine‘s list of outstanding academic titles for 2005 (January 2006 issue).
David W. Moore is Senior Fellow with the Carsey Institute at the University of New Hampshire. He is a former Vice President of the Gallup Organization and was a senior editor with the Gallup Poll for thirteen years. He is author of The Opinion Makers: An Insider Exposes the Truth Behind the Polls (Beacon, 2008). Publishers’ Weekly refers to it as a “succinct and damning critique…Keen and witty throughout.”
(1)These include polls by the Associated Press, ABC News/Washington Post, CBS News/New York Times, CNN, FOX, NBC News/Wall Street Journal, USA Today/Gallup, Newsweek, the Los Angeles Times/Bloomberg, and Pew Research. Back