Veteran pollster and author David W. Moore presents the 2013 Fifth Annual Top Ten “Dubious Polling” Awards. Designed as a satirical, though serious look, at some of the most questionable polling practices or results covered by the news media in the past year, this series has included a wide spectrum of issues.
Some are light, while others raise troubling questions about the nature of polling and sometimes even about the ethics of those involved in conducting, releasing, or analyzing polls.
10. Vital Information for the Public to Know Award
(Credit: ABC News, screenshot)
Winner: ABC News/Washington Post Poll on its mid-October report that “Parents Prefer President Obama As Babysitter”.
Yes! Which presidential candidate would be the better babysitter? That’s the key to how voters decide. So, it was enlightening to learn that in mid-October, registered voters preferred Obama over Romney to assume such duties, by a margin of 49% to 36%. And if that weren’t enough, it turns out that two weeks earlier, another poll showed Obama and Romney tied at 41%. Amazing! (By the way, doesn’t it make you wonder what happened in early October to boost Obama’s babysitting creds?)
Oh, and there’s this: Among likely voters, Obama beat Romney by 42 points (61% to 19%) – on the likelihood of going bungee jumping.
Who wudda known this “vital” information – if not for the stellar polling by ABC and the Washington Post?
9. Just Trying to Save the Polling Industry Award
Frank Newport, Editor-in-Chief, The Gallup Poll
Winner: Gallup’s Frank Newport for his post-election article warning of a “significant issue” for the polling industry going forward: “That we could see significantly fewer polls conducted in the 2016 election — at the state and national level — than in this election.”
Reading between the lines, some pundits have suggested that Newport’s commentary was really sour grapes rant against Nate Silver, given Silver’s assessment that Gallup produced the worst record of 23 organizations that polled at least five times in the last three weeks of the 2012 election.
Even worse, perhaps, was Silver’s analysis that “Gallup has now had three poor elections in row. In 2008, their polls overestimated Mr. Obama’s performance, while in 2010, they overestimated how well Republicans would do in the race for the United States House.”
But another way to look at Newport’s comments is to take them at face value, that Gallup does have a genuine concern for the future of the polling industry. As Newport points out:
“It’s much easier, cheaper, and mostly less risky to focus on aggregating and analyzing others’ polls. Organizations that traditionally go to the expense and effort to conduct individual polls could, in theory, decide to put their efforts into aggregation and statistical analyses of other people’s polls in the next election cycle and cut out their own polling.”
And fewer polls would be tragic. After all, there were only about 550 polls on the presidential election conducted by 90 organizations in the last 21 days before voting day. Where would the country be, how would democracy even function, if we had fewer polls for us consumers to get confused about?
8. Jumping the Gun Award
(Credit: Politico, screenshot)
Winner: Politico for breathlessly announcing at the beginning of August, a full three months before the election, that while Obama and Romney would like to win over undecided voters, “there just aren’t many of them left.” How many left? Only about five percent of the electorate, Politico estimated (based on several media polls).
It may seem a bit naïve to jump to this conclusion so early, but maybe Politico was trying to do the country a favor, by shortening the what-seemed-like-it-would-never-end campaign a bit early. But reality intervened.
Two weeks later, an Associated Press/GfK poll reported that about one quarter of all registered voters (500% higher than the average reported by Politico), were still mulling over their choices. Then at the end of August, ABC’s Gary Langer reaffirmed that “one in four registered voters may be persuadable in the 2012 presidential election.”
On Election Day, the national exit poll revealed that fully 21% of all voters said they hadn’t made up their mind even by the end of September – more than 420% higher than the percentage reported by Politico two months earlier.
Oops!
7. You Say Tomayto, I Say Tomahto, They Say “What’s That?” Award
See above a screenshot from Mark Mellman’s post for The Hill. (Credit: the Hill, screenshot, highlight added)
Winner: Pew and Gallup for their conflicting results about public opinion on the government’s efforts to save the U.S. auto industry, revealing how easy it is to create different “public opinions” by pressuring people to come up with views they don’t have.
Gallup reported a 7-point majority of Americans disapproving of the “financial bailout” of the industry, while the very same day (Feb. 23), Pew announced that an even larger majority (by 18 points) had a contrary opinion – that the “government loans” to the industry were “mostly good” rather than bad for the economy.
You might think that the 25-point difference between the polls was caused by question wording (“bailout” vs. “government loans”). But as a subsequent iMediaEthics poll shows, more important in explaining the difference was the failure of the pollsters to measure non-opinion. Using “forced-choice” questions, both pollsters reported about 95% of Americans with meaningful opinions. But the iMediaEthics poll revealed that, realistically, about half of the public was unengaged on the issue and had no strong views one way or the other.
To some, that’s a shocking finding! How could so many Americans not find time in their daily lives to follow the intricacies of all government relief programs? Yet, sadly, it’s true. With non-opinion taken into account, but keeping the wording differences (bailout vs. loan), the iMediaEthics poll showed only a 6-point difference between the results.
Whether I say “tomayto” and you say “tomahto,” it turns out, is less important than whether we all know what we’re talking about in the first place!
6. It’s Not Really Lying Award
Winner: CNN’s Anderson Cooper for his interview with Robert Gibbs, Obama’s senior campaign manager, when Cooper asked Gibbs to explain how the president’s approval had dropped to just 41%. It turns out that CNN’s own graph(which Gibbs could not see) showed four polls all taken about the same time – with approval ratings of 50%, 41%, 46%, and 49%. But Cooper mentioned only the lowest one.
OK! It wasn’t really lying. It was just Anderson Cooper trying to play gotcha, by giving Gibbs only partial information. How else to reaffirm the integrity of the media than by fooling its guests with misleading information?
5. Cookie Jar Award
(Credit: CBS, screenshot)
Winner: The CBS/New York Times poll for its May 14, 2012 report on the presidential race, with the vote question asked after questions about the state of the economy.
A cardinal principle of measuring voting intentions is that the vote choice question (who do you support for president, for example) has to be asked before any substantive questions about policy issues. Otherwise, the results can be tainted, with respondents being led to think of one issue rather than another right before deciding who to vote for.
But in its May poll, CBS and the Times violated this cardinal principle when they asked their presidential preference question after first asking two questions about the economy – its current condition and its future prospects. The result: an outlier result showing Romney up by three percentage points over Obama.
As I pointed out at the time, the pollsters would never consider asking a vote choice question after policy questions in their final poll before the election, because they would want as accurate a measure as possible.
But, hey, so what! Early on, no one really cares if the results are accurate. And who’s going to catch them with their hands in the cookie jar anyway?
4. Incorrigible Addict Award
Winner: The major media polls and talking heads for their continual addiction to national primary polls.
Between the middle of January and the middle of February, Gallup’s national primary polls produced three different front-runners for the GOP presidential nomination: Mitt Romney, Newt Gingrich, and Rick Santorum. (Gallup was not alone in conducting national primary polls. Most of the other major media pollsters also reported national poll results of GOP voters.)
Yes, it was fun to see this exciting virtual horserace! But it was based on an electorate that doesn’t exist – national primary voters. This is almost like the NFL’s Fantasy Football – except that it’s not football, and you don’t get to draft or train the candidates, nor make them compete. But it’s just as much a fantasy!
In the real world, the nomination is determined not by a national primary, but by a succession of state primaries. And the national polls don’t predict anything about what happens at the state level – a point that even Gallup admitted.
So, why the addiction to discussing results that tell us nothing meaningful about the nomination contest?
Have you ever tried to stop smoking?
3. Covering Up Research We Don’t Want to Acknowledge Award
(Credit: Huffington Post, screenshot)
Winner: The Robert Wood Johnson Foundation (RWJF) and RTI International for their news release on a Middle School Dating Violence poll, in which they deliberately omitted important results in an effort, apparently, to cover up politically incorrect findings.
In March 2012, RTI announced the results of their latest poll, funded by the Robert Wood Johnson Foundation, showing that one of six students, mostly age 12, say they have experienced physical dating violence. The report did not release the figures for boys and girls separately.
Given the recent campaign to eliminate male violence against females, iMediaEthics was interested in what the poll showed about victimization rates for both boys and girls. Shari Miller of RTI agreed to share the information with iMediaEthics, but was subsequently overruled by others at RTI – a decision upheld by RWJF.
Why? It appears they may not have wanted to admit publicly that girls are often violent toward their partners. A substantiating clue lies in the results of another study in which Miller was a major participant. That one showed that among sixth grade students, boys were twice as likely as girls to report being slapped, hit or punched by their dating partner.
A report by the Center for Disease Control also shows that high school boys are a little more likely to be victims of physical violence by their dating partners than are girls.
For the general public, used to campaigns that focus on male violence against women and girls (such as the “1 is 2 many” White House campaign), these results could be shocking. Indeed, one TV station that commented on the study, immediately assumed the RTI figures were all about male violence.
No one can deny that girls are at higher risk of injury than boys, and that they need more protection than boys. But should researchers not tell the whole truth about their data? Should they continually try to dupe the public into believing that in partner relationships, female violence is never a complicating factor?
Since when is it appropriate for researchers to cover up serious research data because they don’t want to acknowledge its results?
2. Be Sure to Read the Fine Print Award
Winner: The AP news release of its poll on racial attitudes, conducted by researchers from Stanford University, the University of Michigan, and NORC at the University of Chicago for its misleading analysis that “51 percent of Americans now express explicit anti-black attitudes.” It turns out that the “explicit” measure of black attitudes used by the researchers was not necessarily a measure of racial attitudes after all.
A reader wouldn’t know that from the AP news release itself, because it treated the measure of “explicit” racism as though it were completely valid. But at the end of the story, in fine print, is a link to the academic report, which provided the basis for the news story. There the academic researchers admit that their measure of racism may not be valid at all, and that any conclusions they make in their report about racism may in fact may be “partly or mainly” due to other factors.
What other factors? As I noted in an earlier post, the researchers cite a study that suggests the measure of explicit racism they used could well be more a measure of non-racial conservative attitudes, rather than prejudice against blacks.
There is little doubt that racism is alive and well in the United States. And it’s important to use valid measures to assess how widespread it is. But did 51 percent of Americans express explicit anti-black attitudes as AP reported? Or was it just that about half the country expressed explicitly conservative attitudes? Based on the AP poll and the academic report, the latter conclusion is just as probable as the former.
But we wouldn’t know that if we hadn’t read the fine print.
1. Wishing Won’t Make It So Award
Neil Newhouse, Mitt Romney’s pollster. (Credit: AP, screenshot)
Winner: Neil Newhouse and Public Opinion Strategies for giving false encouragement to Republicans on the likelihood of a Romney victory.
It was in the news everywhere: Mitt Romney was so convinced he was going to win the election, he was stunned when Election Day showed otherwise. And that certainty had spread to other Republican leaders and pundits across the country as well.
Who was responsible for that misplaced optimism? The pollster, Neil Newhouse, and his associates at Public Opinion Strategies. Their polling suggested that Florida, North Carolina, and Virginia were clearly in Romney’s camp, and that Romney also had significant leads in Colorado, Iowa and New Hampshire. (Obama won all those states except North Carolina.)
Yes, there were the really, really far-out (Romney electoral landslide) predictions by GOP pundits, Karl Rove and Dick Morris, but they aren’t pollsters. Public Opinion Strategies, however, is one of the most reputable GOP polling organization in the country, and Neil Newhouse among the most respected pollsters. As Ed Goeas of the Tarrance Group, reputable Republican polling organization itself, commented before the election: “He’s [Newhouse] one of the best and certainly has to be seen as the premier pollster on the Republican side this year.”
But reading the explanations Newhouse and his associates offer for what went wrong suggests they abandoned empirical observations for speculative assumptions. They didn’t anticipate the increased support for Obama among Latinos and Asians, and nor the large turnout by young people and Democrats.
But that’s what polls are for – to measure just those voter characteristics!
Newhouse and his associates also had to dismiss the mountain of polling data, accumulated by several websites (such as Huffington Post and Nate Silver’s Fivethirtyeight Blog), which showed Obama significantly ahead – even with a 90%+ chance of winning.
It appears that Newhouse and his associates pollsters were more influenced by what they wished to see happen than by what was actually happening. As Lev Grossman reminds us in The Magicians, however, “Wishing doesn’t make it so.”
Note: To see all 50 winners of the last half decade, go to our “Dubious Polling” Awards section.