8th Annual Dubious Polling Awards: 2015’s Worst in Media Polling

Editor’s note: Every January, iMediaEthics’ polling director David Moore assembles the top ten “Dubious Polling Awards” for iMediaEthics. The tongue-in-cheek awards “honor” the previous year’s most questionable actions in media polling. 

Moore, Senior Fellow with the Carsey School of Public Policy at the University of New Hampshire and a former Gallup Vice President and senior editor, analyzes and fact checks polls, always calling for the media to abide by best polling practices. His work can be found in our Polling Center. Check out previous years’ Dubious Polling Awards. And the winners are:

 

10. “The Left Hand Doesn’t Know What the Right Hand Is Doing” Award

Winner: CNN for announcing two conflicting reports on the results of its own poll about the 2016 presidential race – one showing Hillary Clinton’s ratings hurt by an email controversy, the other showing them unaffected.

When networks conduct their own polls, you might think the director of polling and the reporters might talk with each other – you know, so the reporters know what the data actually mean.

Somehow that apparently didn’t happen last March 16, when Briana Keiler presented a report for the Situation Room that claimed “Clinton rating drops amid email uproar.”

Two days later, CNN’s polling director, Jennifer Agiesta, announced that CNN’s new poll found Clinton in a commanding position vis a vis her GOP opponents, and that “her prospects for 2016 are largely unchanged compared to polls conducted before the news broke about her use of a personal email address….”

Oops! Shudda checked before sounding off!

 

9 . “Leading the Witness” Award

Winner: Bloomberg, for its innovative questions about Iran and income inequality, which let respondents know which answers were most acceptable.

Suppose for some odd reason you want to know whether Americans think Iran is a reliable negotiating partner. At first blush you might hesitate, knowing that most people probably don’t know even where the country is. Still, it’s been in the news recently, almost all stories being negative, so at least most people will recognize the name. And, of course, given all the bad things that have been said about the country, it would be shocking if the poll showed many people with a positive view.

But…! What about those people who are totally clueless about world news? When pressured in a poll to come up with an evaluation of Iran’s negotiating reliability, such ignoramuses might accidentally offer neutral, or even positive responses. Bloomberg to the rescue.

A less creative pollster might ask a simple, non-tendentious question: “When it comes to negotiations between countries, do you think the Iranian government is reliable, or unreliable – or aren’t you sure?” But that’s too straightforward for Bloomberg, which recognized that some respondents might not know the “right” answer. So, it designed question that makes the right choice a little clearer:

“Do you think the Iranian government has evolved to be reliable as a negotiating partner, or does their continuing religious theocracy make them unreliable as negotiators?” (emphasis added)

Yes. “Unreliable” is the acceptable response. How did you guess? Just 20 percent said Iran had “evolved” to be a reliable partner, while 68 percent took the cue to say “unreliable.” (How dense is that 20 percent?!)

And that wasn’t the only creative question Bloomberg asked. While a simple, straightforward question by CBS revealed a public largely favoring government action to reduce the income gap between the rich and the not rich, Bloomberg was able to measure an equally divided public. How? By suggesting any government action would undermine that shibboleth of American capitalism – the free market.

OK! So, that’s a false trade-off. The point is: Bloomberg really wants the respondents to get the right answer.

 

8. “The Pollyanna” Award

Winners: ABC News/Washington Post, CNN, and Pew polls for creating, almost out of whole cloth, the illusion of public opinion on the nuclear deal with Iran.

It’s tempting to think that in a democracy, most people have carefully considered, informed views of the major issues facing the state. And with the help of modern polls, we can create the illusion of that ever-optimistic Pollyanna world.

Take, for example, the issue of the nuclear deal with Iran that included the United States and several other countries. Do Americans support or oppose it – or do they not have an opinion either way?

Well, according to three different polling organizations, at least 78 percent of all Americans have a meaningful opinion on this issue (Pew), to as high as 93 percent (ABC/WP) and 95 percent (CNN).

That’s really good news for democracy! So many people engaged in the issue.

If only it were true.

An iMediaEthics poll that chose to reveal the gritty truth about public opinion, whatever it might be, suggests that in reality, three-quarters of the public have no idea what the nuclear deal is all about, and two-thirds of the public doesn’t have a meaningful opinion on the issue one way or the other.

The Pollyanna world may be more conducive to our concept of democracy, but it contains some confusing views: either a 19-point margin of support for the nuclear deal (ABC/WP), or 8-point (CNN) and 9-point (Pew) margins against.

Well, no world is perfect. Not even a Pollyanna one.

 

7. The “Netanyahu Stand-Off” Award

Winners: CNN and Paragon Insights for devising questions that produced opposing findings about the American public’s view of the Israeli Prime Minister’s visit to the United States.

It’s really difficult to ask questions about foreign leaders of a public that is largely uninformed about events overseas. But if a prominent leader is coming to America, Journalism 101 requires – yes, it might as well be written in the Constitution – there be a poll to measure what the public is thinking, even if not at all.

Kudos to the CNN poll and, separately, to Paragon Insights‘s poll, for creating the illusion of a highly engaged public, even though only about one in six Americans had heard much about the visit and most couldn’t have cared less.

CNN reported public hostile to the visit by a 30-point margin, while Paragon Insights found much friendlier public, with an 18-point margin more positive than negative.

How is that possible you say? Good question. Theirs weren’t.

6. The Loyal Trump Fan Club Award

Winner: Gallup for its poll showing Donald Trump as America’s second most admired man in the world, tied with Pope Francis.

Who wudda thought? With the Republican Party hoping the Donald’s campaign will collapse and Democrats outraged at many of the businessman’s pronouncements, the self-admitted billionaire and man with “a very good memory,” was nevertheless cited by Gallup as more admired by Americans than anyone except Barack Obama.

Or maybe not.

It turns out, it all depends on how badly one can design a questionnaire. And none better on this issue than Gallup.

Which even Gallup admits.

Hmm…

 

5. The “Melodrama” Award

Winner: Politico and a gaggle of pollsters for their reaction to Gallup’s announcement it would not conduct horserace polling during the primary season.

Noting that “Gallup has been the country’s gold standard for horse race election polling ever since its legendary founder, George Gallup, predicted Franklin Roosevelt’s landslide re-election in 1936,” Politico called Gallup’s decision a stunning move. Other pollsters lamented the move, suggesting Gallup’s accuracy could no longer be checked against the reality of an election.

Come on, guys!

Gallup’s decision to forego national polls of hypothetical voters was a good thing. Really! Now journalists don’t have to pretend that such results actually tell us anything of value.

As I noted last October in my positive reaction to Gallup’s decision, New Hampshire is the only state where Gallup has ever polled so its results could not be compared against a national election. And with no big company announcement, there were no lamentations or beating of breasts from the polling community when Gallup didn’t poll there four years ago either.

Moreover, Gallup’s extensive tracking of the national primary electorates that consumed so much of the company’s resources four years ago could never be compared against an election anyway – because (remember?) there was no national primary. As there never has been. Nor will there be this election cycle.

 

4. The “Impossible Dream” Award

Winners: Pew Research and Princeton Survey Research Associates International, for producing the perfect poll – where 100 PERCENT OF THE PEOPLE have the same opinion!

Is it possible to find a large population where everyone shares the same view – on any subject? Though it may not be widely known, that is the secret dream of every pollster, the impossible dream of a poll revealing true harmony, the tune of Man of La Mancha ringing in our ears:

This is my goal, to follow that dream

No matter how silly it might make me seem

When people together can ALL agree

With at least something, whatever it be

Well, folks: It’s no longer just a dream! Welcome to the Don Quixotes of the polling world: Pew Research (which sponsored the poll) and Princeton Survey Research Associates International (which conducted the poll). They have accomplished what no pollster has ever done before in the history of the world: They have found 100 percent of a public with the same view!

Note that in Lebanon, a country of over six million people with some 18 religious sects, 100 percent of its citizens have an unfavorable view of ISIS. (Another 1 percent didn’t have an opinion, making the total 101 percent. But let’s not quibble about rounding error.)

I can’t think of a question a pollster could ask of Americans that would generate such unanimity. Even if we asked, “Would you favor drinking water or swallowing a razor blade?”, I’m sure there would be at least some measurable percent who would choose razor blade – just to show their independence.

But no such prank occurred in Lebanon. No sireee. Somehow everyone agrees that ISIS is not good.

Now some doubters might suggest that such unanimity could never be achieved in a country so torn by sectarian strife that its elections have been put off to 2017, though they were initially scheduled for 2013. As the European Forum describes the situation:

“The Syrian conflict has divided Lebanese society along religious lines. Hezbollah fights and lobbies to save the Syrian regime, the Resistance is backing the Syrian government out of fear of future reprisals, the Sunnite leadership claims neutrality and Christians are split: some are neutral, others support Assad as they fear a power vacuum will create Islamist control in the country.”

Could such a divided society really all have the same view of ISIS?

Such unanimity might make some skeptics wonder whether people asking the questions might have intimidated respondents into giving an acceptable answer.

Were the interviewers, for example, the ones who actually spoke with the citizens perhaps Hezbollah soldiers in full military garb wearing “I Hate ISIS” buttons?

Did the question about having a favorable or unfavorable view of “the Islamic militant group in Iraq and Syria known as ISIS” get translated into Arabic as “the people who disrespect Muhammed and deny the existence of Allah, known as ISIS”? I can see where anyone might be reluctant to have a favorable view of such a heretical group.

By the way, why did the pollster call it a “militant” group anyway? Does “militant” get translated as “extremist and radical”, which essentially tells respondents the group is really, really bad?

Whatever! Pew and PSRAI have achieved what we pollsters thought was simply unachievable – the Impossible Dream with the Impossible Poll. They should apply to the Guinness World Records so posterity will never forget.

 

3. “Ignoring the Elephant in the Room” Award

Winners: Nate Silver and Cliff Zukin, for their otherwise enlightening articles that focus only on the problems of election polling, but say nothing about the ongoing problems of public policy polling, Silver’s “The World May have Polling Problem,” and Zukin’s “What’s the Matter with Polling?”

 

 

 

 

 

It’s not that election polling doesn’t have problems these days. Both Silver and Zukin address those issues – mostly by focusing on the difficulty of obtaining accurate samples of voters. Several recent election miscalls, noted by both Silver and Zukin, illustrate that difficulty. The “problem” they are concerned with is that the final election polls – some 24 to 48 hours before the election – can be off by several percentage points.

But let’s be serious, folks. Who really cares if the polls are wrong so close to the election? All we have to do is wait for a couple of days, and we get the right information.

It’s the polls that measure public opinion on policy matters that should concern us. While there’s no election to help validate the polls, we know that when polls come to wildly contradictory conclusions, they can’t all be right. And as I’ve noted earlier, such situations of conflicting polls are quite common and have been for years – long before concern about election polling arose.

But neither Silver nor Zukin have anything to say about these examples of contradictory results. The elephant has been in the room so long, hardly anyone notices.

 

2. “Be Afraid! Very Afraid!” Award

Winners: The British newspaper, The Sun, and PunchingBagPost for finding scary poll numbers in what most sentient beings would find quite reassuring.

 

Although two polls – one by Survation (reported on by The Sun) and the other by Pew (reported on by PunchingBagPost) – showed the vast majority of Muslims with negative views of ISIS, somehow both The Sun and PunchingBagPost were able to divine shocking public support for that militant group in Syria.

Even when it received 1,200 complaints about its over-the-top report of the poll results, The Sun stood by its distortion. And PunchingBagPost admitted up front it was not trying to be objective when it manipulated the data to make the results as scary as possible.

Yes! Conjuring up scary headlines, regardless of truth, gets more attention than reassuring ones.

If you’re not afraid now, you will be!

 

1. “Giving the Store Away” Award:

 

Winners: The Republican and Democratic Parties for allowing the media to apply arbitrary polling standards for candidates to qualify for debates.

Most pollsters and pundits recognize that the use of polls to include some candidates and exclude others from the presidential debates is arbitrary and unwise. As Politico reported, there was unanimous opposition to this idea among the pollsters it interviewed, because pollsters recognize that early polls are not good indicators of eventual voter support.

Why? Professor Cliff Zukin of Rutgers University, and past president of the American Association for Public Opinion Research, explained to NPR reporter, Danielle Kurtzleben, that early polls measure mostly name recognition. “But,” he asks, “is name recognition really the way you want to choose a presidential candidate?”

Well…apparently, yes! At least, that seems to be the case based on the actions of the two political parties.

After all, the two parties could have established their own criteria. They did set the number and timing of the debates, and the formats used by the media. But they allowed television producers to exclude some candidates from the so-called “main stage” on the basis of polls that anyone who knows anything about polls also knows that early national polls of hypothetical primary voters are close to useless in predicting actual voter support.

And to compound the error, the television producers then arranged all the candidates on stage in accordance with their standings in these meaningless polls.

Of course, there is a self-fulfilling prophecy in this operation. Once the previously meaningless polls are used to select candidates, they do in fact become meaningful. As the NPR reporter noted,

“So the debates could create a feedback loop. Good poll numbers help a candidate get into the main debate. More exposure there can get a candidate more name recognition, which can then boost that candidate in the polls.”

While endorsements and help in fundraising, along with control over the debates, were the means by which party leaders could influence the nominating process, the infusion of large sums of money into the campaigns, permitted by Citizens United, has eroded the influence of party leaders. And then they give up even their control of debates to the media, ceding virtually all influence over a process that is central to the parties.

Of course, this year the consequences seem far greater for the Republican Party than the Democratic Party, though both parties have set precedents which will not serve them well.

They have indeed given the store away.