Editor’s note: Every January, iMediaEthics’ polling director David W. Moore assembles the top ten “Dubious Polling” Awards for iMediaEthics. The tongue-in-cheek awards “honor” the previous year’s most questionable actions in media polling news. The 2021 awards are the 13th in this series.

10. Captain Crozier Fan Club Award

Winner: Yahoo News/YouGov for its over-the-top favorable poll about Navy Captain Brett Crozier’s efforts to protect his men from Covid-19. 

It’s understandable why so many people might sympathize with a leader who is trying to protect his troops – especially during this pandemic period.

That assumes, of course, that people had even heard of CPT Crozier. According to the YouGov poll, 42% had not.

That so many respondents were unaware of the incident, however, didn’t matter to YouGov.

YouGov told them he was a hero and then asked if they agreed or disagreed that he should have been “fired.” (He was actually relieved from that specific assignment, not fired.)

YouGov didn’t mention the problems with a military officer using the media to pressure his superiors. The issue was a tad more complicated than what YouGov told its respondents.

Not a surprise – seven in ten respondents said they sided with CPT Crozier, while 13% sided with the military authorities. Another 18% were unsure.

I personally agree that CPT Crozier was a hero. But polls should be designed to find out what the public thinks, not what I and YouGov and Crozier’s Fan Club think. 

Next time, perhaps, YouGov could adopt a little more professional approach to complicated issues?

9. Cognitive Dissonance Award

Winner: Fox Host Melissa Francis for her ability to argue two contradictory ideas at once.

On the one hand, she is adamant that she doesn’t trust “any polling at all,” thinks her own network’s polls are also “garbage,” and said she will deliberately lie to interviewers when she is polled. 

On the other hand, she loves polls that support her political views and will hype them on Twitter.

The dissonance might seem to be overwhelming for most mortals. But for many at Fox, dealing with cognitive dissonance – the conflict between conspiracy and reality – seems to be the modus operandi.

8. Defender of the Faith Award 

Winner: Professor Robert Y. Shapiro at Columbia University for coming to the defense of polls in the Washington Post, arguing that whatever the quirks with election polling, that does not mean that “mass public opinion polling” has problems. In fact, he claims, it is “alive and well.”

That is a nice bromide for the media and the general public, but it raises the question: What exactly does it mean that mass public opinion is “alive and well”?

As numerous posts on this site have demonstrated (see, for example, here, here, here, and here), general public opinion polls often show differences that are far greater than what election polls show. 

For a very recent case, see the Polling Report about President Trump’s approval rating post-January 6. The net ratings (percent approval minus percent disapproval) range from -12 to -28 percentage points, a 16-point variation. 

Public opinion may be “alive,” but can we believe it is “well”?

The truth is that pollsters are facing a stiff challenge in obtaining representative samples of respondents, because the vast majority of people simply won’t participate – a problem noted by Cliff Zukin five and a half years ago that has only worsened. We can’t be sure that any given poll accurately represents the views of the general public. 

And contrary to what Shapiro argues, public opinion polling is actually more fraught with potential errors than election polling. 

In addition to the problem of getting decently representative samples, there is the challenge of crafting objective question wording. Also, unlike election polling, which produces numerous polls that can be averaged, in public policy polling there are often only a few polls that address any given issue. Averaging the results is much less useful with so few polls on an issue, especially when they provide contradictory results.

Being a Defender of the Faith, when the Faith is so obviously under stress, is a thankless task.

7. 2 + 2 =3 Award

Winner: CNN for its poll on Biden, Trump & battleground states in May that found Biden leading Trump by five points nationally, but had Biden losing in battleground states by seven points – numbers that just don’t add up!

As I noted at the time, CNN reported that while Biden’s national margin was 2.5 times greater than Clinton’s final lead in 2016, he was actually doing worse in battleground states – with a losing margin that was 4.2 times greater than Clinton’s. 

Impossible. 

For Biden to be trailing Trump by seven points in the battleground states, and still leading nationally by five points, he would have had to be losing in Colorado, Georgia, Maine, Minnesota, Nevada, and New Mexico by an average of more than ten points. (Biden actually won all those states.)

CNN seemed to recognize the dilemma and downplayed its battleground results. At the time, I wrote to Jennifer Agiesta – CNN Politics Polling Director –to ask about the methodology that had led to such an irresolvable result. She didn’t respond. 

Apparently, the less said, the better.

6. Hindsight is 2020 Award

Winner: Professor Helmut Norpoth of Stony Brook University for his post-election explanation as to why his presidential election model failed so spectacularly.

It predicted Trump to have a 91% chance of winning a landslide re-election, earning 362 electoral votes to Biden’s 176. That total represented an increase in Trump’s 2016 electoral vote margin of 306 to 232. One could only wonder what planet Norpoth was on for the past four years, that he didn’t recognize how dotty that scenario was!  

He made the prediction in March and stuck with it until the election. It was based solely on how well the candidates fared during the primary contests. Nothing else mattered – the pandemic, the economic devastation, Black Lives Matter protests, behavior of the president – all, according to his model, were irrelevant.  

In a press release from his employer calling him the “maverick modeller,” he proudly averred, “I don’t use any polling data or data related to opinions.”

So, what happened that caused his model to be so far off the mark? After the election, Norpoth lamented:

“The answer is simple and apparent: a perfect storm. A once-in-a-century combination of election-year surprises wreaked havoc on a forecast posted as early as March 2.  The first was the outbreak of the coronavirus pandemic; then, triggered by the lockdown aimed to keep people safe, came an economic downturn of a scale not seen since the Great Depression; then, with the same goal in mind, came an unprecedented expansion of voting by mail; add to that the killing of George Floyd, which sparked a wave of racial unrest not seen since the 1960’s.  

“This cascade of election-year ‘surprises’ was unprecedented. It was bound to affect the outcome of the election in November.”

Of course, polling data during the campaign could have provided Norpoth a warning that his model was not working. And, ironically, he agrees: The key indicator that his model was not working was that “Trump failed to raise his approval rating in the course of the election year above the 50% mark, which would promise victory.”

If only he hadn’t been such a maverick and instead had paid attention to the polls.

5. Public Cynicism Award

Winner: NBC News/Wall Street Journal poll – for its creative question in a January 2020 poll that made the public appear to be dyed-in-the-wool cynics. Not true!

The poll was about the public’s reaction to the first impeachment of Donald Trump. One question asked if respondents thought that Democrats, and separately Republicans, in the House and Senate, were more focused on “upholding the U.S. Constitution” or “more focused on politically motivated considerations.”

The results showed about six in ten Americans chose “politically motivated considerations” for each group, while about a third chose “upholding the U.S. Constitution.”

It is the case that the public is generally skeptical about unnamed politicians. And asking questions about the motivations of politicians in general, is a cheap way to show public cynicism.

It’s even cheaper when the question itself confuses the issue with a vague option of “politically motivated considerations.” After all, an impeachment proceeding is about politics. And it is about the U.S. Constitution. It’s about both things.

In contrast to this poll, Gallup polls over the years have shown that when people know about their own congressional representative, they tend to be quite positive. 

Those results, I believe, are more reflective of the American public than what the NBC/WSJ poll would have us believe.

4. Oops! Award

Winner: Washington Post columnist Philip Bump. In a July article, Bump dismissed several polls that suggested Trump had a chance to win the electoral vote. “If these are the polls he is using, no wonder Trump thinks 2020 is going well,” said the headline.

And then Bump went on to write off those poll results because they were so at odds with polls included in RealClearPolitics averages. Except…those RCP averages weren’t so hot. And, as the table below shows, Trump’s polls were not far off the final result.

The president had good reason to believe that the electoral vote could possibly go his way, although certainly not the popular vote.   

In fact, though Biden won the popular vote by more than seven million votes, he won the electoral vote by just 65,485 votes – because of his victories in Wisconsin, Arizona, Georgia, and Nebraska’s second congressional district. That total represents four hundredths of one percent of the vote (.04%). Biden won the popular vote by more than 100 times that amount (4.45%). 

Had Biden not won those 38 electoral votes, Trump would have won the presidency by 270 to 268 electoral votes (a far cry, it should be noted, from the 362 predicted by Norpoth).

So, the polls that Trump was seeing in July and that Bump so scornfully disdained were, actually, not so bad after all. 

What’s a journalist to say?

Oops!

3. Wizard of Oz Award

Winner: ABC News Director of Elections, Dan Merkle – for elucidating the public about the nature of exit polls and how they would operate on Election Night, without ever mentioning the magic behind the curtain. 

Merkle explained what exit polls were, how they were conducted, the kinds of questions that are included, whether they are accurate, how they account for people who vote early or by mail, how exit polling has changed because of the coronavirus, and when exit poll results would be reported. He also noted:

“On Election Day, there is a strict embargo on any data coming from the early waves of exit poll data until 5 p.m. ET. By about 5:45 p.m., some initial demographic information about voters and their views on key issues in the election will be available on ABCNews.com. After the polls close in a state, the complete exit poll crosstabs will be posted on ABCNews.com.

“ABC News will not project a winner until the last scheduled poll closing time in each state. If a race is not projected at poll closing time, the projection will incorporate actual vote data and will be made as soon as the data warrant. Information will be constantly updated throughout the evening on ABCNews.com and on all ABC News programs.”

It would appear to the casual reader of Merkle’s post that ABC, and ABC alone, does an awesome job with the exit poll operation. 

Yet, behind the Wizard’s curtain is – Edison Research!

Yes, it’s Edison Research that designs, conducts, and analyzes exit polls on Election Night, providing the data to the National Election Pool, which is a consortium of major news networks – ABC, CBS, CNN and NBC.

Merkle may want us to believe they are the exit poll gurus, but a peek behind the curtain reveals otherwise. 

2. Irrational Public Award

Winner: Pew, Quinnipiac, USA Today/Ipsos, and Morning Consult/Politico – for their separate polls on Iran following the killing of Iranian General Qassem Soleimani a year ago, which makes the public appear irrational, if not completely loony. 

Polls asked if respondents supported or opposed the killing, and then asked whether it made the U.S. more safe or less safe. In each poll, majorities or pluralities favored the action, but then also opined that the action made the U.S. less safe!

How can Americans support an action that they also believe, by substantial margins, makes the country less safe? That does not make sense. By any normal standard, it is irrational.

The USA Today/Ipsos poll, for example, found a 9-point margin in favor of the killing (42% in support, 33% opposed). But the very next question showed a two-to-one majority saying the killing made the U.S. less, rather than more, safe (24% to 55%) – a 40-point swing!

The other polls also found what appears to be an irrational public, though with smaller swings: Pew by 21 points; Morning Consult/Politico, 25 points; and Quinnipiac 17 points.

But appearances deceive. The picture of an irrational public is a function of the polls, not the public. 

Over the years, polls have consistently shown that few Americans know much about world affairs or about U.S. foreign policy more specifically.  

Yet none of those polls bothered to find out if respondents had even heard of the killing, much less how many had thought much about the implications of such an assassination.

Asking an ill-informed public about a complicated issue will inevitably produce results that appear contradictory, and even nutty. But we shouldn’t confuse ignorance with irrationality, regardless of what some polls might signal.

1. Shooting Blanks Award

Winner: John McLaughlin of McLaughlin & Associates – for his June accusation that NBC, ABC, and CNN were “intentionally” producing biased polls (i.e., ones that showed Trump trailing Biden in the national contest), in order to suppress the Trump vote.

It was highly unusual for a pollster to attack the motives of his professional colleagues, including fellow GOP pollster, Bill McInturff (one of the two lead pollsters for the WSJ/NBC poll).

Prior to that outburst, McLaughlin was probably best known for his less-than-pinpoint accuracy in 2014 for then-House Majority Leader Eric Cantor. Two weeks before the primary election, he had Cantor ahead of his opponent by 34 points, but Cantor lost by 11 points – a 45 point error that probably set the record for the largest miscalculation by a reputable pollster since God created polling. 

It turns out, that unfortunate debacle was not an aberration. In 2012, he racked up double digit polling errors in Connecticut, Pennsylvania, Virginia (two contests), Illinois, Massachusetts, and New York. “I don’t understand how he wasn’t run out of the business years ago,” according to one GOP strategist. 

An indication of how disingenuous were his latest claims is that no mention was made about the Fox News poll of the same time period, though its results and method were similar to the NBC/WSJ poll that McLaughlin had targeted.

As it turns out, however, there was never any evidence produced to support the accusation against NBC, ABC, and CNN, and no other public statements or claims were ever made against any of those networks for those alleged crimes.

The pollster was firing blanks. As, of course, he likely knew.