Who is going to win the presidential horse race? (Credit: J. Cameron, 1884 Currier & Ives print)

Too much is being made of Gallup’s recent announcement that it will not conduct any horserace polling during the primary season.

Politico called it a “stunning move,” given that “Gallup has been the country’s gold standard for horse-race election polling ever since its legendary founder, George Gallup, predicted Franklin Roosevelt’s landslide reelection in 1936.”

Reactions from pundits and pollsters, posted on HuffPollster, suggest the move is a sad commentary on the state of the polling industry overall, and a poor reflection of Gallup specifically.

I disagree.

Primaries are notoriously difficult to poll, because the competition within each party is often among candidates who do not differ much from each other. It’s very easy for voters to fluctuate in their preferences from one candidate to another, based on the latest piece of information they might hear – either in ads or in the news media. Even by election time, many voters are undecided, making poll predictions tenuous at best.

Moreover, continuous polling provides little useful information about the stakes in the election. Polling the national electorate, voters in each party who think they might participate in their state’s caucus or primary, is even less useful than horserace polling within each state. That’s because there is no national electorate. There are only state contests. And the results in the first contest in Iowa will affect voters in New Hampshire, and the results there will affect voters in South Carolina, and on and on through Nevada and then Super Tuesday, and the other states, so that trying to understand the nomination process based on national polls is a fool’s errand.

The Acid Test of Polls: Election Results

Still, some commentators argue that Gallup’s move means that we’ll no longer be able to judge the accuracy of its polls, because we’ll not be able to compare the poll results with the actual election. Mark Mellman, a Democratic pollster, in an email to Huffpollster, wrote:

“Elections are a test for pollsters and it feels like they are shying away from taking the test. And if the company says, in effect, ‘you can’t trust our election numbers,’ why should anyone trust their other numbers?”

A similar comment comes from William Jordan, YouGov:

“Problem here is accuracy of horserace is key indicator of accuracy in non-horserace.”

And Harry Enten, writing on Fivethirtyeight.com:

“By forgoing horse-race polls, Gallup has taken away a tool to judge its results publicly.”

What these critics apparently do not realize is that even in the 2012 election cycle, Gallup did not poll in any of the primary or caucus states, where the final election results could be a test of their polling accuracy.

Gallup did poll the national GOP electorate; in fact, Gallup tracked the results on a daily basis for several months. But as already noted, there is no national primary. There are only state primaries and caucuses. So, there is no way to compare polls of a national sample of potential primary voters with an election outcome in any given state.

But it wasn’t just Gallup in 2012 that refrained from polling in a primary or caucus state. As it turns out, none of the other major media pollsters, except NBC, did so either.

For example, HuffPollster was tracking 55 polls from 16 polling groups during the GOP campaign in the 2012 New Hampshire primary. Of the major media polls (which I identify as Gallup, Pew, ABC/Washington Post, CBS/New York Times, CNN, FOX, and NBC/Wall Street Journal, plus possibly LA Times/Bloomberg), only CNN and NBC (without the Wall Street Journal) reported poll results.

But CNN opted not to poll in the final few days, which meant its results would not be compared with election results. Only NBC, among the major media pollsters, subjected itself to the “election test” that Mellman says is so important.

Yet, there was no outcry from pollsters and pundits in 2012 about our inability to judge the accuracy of the major media pollsters.  

By the way, polling in New Hampshire early on, but then not polling close to the election in order to avoid the election test, was a strategy followed in 2008 by ABC/Washington Post, Pew, New York Times, and LA Times/Bloomberg – all of which polled at least once in the months leading up to the primary, but did not poll in the final five days before the election.

The only major media polls that subjected themselves to the election test in New Hampshire that year were Gallup, CBS (without the New York Times), CNN and FOX.

The 2008 Debacle in New Hampshire

And the results were terrible for pollsters. 

Nine pollsters (the four major media pollsters noted above, and five other polling units) all released their final pre-election polls for the New Hampshire Democratic primary at least four days prior to actual balloting – between Friday and Monday before the Tuesday election. In the same time period, eight pollsters released their final pre-election polls for the GOP primary (CBS apparently polled only the Democratic electorate).

All but one pollster correctly predicted John McCain to beat Mitt Romney, with the victory margins ranging from four to nine points (McCain won by five). That was fine. But all of the pollsters incorrectly predicted Barack Obama to beat Hillary Clinton, who actually won by two points, with the polls’ margins of victory off by a low of five points to a high of 18.

That debacle apparently scared off the major media pollsters four years later, leaving NBC – which did not poll in New Hampshire in 2008 – as the only major media pollster in the 2012 New Hampshire primary.

Not Polling the National Electorate

Gallup’s decision not to poll in the primary means only that it will not poll in New Hampshire (which was the only state Gallup typically polled during the primary season, but did not do in 2012 anyway), and that it won’t poll the national primary electorates.

Gallup’s experience from the past two elections should have demonstrated that polling the national electorate after the Iowa Caucuses is a waste of time and resources. Preferences of voters nationally reflect candidates’ performances in the state contests. The national electorate does not predict how candidates will fare in subsequent contests.

Note in HuffPollster’s chart of the 2012 “National GOP Primary” (a misnomer to be sure), Rick Santorum languished far behind the other candidates until he essentially tied with Romney in Iowa, after which his support soared. But when he lost badly in New Hampshire and then South Carolina, his support declined. Gingrich got a boost from his win in South Carolina, but his subsequent losses resulted in a drop in support nationally. As Romney prevailed in state contests, voters nationally rallied around him.

The same pattern also prevailed in both the Democratic and Republican national electorates in 2008. As candidates did well in state contests, their support nationally increased.

In the end, the key to tracking the race for the presidential nomination is how many delegates a candidate is accruing during the primaries, not how good his/her poll numbers might be among the national party electorate.

Thus, kudos to Gallup for not wasting its resources on polling the national GOP and Democratic electorates. Gallup’s Frank Newport explained the decision this way:

“In the 2012 cycle we invested a huge amount of time, money and interviewing in tracking the horse race on a nightly basis. Our question in this cycle: is this the best investment of resources to fulfill the mission of helping understand what is going on in a presidential election and hopefully helping make the nation better off as a result.

“Our thinking is that it is not; that tilting those resources more toward understanding where the public stands on the issues of the day, how they are reacting to the proposals put forth by the candidates, what it is they want the candidates to do, and what messages or images of the candidates are seeping into the public’s consciousness can make a more lasting contribution.”

In principle, the rationale seems unassailable. In practice…we’ll see.

Disclosure: iMediaEthics’ polling director David W. Moore was a senior editor of the Gallup Poll for thirteen years, from 1993-2006.