There’s buzz over at the Associated Press, which has just updated its AP Stylebook with a new chapter on how to report on polls. Their Polling Editor, Emily Swanson, says “We’re excited about this much needed update to our survey standards.”
As the AP announcement notes:
The new chapter, available immediately to AP Stylebook Online subscribers, leads with longstanding guidance that the mere existence of a poll is not enough to make news. It adds that “poll results that seek to preview the outcome of an election must never be the lead, headline or single subject of any story.”
Why the update?
David Scott, the Deputy Managing Editor for Operations, who oversees AP’s polling unit, noted that “the 2016 election was a reminder that polls aren’t perfect. They’re unquestionably a piece of the story, but never the whole story. The Stylebook update aims to serve as a steady reminder of that fact.”
Steven Shepard of Politico writes that the update comes in the wake of “successive presidential contests in which the polling industry and the media have come under harsh scrutiny for their election-year practices,” and that the new Stylebook chapter on polling “is the latest effort from pollsters and journalists to improve coverage of survey data.” He specifically notes the incorrect predictions of a Hillary Clinton victory, based on faulty polls, mostly at the state level.
But does the update actually address the faulty poll problem?
Not really.
AP appears to be conflating two major problems: 1) the faulty predictions of a Clinton victory based on polling data, and 2) excessive coverage of horserace polling, presumably at the expense of covering issues and candidates.
The catalyst for the update, AP indicates, is the faulty prediction, but the update itself seems designed to address excessive poll coverage. And the two problems are not the same.
As Everett Carll Ladd, then director of the Roper Center for Public Opinion Research, wrote in 1980: “The prominence given to opinion data by the press is unfortunately quite apart from the inaccuracy of what is so often reported.”* (emphasis added)
So, let’s look at the two problems and how the AP update affects each.
Problem #1: Inaccurate Election Prediction
It’s important to note that the national polls were fairly accurate in their prediction that Clinton would win the popular vote. The Real Clear Politics final average showed Clinton ahead by 3.2 percentage points; she actually won by a 2.1 percentage point margin.
So, when AP’s David Scott notes that the polls in 2016 weren’t perfect, he’s really referring to the widespread consensus, based on polls, that Clinton would win the electoral vote. And, of course, she didn’t.
The main reason the media portrayed the election as an almost certain win for Clinton is that several reputable polling sites used aggregated poll results to show Clinton with a high probability of winning the electoral vote:
- Nate Silver of FiveThirtyEight gave Clinton a 71% chance of winning.
- The Upshot at the New York Times showed her with an 85% chance.
- And HuffPollster reported a 98% chance of a Clinton victory.
However, from the limited description of the update on the AP website (one has to subscribe to read the whole chapter), it appears that the new guidelines do not really address the problem of overstating Clinton’s electoral strength. That problem was based on the predictions provided by the data aggregators mentioned above – all of whom enjoy decent professional reputations.
The guidelines do suggest certain standards for individual polls, but nothing for how to assess the quality of the predictions made by the statistical polling wizards who run those sites.
If all the reporters had followed the new AP Stylebook on polling in 2016, they would still have reported the overwhelming consensus of a Clinton victory, based on the statistical models reported by FiveThirtyEight, Upshot, and HuffPollster.
Of course, there were many faulty polls that distorted the election predictions, mostly at the state level, but that problem needs to be addressed by the data aggregators. And there is no doubt they are already aware of the problem. The new AP guidelines are hardly likely to contribute to that solution.
Problem #2: Excessive Coverage of Horserace Polling
The real emphasis of the AP announcement seems to be on too much coverage of horserace polls. For over half a century, political scientists have lamented that the news media spends too much time on which candidates are ahead and behind in the polls, and too little time on the issues.
To counter this tendency, AP has long cautioned its readers that a horserace poll is not by itself worthy of a news story. It must be linked to something more than who is ahead and behind. The new guidelines are more emphatic. They declare that horserace poll results “must never be the lead, headline, or single subject of any story.” (emphasis added)
Well, good luck with that.
In 1976, the New York Times and CBS, which had formed a joint polling unit, agreed that they would never release horserace numbers at all. Instead, all their polls would focus on issues. This was an experiment in what the two organizations felt was enlightened news reporting. But as CBS’s Warren Mitofsky later acknowledged, the experiment was a disaster. Reporters rebelled. Academic pollsters rebelled. They all wanted to know the whole set of poll results – about the issues, and the relative standings of the candidates. Reporters did their best to work around the restrictions, characterizing the standings as well as they could without mentioning the specific percentages.**
The two news organizations essentially admitted failure, and have never tried that experiment again.
Of course, the AP guidelines do not suggest omitting any mention of candidate standings, only that they be integrated into larger news stories. Still, I suspect there are two likely outcomes of this AP effort to improve polling coverage.
The first is that many reporters will simply ignore the mandates. I certainly do expect to see news headlines that focus on which candidates are doing well, and which are not. Such headlines are too much part of the news culture to simply go away.
The second likely outcome is that other reporters will simply work around the restrictions, so that the “excessive” coverage of poll results will continue unabated.
How can reporters get around the restrictions?
It would be easy to meet the requirement that a horserace story shouldn’t be the “single subject of any story.” All a reporter has to do is throw in some issue or campaign tactic, along with the poll results about who is leading, to meet that requirement. Most stories I’ve seen, in fact, already do that.
It’s a tad more difficult, perhaps, for a reporter to avoid covering the horserace aspects in the lead of the story. Still, I can envision a story talking about recent appearances of the candidates someplace, and then immediately launching into poll results showing who is leading and who is behind.
Such minor adjustments to the story may meet the letter of the new Stylebook, but will hardly change the overall reporting of poll results.
The other mandate, that horserace poll results should never appear in a headline, is more difficult to circumvent. Or maybe it’s not much of a mandate at all. It depends on how to interpret the wording.
As noted earlier, “Poll results that seek to preview the outcome of an election” is the actual phrase. Most headlines I’ve seen don’t put numbers in a headline, but instead indicate one candidate is “leading” or “trailing” another candidate.
Does that formulation qualify as putting “poll results” into the headline, even if no numbers are mentioned? If not, then the mandate means very little. If so, it could be difficult for reporters (actually, the editors who usually provide the headlines) to get around that restriction. Thus, they may simply ignore it.
No one knows for certain, of course, how the journalistic world will react to the new AP guidelines related to polls. My sense is that however well intentioned, they will have little effect on how much coverage is given to polls.
As superficial as it might be, we all want to know who’s winning the horserace.
_________
*Everett Carll Ladd, “Polling and the Press: The Clash of Institutional Imperatives,” Public Opinion Quarterly, 44 (Winter 1980): 574-584.
**See David W. Moore, The Superpollsters: How They Measure and Manipulate Public Opinion in America (New York: Four Walls Eight Windows, 2nd Ed., 1995), 278-79.