The day after Britain voted to leave the European Union — Brexit — The New York Times’ Nate Cohn wrote an article exonerating the polls with the headline, “Why the Surprise over ‘Brexit?’ Don’t Blame the Polls.”
I disagree.
Yes, the polls showed a close race. And perhaps, as a consequence, everyone should have known that the vote could have gone either way. But, based on those polls, a general consensus emerged that ‘Brexit’ would fail. The polls, and many of the people who interpret them, misled us.
As Huffington Post’s Natalie Jackson wrote the day after the vote: “Polls had indicated the vote would be very close, but most last-minute surveys showed ‘remain’ leading. Two internet pollsters conducted day-of surveys on Thursday that were released after the polls closed. YouGov and Ipsos-MORI both indicated that ‘remain’ would win the day — YouGov had the status quo up by 4 points, and Ipsos-MORI had it up by 6 points.”
Jackson does indicate that in general, the online polls trended toward Leave, while the live phone polls trended toward Remain. But there’s little reason why most people would trust the online polls more than the phone polls, when the latter method has been the gold standard for years. Only recently have some pollsters suggested there is no gold standard any more.
Moreover, despite Cohn’s headline suggesting the polls shouldn’t be blamed for the surprise over the ‘Brexit’ vote, Cohn himself seems to suggest the polls were at fault – at least to some degree. As he noted in his article,
“The late polls really did break toward Remain, especially after the British Labour Party member Jo Cox was killed on June 16. Over all, Brexit led in just four of the final 11 pre-election surveys — enough to give Remain 0.5-point lead in the HuffPost Pollster average. The Remain side did even better in the very last polls, including a four-point lead in a YouGov poll conducted on Election Day.”
Then he adds:
“One could certainly argue that the polls were ‘wrong’ in the sense that they tended to show a slight Remain advantage heading into the vote count. But it was clearly a distinct possibility that Brexit would win, based on the available survey data. So it’s hard to argue that this was a big polling failure, and it’s a bit strange the financial markets appear to have been caught completely by surprise.”
So, in Cohn’s view, the surprise over the ‘Brexit’ results wasn’t a “big” polling failure. But it was a failure of some magnitude. A failure which, as he admits, caught the financial markets completely by surprise. So, maybe it was a “big” polling failure after all.
It wasn’t just the vote results that fooled the markets. It was another question, which Cohn notes, also suggested a win for the Remain option: What the public’s expectations were about the vote. And on this question, the polls found most people predicting Remain would win over Leave.
The day before the vote, Professor Claire Durand of the University of Montreal, who is vice president and president-elect of the World Association for Public Opinion Research, announced that her compilation of polling data showed a distinct trend in favor of Remain: “With these new results, we may conclude that Remain is likely to end up with a clear advantage of at least four points.”
In fact, the four-point margin was in the opposite direction.
In her postmortem the next day, Durand suggested that maybe the polls weren’t all that wrong. Rather, she wrote, it was the analysts (including her) who misinterpreted the data.
Perhaps. But to the general public, that distinction can get lost.
Cohn offers advice: “If there’s a lesson here, it’s a fairly straightforward one: Don’t get too confident in contests when the polls show a very tight race.”
Good advice for the pollsters, pundits, and reporters who inform us what the polls allegedly reveal.