Three recent polls have all purported to measure the public’s reaction to the revelations that the National Security Agency (NSA) has engaged in tracking massive numbers of Americans’ email and phone records. But the polls have come up with wildly contradictory results. Pew says the public is greatly in favor of the tracking efforts, while Gallup and CBS say the public greatly opposes them.
Which is it? Does a majority of the public support, or oppose, the tracking program?
Here are the three headlines:
- Pew/Washington Post poll: “Majority Views NSA Phone Tracking as Acceptable Anti-terror Tactic”
- CBS News poll: “Most Disapprove of Gov’t Phone Snooping of Ordinary Americans”
- Gallup poll: “Americans Disapprove of Government Surveillance Programs”
Why do the pollsters show such disparate results, anyway?
Note a 15-point margin (Pew) in favor of the NSA program, compared with a 20-point margin against (CBS) or a 16-point margin against (Gallup). Between Pew and CBS, there is a 35-point swing in opinion.
The pollsters will quickly explain the discrepancies by pointing to the different question wording of each poll. But we shouldn’t be fooled by these specious explanations. If slight changes in question wording can produce such contradictory results, how can we really trust any poll that presumes to measure the public’s policy preferences?
Pew Priming Its Respondents In Favor of Terrorist Threat
Apart from question wording, probably the main difference between Pew on one hand, and CBS and Gallup on the other, is that Pew primed their respondents to think about the fear of terrorism before asking about the NSA program, while CBS and Gallup did not. In fact, CBS (and perhaps Gallup) primed their respondents in the opposite direction.
The prior question that Pew asked was framed this way (response options were rotated):
“What do you think is more important right now:
- “For the federal government to investigate possible terrorist threats, even if that intrudes on personal privacy, or
- “For the federal government not to intrude on personal privacy, even if that limits its ability to investigate possible terrorist threats?”
As is immediately evident, the question is so vague it invites most people to pick option #1. Of course we want the federal government to investigate possible terrorist threats! Yes, we expect the government to intrude to some unknown extent on personal privacy, but isn’t that what happens every day when people go through the security line to board an airplane?
By contrast, option #2 implies the government will be severely constrained in fighting terrorism if it doesn’t intrude on personal privacy at all.
So, it’s not surprising that most people do in fact choose option #1, by close to a two-to-one margin, 62% to 34%.
Now, with most people “primed” to be thinking about how important it is to sacrifice some personal privacy to fight terrorism, Pew presents this question:
“As you may know, it has been reported that the National Security Agency has been getting secret court orders to track telephone call records of MILLIONS of Americans in an effort to investigate terrorism. Would you consider this access to telephone call records an acceptable or unacceptable way for the federal government to investigate terrorism?”
For many respondents, who have just said that the government needs to investigate terrorism threats even if it intrudes on personal privacy, they almost have to agree that the NSA program is “acceptable” – because it is an actual program that fits the more general description respondents have just rated favorably.
Thus, with that series of questions, Pew found most people responding in favor of the NSA program, 56% to 41%.
CBS Priming Respondents in Favor of Personal Privacy
It appears as though CBS primed its respondents in the opposite direction from what Pew did. (CBS did not provide a topline on its website, but the questions and results are included on the Polling Report.)
Prior to asking about government phone and Internet tracking practices, CBS asked how concerned people were “about losing some of your privacy as a result of steps taken by the federal government to fight terrorism?” About six in ten were very or somewhat concerned.
Having just raised this concern about loss of privacy, CBS asked:
“It was recently revealed that federal government agencies have been collecting records of phone calls and internet activity in their efforts to fight terrorism. How closely have you been following news about this: very closely, somewhat closely, not very closely, or not at all closely?”
“In order to reduce the threat of terrorism, do you approve or disapprove of federal government agencies collecting phone records of ordinary Americans?”
With most respondents having admitted concern about loss of privacy, it’s not surprising that most also disapproved of the government collecting phone records of ordinary Americans – by 58% to 38%. These results provided the basis of the CBS headline, that most people oppose the government’s snooping on ordinary Americans.
Gallup Provided No Priming
In this initial post, I indicated that it was unclear whether Gallup preceded its NSA question with a prior question about personal privacy, because until then, no one from Gallup had responded to my request for the relevant information (and there was no topline posted on its website, nor on the Polling Report).
However, since then, on June 17, the Managing Editor, Jeff Jones, sent me the topline results which showed that, unlike the CBS poll, the Gallup poll did not prime its respondents to think about privacy issues before asking the major policy question about the NSA tracking program.
Gallup’s first question on that issue was as follows:
“As you may know, as part of its efforts to investigate terrorism, a federal government agency obtained records from larger U.S. telephone and internet companies in order to compile telephone call logs and internet communications. How closely have you been following the news about this — very closely, somewhat closely, not too closely, or not at all?”
After that question, Gallup asked:
“Based on what you have heard or read about the program to compile telephone logs and internet communications, would you say you approve or disapprove of this government program?”
As shown in the graph at the beginning of this post, Gallup’s results showed a net disapproval, similar to the results that CBS obtained, even though Gallup did not prime its respondents in that direction.
These results suggest that the “top-of-mind” inclination of a majority of Americans is against the government’s efforts to track citizens’ telephone and internet use, and that Pew’s finding in the other direction was probably due to its first question on the trade-off between investigating terrorists threats and intruding on privacy versus not investigating terrorist threats.
A Realistic View of Public Opinion
All three pollsters asked their respondents how closely they had followed the NSA issue, and all three found just over a quarter of Americans following the issue “very closely,” while they found from about a third to half not paying close attention at all.
So, how could they report 90% or more of the public with a meaningful opinion about the NSA tracking program? The answer: They all used a “forced-choice” question format, which pressures respondents to make an on-the-spot decision, regardless of how committed they might be to that view. Thus, many people with no real views on the matter had to come up with one, and were thus highly influenced by the priming they had undergone during the interview itself.
To its credit, Pew asked people whether they felt “strongly” or “somewhat” about their responses, with the result that among those who had a strong opinion, sentiments were about evenly divided: 30% felt strongly the NSA program was acceptable, 27% felt strongly it was unacceptable.
Oddly, Gallup helps reinforce one side of that equation. In the follow-up question about the NSA program, Gallup asked the opponents if there would ever be circumstances when it would be right for the government to establish a large data base on its citizens, or would it not be right ever to do that? This follow-up is essentially an intensity question, showing that 30% of Americans felt such tracking would never be right.
The 30% of Americans reported by Gallup as being firmly against such tracking is quite close to the 27% that Pew found – even though Gallup and Pew presented quite different pictures of the public when opinion intensity was ignored.
So, what seems to be the most realistic picture of public opinion? It appears as though anywhere from a third to half the public is unengaged on the issue, and doesn’t have a meaningful opinion at this point. The rest of the public appears to be evenly divided.
This is a very different picture from what any of the three pollsters discuss. Instead, they all focus on the collective “whim” rather than “will” of the public, and that “whim” can show a large majority in favor of the program (Pew) or a large majority opposed (Gallup and CBS).
Polls can provide meaningful measures of what the public is thinking, but by mostly ignoring intensity, pollsters overlook some of the most important characteristics of public opinion. Instead, they highlight results that just don’t make sense – as the chart at the beginning of this article makes clear.
UPDATE: 6/18/2013: 2:58 PM EST: iMediaEthics wanted to know if Gallup asked the question about people’s concern for security before it asked about the NSA tracking program. This information was not was not available online, and Gallup hadn’t responded to iMediaEthics’ request before publication. Gallup has now responded that they asked the security question after the NSA tracking question, so iMediaEthics has written an update to incorporate this information into the story.