Earlier this week, a new USA Today/Suffolk University poll made big claims about the Mueller investigation, President Trump’s “witch hunt” claims and Trump impeachment possibilities.
Specifically, the poll report claimed “half of Americans agree with President Trump’s contention that he has been the victim of a ‘witch hunt,'” and the public’s support for a potential House of Representatives impeachment has dropped.
This “shock poll,” as one media outlet announced, is nevertheless plagued by several problems, which call into question its “stunning” results.
The faulty question about the witch hunt has already received push back from several polling experts, because of the wording:
“President Trump has called the Special Counsel’s investigation a ‘witch hunt,’ and said he’s been subjected to more investigations than previous presidents because of his politics. Do you agree?”
There are three major problems with this question.
- The question is triple-barreled. It asks whether voters agree that there is a witch hunt, whether voters believe Trump has been subjected to more investigations than previous presidents, and whether the reason for those investigations is Trump’s politics. It’s quite possible, for example, that respondents could disagree the investigation is a “witch hunt,” but acknowledge Trump has been subjected to more investigations than other presidents. We don’t know what respondents actually agreed with.
- The question itself is unbalanced: It asks only if voters agree. Research on question wording going back several decades has shown that such unbalanced questions are biased, subtly suggesting to some respondents that “agree” is the right answer. At the very least, the poll should have asked if respondents “agree or disagree.”
- The question provides information to the sample of respondents – that Trump has called the investigation a “witch hunt.” Probably most voters across the country actually know that, but certainly not all. By insuring that all respondents know that information, while some unknown but probably significant part of the electorate does not, means that the sample no longer represents the larger population of American voters.
Wording is not the only problem. The context of the question is also important. Immediately preceding the “witch hunt” question was another (biased) one that provided information to respondents that could well have influenced some respondents to be critical of the investigation.
It’s important to remember that on any given issue, large segments of the public are simply unaware of any meaningful details. People have lives to lead and can be overwhelmed by the abundance of detailed political news that floods the various media outlets.
In polls like this one, such low-information respondents rely heavily on information provided in the questionnaire itself to formulate their opinions. This is especially true when the questions are “forced-choice” – questions that ask for A or B, and do not explicitly tell the respondent that it’s okay to say “I don’t know.” Under pressure to come up with an opinion, respondents are especially susceptible to cues in the questions.
Right before the question about the Special Counsel’s investigation, the poll asked this question:
“As you may know, the new Democratic majority in the House of Representatives has launched a series of investigations, including issuing subpoenas, into President Trump and his administration. Which comes closer to your view?[ROTATE]
- “Congressional Democrats are doing the right thing in pursuing these investigations aggressively
or
- “Congressional Democrats are going too far with these investigations
After receiving this information, the sample of USA Today respondents no longer represented the American electorate.
We don’t know (because the poll didn’t ask) how many respondents already knew that 1) there was a new Democratic majority if the House, 2) that it had launched a series of investigations, and 3) that Democrats were pursuing them “aggressively.”
.
What we do know is right after this tendentious question, the poll then asked if respondents agreed with Trump’s views about the Special Counsel’s investigation. For those who were generally unaware of House activity, being informed of it (and having it characterized as “aggressive”) right before the “witch hunt” question, could certainly have influenced their responses.
The net result is that we can’t trust the results of this poll as they pertain to voters across the country.
Impeachment Sentiment Declined?
Another claim by the March poll is that support for impeachment fell by ten points since last October. This also is a finding that seems more a product of faulty questionnaire design than what actual public opinion is.
The question in both polls asked:
“Do you think the House of Representatives should seriously consider impeaching President Trump?”
Notice that this question, like the “witch hunt” question, is unbalanced. Elementary question-wording standards demand that any policy question provide both a positive and negative response. In this case, at the very least, the question should have included “or not” at the end. A better way would probably be to say “should or should not” in the text of the question.
For the time being, we can overlook the faulty wording and focus instead on the change in responses from October, when 38.5% said yes, and March, when 28.4% said yes.
As the USA Today article noted, that represented a 10-point drop. Given that the identical question was asked, one might be inclined to accept this conclusion.
However, there is a major questionnaire context problem with the two questions. In October, the impeachment question was asked shortly after the question on presidential approval (54% disapproved).
In March, the impeachment question was asked immediately after a question that informed respondents that “Nancy Pelosi said she was not for the impeachment of President Trump.”
The question went on to ask if her opinion had an impact on the respondents’ views. Four in ten indicated at least a “little” impact. Almost three in ten said the information had either “some” or “a lot” of impact.
Note again – once the sample of respondents was informed of Pelosi’s statement, the sample no longer represented the larger American electorate. Not all voters in the country know about Pelosi’s views, and many may still believe impeachment is acceptable.
Furthermore, by giving the information about Pelosi right before the impeachment question, the poll subtly suggested the “right” answer was not to impeach.
We can’t say for certain whether the lower percentages reflected an actual decline in support among Democrats and independents nationally, or whether the lower numbers pertain only to the sample of respondents.
But respondents themselves believed their views were influenced by the information given to them in the poll. It seems likely that if the Pelosi question had not preceded the impeachment question, the results in March may well have been similar to those last October.
Lower Confidence in Mueller?
A complementary finding in the USA Today story is that public trust in Mueller has “eroded.” There is some plausible support for this conclusion, but it is weak.
In the October poll, 57% of the sample said they had either “a lot” of trust (37%) or “some” trust (20%) in Mueller’s investigation. In March, 55% said they had either “a lot” (28%) or “some” (27%) trust.
There was only a two percentage point difference in total of people who had “a lot” or “some” trust. The decline was in the percentage who say “a lot” – by nine points.
It’s not clear how to interpret this fine difference between “a lot” and “some.” If there really is a drop in Mueller’s image, it should be reflected in other questions. The results of several questions, including Mueller’s favorability rating, have not yet been released.
It’s worth noting that the percentage of respondents who said they had “little or no” trust in Mueller was essentially unchanged – 32.2% in October, 35.5% in March. That three point difference is well within the poll’s margin of error.
It’s not clear from the poll that a significant change in overall public opinion about the Mueller report has occurred.
And it would appear that the most shocking result of this poll – that half the public believes Trump is a victim of a “witch hunt” – is more a reflection of faulty questionnaire design than of public opinion itself.
Disclosure: In July 2017, USA Today/iMediaEthics conducted a poll on the public opinion of possible impeachment of Pres. Trump. Read that report. iMediaEthics’ policy is to not accept any money from news media outlets; we received no monies form USA Today for this joint poll.