A recent poll, conducted March 3-6, 2016, by NBC News and the Wall Street Journal shows how misleading is the industry’s “message testing” approach to measuring public opinion.

Instead of measuring what Americans are currently thinking, this approach measures what Americans might think if they were fed the same biased information as provided to respondents in a poll – but then treats the results as though they represent what Americans are thinking now.

The poll sample included 1,200 registered voters nationwide, but was split into two randomly assigned groups, Form A and Form B. The first form, asked of half the sample, included this question:

NBC/WSJ (Form A): “Recently, a Supreme Court Justice passed away leaving a vacancy on the court. President Obama will nominate a new person to serve on the U.S. Supreme Court.

Would you prefer the U.S. Senate vote this year on the replacement nominated by President Obama, or leave the position vacant and wait to vote next year on the replacement nominated by the new president, or do you not have an opinion one way or the other?”

This question format is fairly objective, giving no cues in favor of holding hearings now or waiting until next year. The results show an 11-point margin in favor of holding hearings this year (48% to 37%), with 15% expressing no opinion.

Unlike most polls, this one did include an explicit “no opinion” option, much to the credit of the pollsters. However, the poll did not ask an intensity question to discover how strongly respondents felt about the issue. Had that been done, the poll would no doubt have shown even more than 15% of respondents lacking a meaningful opinion – perhaps as much as a third to one half.

The Form B version of the question, asked of the second half of the sample, was clearly intended to test how people might feel if they were given an argument in favor of holding hearings now, with no counter argument in favor of the opposite position. This was the message testing question wording:

NBC/WSJ (Form B): “The U.S. Constitution requires the president to nominate a candidate to fill an opening on the Supreme Court. The U.S. Senate has the responsibility to confirm or reject a nominee. Republicans who control the Senate say they will not hold confirmation hearings and have no plans to consider a nominee put forward by President Obama.

Do you approve or disapprove of this decision not to consider President Obama’s nominee, or do you not have an opinion one way or the other?”

Note that this “question” first tells respondents that, according to the U.S. Constitution, the president is required to nominate a candidate to fill the current opening (left by Justice Antonin Scalia’s death), and the Senate has a responsibility to conduct hearings. The preamble then tells respondents the Senate Republican leaders have refused to consider any candidate President Obama might nominate. Then, finally, respondents are asked whether they approve or disapprove of the GOP leadership decision.

This phrasing, of course, reflects the Democrats’ argument as to why the Senate should hold hearings, and makes no mention of the arguments that Republicans are using to justify their refusal to do so. It doesn’t even note that the GOP leaders say they will hold hearings with a new president.

The results, predictably, show a strong margin of support for the implied option: By a 2-to-1 margin, respondents disapprove of the Senate’s strategy.

The Usefulness of Message Testing

Such a biased preamble to the question can be useful for campaigns, as pollsters engage in “message testing” – trying to see which arguments might be most appealing to the public. The NBC/WSJ message testing in Form B shows that the Democrats’ argument, in favor of the Senate holding hearings, is quite persuasive. But those results do not characterize “public opinion” as it exists right now.

The pollsters could have also done some message testing about the Republicans’ arguments. Instead of splitting the sample into two groups, they could have split the sample into three groups – one group asked the mostly objective question (Form A), a second group asked the biased question that favored the Democrats (Form B), and a third group that asked a biased, message-testing argument in favor of the Republicans.

But the pollsters didn’t do that. And even if they had, only the results produced by the first form could be considered representative of what the public is currently thinking.

The two message-testing forms don’t measure current public opinion at all. Once respondents have been fed information by the pollsters, the sample is tainted – it can no longer be used to represent the public at large, which has not been subjected to exactly the same information as the sample.

In the real world, the public is subjected to a wide variety of arguments, and the only way to avoid tainting a sample is to refrain from giving the respondents any substantive information. Respondents should be offering their opinions based on what they have heard prior to the start of the interview, rather than on what the pollsters tell them.

Otherwise, the poll is message testing, not current public opinion.

Postscript:

It’s surprising that the NBC/WSJ poll included such a partisan question wording in Form B. The poll is jointly conducted by a Democratic polling firm (Peter D. Hart Research Associates) and a Republican polling firm (Public Opinion Strategies), which advertises itself as “the leading Republican polling company” in the country.

It appears as though the Republicans got snookered into asking a question that overwhelmingly favors the Democrats