iMediaEthics' latest PollCheck raises serious questions about GfK's The Knowledge Panel ™, a web-based form of surveying the public. (Credit: The Knowledge Panel ™, screenshot)
STORY HIGHLIGHTS :
|
“Top-of-Mind” Public Opinion on Trans Fats
In early November, iMediaEthics commissioned a poll by GfK that was designed to be a “poll check” on an October Pew poll, which we felt represented a “top-of-mind” view, but not a very realistic measure of what the public was really thinking.
In half the sample, we had GfK replicate the Pew poll, by asking straight out whether respondents favored or opposed a policy that would prohibit restaurants from using trans fats. In the other half of the sample, we preceded the policy question by first asking whether people had heard of trans fats and if they knew whether fats and trans fats were the same.
For the time being, we can forget about the results from the second half the sample, because the results based on the first half were so startling they raised serious questions in our minds about the validity of the GfK poll more generally.
While Pew had reported a majority of Americans opposed to the policy (44% in favor, 52% opposed), the new GfK poll – three weeks later – showed a substantial majority in favor – 64% to 33%.
We were highly doubtful that public opinion on the matter could have changed so dramatically in just three weeks, so we asked GfK’s Randall Thomas, Vice President for Online Research Methods, why he thought there was such a discrepancy. He speculated that in fact public opinion probably had changed over the three-week period.
4 Points About 3 iMediaEthics’ Polls
1. In November, GfK conducted a web-based poll for iMediaEthics. This poll was to test Pew’s October phone poll saying 44% of Americans are in favor of banning trans fats in restaurants. The new GfK poll found a stunning 64% in favor. This 20-point increase could be attributed either to a change in public opinion or to the mode of surveying (using a web-based panel instead of phone polling).
2. To test which explanation was more accurate, we commissioned two more polls in December — one by GfK, using its web-based sample, and one by PSRAI, which conducted the original poll for Pew by telephone.
3. The results were again very far apart: GfK found 65% in favor of banning trans fats, while PSRAI found 48%.
4. These results confirm that the differences in results were caused by different samples (web vs. phone), and not by a change in public opinion.
To test this hypothesis, we re-ran the GfK poll from Dec. 6-8, and at the same time commissioned Princeton Survey Research Associates International (PSRAI) to run the exact same question in the same time period. PSRAI conducted the early November poll for Pew, so we would have a direct comparison between the two types of polls. The PSRAI poll we commissioned was fielded from Dec. 5-8. This meant that any differences between the PSRAI phone poll and the GfK web-based poll could not be due to opinion changing over time.
Again, the results showed a wide discrepancy. The PSRAI poll found 48% of Americans in favor of the policy to prohibit restaurants from using trans fats, with 46% opposed. These results are a bit more favorable than the Pew poll conducted by PSRAI a month earlier, but overall they are consistent with an interpretation that the public is about evenly divided over the issue.
The GfK poll, by contrast, showed – once again – a largely favorable public: 65% in favor, 33% opposed, virtually identical results to what GfK obtained in its first poll for us.
The overall results are shown below:
Note that while the PSRAI results indicate a fairly evenly divided public, both GfK poll results suggest a public about two-to-one in favor of prohibiting restaurants from using trans fats.
Again, we asked Thomas why he thought there was such a large difference between the GfK results and the PSRAI results. He conceded that the cause “most likely” was a “mode effect where how an item and responses are presented can affect response.” The PSRAI survey was administered by phone, where live interviewers are asking the questions; the GfK survey was administered on the web, where the questions are presented in a visual format and respondents check off their responses.
However, we take issue with characterizing the difference between the two polls simply as a “mode effect.” Yes, there are differences in the way the questions were administered, but more important in our minds are the differences in the types of people who were polled.
Phone Samples vs. Online Panels for Web-Based Surveys
The PSRAI phone samples are based on a random selection of both cell phone and landline telephone numbers. As indicated earlier, the GfK sample is taken from The KnowledgePanelTM, a large data base of respondents who have agreed to be part of a group of people who will answer polls several times during the year.
That panel of respondents was obtained (and is updated) by using scientific methods of identifying potential respondents (similar to the process used by PSRAI). The process is too complicated to go into detail here (see Chang and Krosnick, pp.4-6 for a detailed description), but the important point to know is that both phone and GfK’s web-based samples are initially based on scientifically-designed probability selection methods.
However, after that, the process for GfK differs in very significant ways from the way phone samples are obtained.
After identifying potential respondents, pollsters conducting phone surveys ask the individuals to participate in just one poll for a few minutes. In contrast, GfK asks the potential respondents to participate in an on-going series of web-based polls, conducted several times a year, for which they are given monetary incentives. If the respondents don’t have access to the Internet, they are given a web browser to do so.
Scholarly studies show that panel participants (as opposed to one-time respondents for typical telephone surveys) tend to be much more interested, informed and engaged in following the news. That certainly seems to be the case with respect to the issue of trans fats.
Recent news stories have reported on the dangers of trans fats, and that there are healthier substitutes. One would expect most well-informed people to be in favor of a policy that helps to get rid of unhealthy foods, especially since there are substitutes that enhance food texture in the same way that trans fats do – but without the bad effects.
Still, Thomas argues that we can’t know which set of results about public opinion on trans fats is the more valid – those from the PSRAI phone surveys or the GfK web surveys. Technically, he is correct. We can know public opinion only from surveys, and if the polls disagree, we can only speculate as to which is the more valid.
Still, we think the phone surveys are highly likely to be more accurate reflections of the public than the web surveys, because of the nature of the respondents. It’s one thing for respondents to answer a few questions if they are called to complete a phone survey, but it’s much more burdensome to agree to be on an ongoing panel like The KnowledgePanelTM. And many fewer respondents would be willing to commit themselves to that process.
Moreover, as indicated above, the characteristics of people who are willing to be part of such a panel, and to regularly use computers for monetary incentives, are quite different from the characteristics of people generally across the country.
Panel members are much more likely than the general public to be aware of current issues, like the movement to rid the food industry of trans fats because of their unhealthy characteristics. And it’s that kind of engagement bias that, we believe, explains why the GfK panel is more supportive of the policy to get rid of trans fats than the phone samples interviewed by PSRAI.
Meaningful Public Opinion on Trans Fats
In this report, we’ve characterized the results of the GfK and the PSRAI polls as reflecting “top-of-mind” public opinion. That’s because the question is in a forced-choice format – asking if people favor or oppose the policy, but not providing an explicit “don’t know” option. Research has shown that people will often offer opinions on a subject in the context of a poll, even if they don’t have one, simply to satisfy the demands of the interview.
In order to measure “meaningful” public opinion – views that people hold with some degree of intensity, rather than just their top-of-mind responses – policy questions should ideally include a tag line, such as “or don’t you have an opinion?”
In the absence of such an explicit approach, pollsters can at least try to determine whether respondents really care about the opinion they’ve expressed. But that was not done in the original Pew report on trans fats.
We suspected then that a lot of people really didn’t have much of an opinion about trans fats, and that Pew’s report in early November was highly misleading. That’s why we commissioned the first GfK poll, to provide a “poll check,” by asking the Pew forced-choice question, but then following up by seeing if people really cared about their answers.
As just discussed, that first GfK poll raised red flags about the validity of web-based surveys, and the discussion so far has focused on why the online samples are not likely to be as representative of the general public as phone polls.
But the issue of measuring a meaningful public opinion remains a central concern. And in these poll results, we can see why the forced-choice format generally provides misleading results.
How Much People Care About Their Responses
To determine how much respondents really cared about the responses they gave in the polls, we added a follow-up question to those who expressed a view about prohibiting restaurants from using trans fats. If they supported the policy, we asked how upset they would be if such a policy were not adopted. If they opposed the policy, we asked how upset they would be if the policy were adopted.
People who said they would be “very” or “somewhat” upset were classified as having a “meaningful” opinion. People who said they would be “not too” or “not at all” upset were classified as not caring one way or the other.
The rationale is that if you say you prefer something to happen, but then immediately acknowledge that you would not be upset if the opposite actually came to pass, then it’s reasonable to conclude that you don’t really care one way or the other about the policy.
The results for the two December polls are as follows:
What we classify as a meaningful opinion is bolded and highlighted in yellow. People who favor the policy, but wouldn’t be upset if it were not adopted; people who oppose the policy, but wouldn’t be upset if it were adopted; and people who initially expressed no opinion at all – they are all considered not to care about the policy one way or the other.
That leaves just 30% of Americans who have a meaningful opinion according to the PSRAI poll, and they are evenly divided on the issue – 15% in favor, 15% opposed. The GfK poll shows 39% of Americans with a meaningful opinion, split two-to-one in favor, 26% to 13%
The results are graphed below:
While these two sets of results clearly differ from each other, they both show that a large majority of the public has no meaningful opinion on whether restaurants should be prohibited from using trans fats. And that is the major conclusion to be drawn about public opinion on this matter.
Among people with a meaningful opinion, GfK and PSRAI obviously show conflicting results – but taken in the context of a generally unengaged public on the issue, the differences appear relatively unimportant.
By contrast, when treating the “top-of-mind” results as though they are meaningful, the differences between the GfK and PSRAI polls appear to be monumental. Indeed, they show a 30-point swing – from just a 2-point margin in favor to a 32-point margin.
Those results are also hardly credible: Who really thinks (as the forced-choice questions suggest) that over 90% of the public has formulated a meaningful opinion on trans fats? As iMediaEthics pointed out in an earlier post, we don’t know how many people are even aware of what trans fats are.
Implications
Importance of Measuring “Meaningful” Opinion
In the polls we discussed in this article, the “top-of-mind” results suggest two very different publics – one that is overwhelmingly in favor of prohibiting restaurants from using trans fats, and another that is evenly divided.
However, when we examine intensely held opinions (views that people actually care about), the differences between the GfK polls and the PSRAI polls are diminished. Both sets of polls show a highly unengaged public, with somewhere between 30% and 40% with meaningful opinions.
Among the engaged, differences between the polls persist – with GfK showing a 13-point margin in favor of the policy, and PSRAI showing an even split. But this difference is much smaller than the 30-point difference when measuring “top-of-mind” opinion.
Skepticism About the Validity of GfK’s Web-Based Surveys
While we had expected the GfK web-based surveys to provide public opinion results very similar to those obtained by good telephone polls, that was not the case. We were very surprised at the large discrepancy between the GfK results and the PSRAI results, because of studies suggesting that GfK’s KnowledgePanelTM provided scientifically representative samples of the American public.
Those studies, however, were limited (as are all such studies, including our own), and for the most part focused on demographic characteristics of the people in the samples – not on their views about policy matters.
Our study focuses specifically on just one policy issue, prohibiting restaurants from using trans fats, so we can’t conclude that GfK’s web samples are not representative of the American public on other issues.
Still, this study suggests a great deal of caution in interpreting any polling results based on GfK’s KnowledgePanelTM. People willing to be part of that panel appear quite different in their level of knowledge and engagement in current events from the public more generally, and thus may have quite different opinions about policy matters.
More studies, of course, can help us understand when the GfK web-based panel is not a good representation of the public, and when it might be. But in the meantime, as pollsters we should be very skeptical about trusting the results of web-based polls, even if – like the GfK polls – they are based on scientifically-selected samples.
GfK responds to iMediaEthics’ results, iMediaEthics responds to its claims
David W. Moore is a Senior Fellow with the Carsey Institute at the University of New Hampshire. He is a former Vice President of the Gallup Organization and was a senior editor with the Gallup Poll for thirteen years. He is author of The Opinion Makers: An Insider Exposes the Truth Behind the Polls (Beacon, 2008; trade paperback edition, 2009). Publishers’ Weekly refers to it as a “succinct and damning critique…Keen and witty throughout.” He and Andrew E. Smith are writing a book about the New Hampshire Primary.
Andrew E. Smith is Associate Professor and Director of the Survey Center at the University of New Hampshire. He teaches in the Political Science Department and has published numerous scholarly articles on polling. He has also conducted numerous polls for the Boston Globe, USA Today, CNN, Fox News, several local television stations, and various state and local government agencies. He and David W. Moore are writing a book about the New Hampshire Primary.
