CNN’s polling director Jennifer Agiesta announced new standards including a 16-question test for reporting on polling last month. “It doesn’t take much to conduct a poll these days, but it does take quite a lot of care to conduct one well,” CNN reminded.
Questions include who conducted the poll, who paid for it, how many people were interviewed, how they were interviewed and in what language and what the sample and selection process was. Other requirements include the text of the questions and instructions, what the margin of error is, and if there was any response weighting. The 16 CNN survey questions are published on CNN’s website.
David Moore, iMediaEthics’ polling director, said he welcomes the new CNN policy as an “excellent contribution to the polling industry” and a reaction to the changing industry practices.
“CNN’s guidelines highlight the importance of vetting polls before their results are covered,” Moore said. “Other media organizations should consider following CNN’s example.”
As Moore explained, online polls were once considered substandard but now online polls help provide public opinion in ways traditional landline calls can’t, if modeled appropriately. “It used to be that only probability samples were considered appropriate for high quality public opinion polls,” Moore said. “With probability samples, in theory everyone in the population being studied has a known chance of being included in the sample. For telephone polls, that means it’s important to include the appropriate proportions of both cell and landline phone numbers. However, it has become increasingly expensive to reach people by phone, because of the massive non-response problem: Literally, most people do not answer calls from pollsters.”
The new CNN standards highlight transparency, likening the polling vetting process to source verification. “A journalist must be satisfied that the source’s information is solid before it goes out to the public,” CNN’s Agieta explained. “Our evaluation of polls and surveys should meet that same standard, and the initial approach ought to be skeptical: A poll’s existence alone does not make it news, and journalists shouldn’t be any more credulous about numbers than they are about words.”
CNN said it thinks that random landline phone samples are no longer “the gold standard” because cell phone users are excluded, but calling cell phones is expensive and hard. CNN won’t cover campaign polls and robocall polls.
CNN said:
“There is not one correct way to answer the 16 questions that make up our standards questionnaire, but there are some things that have not met CNN’s standards for reporting in the past and will not now. And there are some new, clear lines that define the way polls should be done today. We will not cover:
- Polls that are conducted by campaigns or by those who have a financial or advocacy interest in the outcome
- Polls that are conducted by telephone using robocalls rather than live interviewers
- Polls that are conducted without any type of sampling, where anyone who chooses to can participate
- Polls that are conducted solely using unrepresentative sample sources
- Polls that are conducted without accounting for people who take surveys on their cellphones, either over the phone or by web
- Polls on some topics which do not ensure that respondents of all education levels are adequately reflected