(Credit: Flickr, Photos of Money)
Last week I blogged about the ABC/Washington Post poll on the sequester, citing with approval Matthew Yglesias’ characterization of the article reporting the results as “deeply misleading.”
Patrick J. Moynihan, a former senior polling analyst at ABC News, tweeted that my analysis was “unfair,” arguing that “Outright dismissal of research w/o considering more than the headline or b/c few q’s were asked is simplistic.”
Moynihan is wrong to say that I had not considered more than the headline or the limited number of questions in my critique. But even if I hadn’t (and I had), my response was that you can’t write a good analysis if the data are bad from the beginning. Garbage In, Garbage Out.
Further tweets with Moynihan prompt me to write this fuller elaboration of why that poll was misleading, and why Moynihan’s defense of the poll report is misplaced.
Content of ABC/Washington Post Poll
The poll asked just two questions: Whether the polls’ respondents wanted a five percent across-the-board spending cut in overall spending this year, and whether they wanted an eight percent spending cut in U.S. military spending this year. A large majority said yes to the first question, and a large majority said no to the second one.
As I indicated in my blog, that is a biased way to present the questions to the respondents, and can predictably produce these results. Why? Numerous polls over the years have shown that the public will usually support some general statement about balancing the budget or cutting spending, but when it comes to specific programs, the public rarely is willing to cut them.
Evidence for this apparent contradiction in public opinion is highlighted in the Associated Press story last week, which reported on the latest General Social Survey (GSS), conducted by NORC at the University of Chicago:
“As President Barack Obama and lawmakers spar over huge federal deficits, they’re confronted by a classic contradiction: Most Americans want government austerity, a survey shows, but they also want increased spending on a host of popular programs: education, crime fighting, health care, Social Security, the environment and more. Less for defense, space and foreign aid.” (emphasis added)
The survey was conducted in 2012, from March through September, so one might conclude that the results are not relevant. But the series of surveys goes back to 1973, providing a long-term trend of American’s views about cutting and spending. As the survey’s director, Tom Smith, revealed: “Despite a dislike of taxes, more people have always favored increases in spending rather than cuts.” (emphasis added)
This “classic contradiction” is reaffirmed more recently by the two polls I analyzed in my blog: a Pew poll, which showed that among 19 specific programs tested, not one elicited majority support for cutting; and the ABC/WP poll, which found large majority support for the 5 percent across-the-board spending cuts when no specific programs were mentioned.
In a separate question, the ABC/WP poll asked about just one specific program area, and sure enough, the results showed most people opposed to spending cuts.
As the GSS and Pew surveys show, when several specific programs are mentioned, Americans are much less willing to cut a whole host of domestic programs than they are to cut defense spending. But the ABC/WP poll reported exactly the opposite finding.
Moynihan suggests that the article is a “broader” analysis, implying that I overlooked some important caveats. I don’t think so. If there is any doubt about the major message of the article that reports on the poll, one need only take a look at the headline of the article and the first three paragraphs:
Headline: “Most Back Cuts Overall – But Not to the Military.”
The first three paragraphs (I have put in italics the most biased parts of the report):
“For all the dire warnings, most Americans welcome a five percent cut in overall federal spending this year. But the defense budget is another matter.
“The public by nearly 2-1, 61-33 percent, supports cutting the overall budget along the lines of the sequester that took effect last Friday. But by nearly an identical margin, Americans in this ABC News/Washington Post poll oppose an eight percent across-the-board cut in military spending.
“These views come before the $85 billion in cuts this year have taken hold, leaving open the question of how the public will respond once the reductions hit home. Nonetheless, the results suggest that warnings about the nation’s military readiness have resonated, while the public is more skeptical about the damage the sequester poses to federal programs more generally.”
The reason the public seems to favor military spending compared with “overall federal spending” is that ABC/WP mentioned only military spending specifically. Had the poll included almost any other program (except for foreign aid), the poll would no doubt have shown opposition to cuts in those programs as well.
Thus, the overall message of the article is just plain wrong. The numbers are no doubt accurate reflections of the poll results, but the questions were asked in a way that makes any analysis inherently biased.
Later in the article, the author talks about opinion differences among partisan groups, but then reaffirms:
“Strength of sentiment also lands squarely in favor of overall budget cuts, and against those to the military. Strong support for overall cuts outpaces strong opposition by 15 percentage points, while it’s the opposite, by 25 points, when it comes to military spending.” (emphasis added)
No matter how much additional analysis the author does (and there are several additional paragraphs), the article’s message is clear – and “deeply misleading.”
UPDATE: 3/15/2013 10:45 AM EST: Check out the full back-and-forth between Moynihan and Moore prior to this blog post:
Unfair criticism; WaPo/ABC analyses place #s in broader context MT @imediaethics Misleading Polls on Sequestration imediaethics.org/Blog/3803/Two_…
— Patrick Moynihan (@MoynihanPatrick) March 8, 2013
Thx 4 reading @moynihanpatrick.@pollskepticdave says it’s absolutely fair criticism; ABC headline & questions are indeed ‘deeply misleading’
— iMediaEthics (@imediaethics) March 8, 2013
@imediaethics @pollskepticdave The analysis clearly ack. limitations, esp. WaPo. Sorry but parsing headlines w/o full rendering is weak crit
— Patrick Moynihan (@MoynihanPatrick) March 8, 2013
@moynihanpatrick : @pollskepticdave says bad questions don’t produce good analysis. GIGO reigns. [garbage in garbage out]
— iMediaEthics (@imediaethics) March 8, 2013
@imediaethics @pollskepticdave Much can be learned from thoughtful analysis especially when the limits of the study are acknowledged (con’t)
— Patrick Moynihan (@MoynihanPatrick) March 8, 2013
@imediaethics @pollskepticdave Outright dismissal of research w/o considering more than the headline or b/c few q’s were asked is simplistic
— Patrick Moynihan (@MoynihanPatrick) March 8, 2013
@moynihanpatrick F/ @pollskepticdave What’s simplistic is asking 2 q’s in a biased context, producing misleading results.Remember:GIGO (1/2)
— iMediaEthics (@imediaethics) March 9, 2013
@moynihanpatrick Contd. from @pollskepticdave : Another aphorism: You can’t make a silk purse out of a sow’s ear. (2/2)
— iMediaEthics (@imediaethics) March 9, 2013
@imediaethics @pollskepticdave Is your claim the q’s themselves were biased, or that they didn’t cover the range of attitudes in the poll?
— Patrick Moynihan (@MoynihanPatrick) March 9, 2013
@moynihanpatrick Thanks for your tweets.@pollskepticdave has just responded in a blogpost with more on this poll imediaethics.org/Blog/3818/Why_…
— iMediaEthics (@imediaethics) March 14, 2013
For further tweets, check out Moynihan‘s and iMediaEthics‘ Twitter accounts.
