Once again Gallup has announced its annual “most admired” list of living people, based on a poll of American adults.
And it’s official – Glenn Beck is more admired than the Dalai Lama! Not only that, Beck is equally as admired as Nelson Mandela, Bill Gates, Pope Benedict XVI, Billy Graham and Jimmy Carter!
Or maybe not…
After reading the December 2009 announcement, Washington Post’s Dana Milbank noted the results:
It’s official: Americans admire Glenn Beck more than they admire the pope.
This news, at once unsettling and unsurprising, came from the Gallup polling organization on Wednesday. Beck, the new Fox News host who has said President Obama has a ‘deep-seated hatred for white people’ and alternately likens administration officials to Nazis and Marxists, was also more admired by Americans than Billy Graham and Bill Gates, not to mention Bill Clinton and George H.W. Bush.
But pollsters weren’t convinced. As reported by Politifact.com, Karlyn Bowman, polling expert with the American Enterprise Insitute said that “We have to take these low rankings in the most admired poll with a grain of salt. More popular than the pope? I can’t see that.” And Mark Blumenthal of pollster.com wrote, “If the question posed were, ‘Who do you admire more, the pope or Glenn Beck?’ the results would likely be very different.”
|
See above Gallup’s list of “Most Admired” Men in 2009. (Credit: Gallup) |
As surprising as this might seem, Frank Newport of the Gallup Organization concurred with the critics.
He wrote in Jan. 2010 that Gallup’s rank ordering of most admired does not mean that people lower on the list are less admired than people higher on the list.
Moreover, according to Newport, the different rankings with the same percentages do not really mean the higher rankings are actually “higher” than the lower rankings.
In short, Gallup’s long-running (more than six decades long) list of “Most Admired” men and women does not include the most “admired” men and women after all.
Why the “Most Admired” Are Really Not the “Most Admired”
Here is Newport’s argument last year in an article published on Gallup’s website (see here), countering Milbank’s conclusion that Beck was more admired than the pope.
First, Newport wrote, “there are number of ways to measure admiration.” Gallup used an “open-ended” methodology — allowing respondents to come up with their own answers instead of selecting answers from a pre-chosen group — (see the question in the above chart), because Gallup had always done it that way. “The open-ended method for our Most Admired results is based on a historic Gallup precedent. We first asked the question in this format in 1948.”
Still, he conceded, the open-ended methodology does not lead to the conclusions that Gallup regularly touts:
I don’t think these methodological facts of life allow one to conclude that ‘Americans admire Glenn Beck more than they admire the pope,’ as in Milbank’s column….
Milbank’s phraseology implies that Americans were asked to make a direct comparison between the two men and that Beck won. Or that all Americans rated their admiration for both men and that Beck came out higher. We, in fact, did neither.
If we did ask about each of these two individuals separately, it’s a reasonable hypothesis that the pope would have a higher ‘admired’ factor than Beck. [Emphasis added].
OK! Gallup doesn’t believe its own list.
It’s interesting that Newport refers to “Milbank’s phraseology,” which led the reporter to conclude that people higher on the list are more admired than people lower on the list. But it’s Gallup who claims that it is measuring the “most admired.” And it’s Gallup who lists the people in rank order. All Milbank did was take Gallup at its word.
If it doesn’t measure “admiration,” what does the poll allow one to conclude? Newport’s answer: “The question basically measures ‘top-of-mind’ brand awareness.” (emphasis added)
|
See above Gallup’s list of “Most Admired” Men in 2010. (Credit: Gallup) |
So…this year, Beck’s brand awareness is greater than the Dalai Lama’s, and equally potent as the pope’s. Just as Palin’s brand awareness rivals Oprah’s and is greater than many other female luminaries. But being aware of a brand, and admiring the brand, are two different things.
That’s what Gallup tells us, anyway.
So, why doesn’t Gallup just tell us the truth outright and rename its “most admired” list – calling it instead the “highest brand awareness” list?
Maybe it wouldn’t get the same news coverage?
Other Ways of Measuring “Most Admired”
Could Gallup present a more valid “most admired” list than it does now? Yes, indeed. In fact, Newport specifically mentions two different ways.
The first is to follow the same process that Gallup used when it determined the winners of the People’s Choice Awards. As Newport explains:
When Gallup conducted the research for the People’s Choice Awards a number of years ago, we used a two-stage process. We first asked the open-ended questions ‘What was your favorite movie of the year?’ ‘Who is your favorite movie actress?’ and so forth.
To determine the winners, we then took the top vote getters and called a second sample and read them the list, asking them to choose among them.
We’ve talked about doing the same with our admiration rankings. But to date we have decided to stick with our traditional practice. [emphasis added]
There are some problems with the two-stage approach. In the second stage, pitting the “top vote getters” against each other does not ensure that the favorite will win. We see that in multi-candidate political elections, when the “winner” may get less than a majority of the vote. That’s why typically in such situations there is a run-off among the top two vote getters, so that the winner is supported by a majority.
Still, the two-stage process Newport outlines above is essentially the way the Academy Awards process works (eligible members of the Academy choose among five nominees in the second stage), and it’s a better approximation than a one-stage process based on an open-ended question.
A second approach, a refinement of the above process, would be to keep the first stage of the process to identify the top 10 nominees (using an open-ended question). However, in the follow-up poll, respondents would be asked either to rank order each of the top ten mentioned candidates according to how much they admired the candidates, or to give each of the top ten mentioned candidates a score from 0 to 10, with 0 being no admiration and 10 being the highest level of admiration. The resulting scores would provide a valid rank ordering of those specific candidates.
Even this process is not perfect, because of the way the nominees are chosen (based on the original open-ended question). Still, it’s a much more realistic approach than what Gallup currently does.
As Newport says, Gallup has considered alternative ways of measuring its top ten “most admired,” but – as Editor-in-Chief with the authority to make the change – he has decided “to stick with our traditional practice,” despite his explicit acknowledgement that the traditional practice does not really measure the “most admired.”
Margin of Error Problem
One other technical detail is worth mentioning. Gallup gives different ranks even to candidates with equal percentages. But as Newport explains (discussing last year’s results), those different rankings don’t really mean there are significant differences between candidates ranked higher and those ranked lower.
There is no statistical difference between the frequency with which Beck is named and the frequency with which the pope is named. The percentage of Americans who mention Beck and the pope each rounds to 2%. We do note in our story that we list individuals ‘ranked according to total mentions.’
…Beck is mentioned by 22 people (out of the 1,025 interviewed) as the man they most admired. The pope is mentioned by 21 people. As you might imagine, there is no statistical difference, between getting 22 mentions and 21 mentions (again, even though we do put them in rank order).[emphasis added]
Billy Graham and Bill Gates get 19 and 17 mentions, respectively. They too are within the same statistical range as Beck and the pope.
So, broadly speaking, Beck is a part of a group of men who are mentioned as most admired by roughly the same percentage of Americans. (The top vote getter in the poll, Barack Obama, is mentioned by 305 individuals. George W. Bush, who comes in second, is mentioned by 44 individuals.) We probably could have made this clearer in the story.
Actually, given the maximum margin of error of plus or minus 4 percentage points, it’s likely that there is no statistically significant difference between the second-ranked man (George W. Bush at 4%) and the two men ranked 10th (Bill Clinton and Tiger Woods, each with 1%). (The margin of error for each percentage will be smaller than the maximum margin of error, but typically pollsters announce only the maximum margin of error.)
What the Results Really Show
So, this 2009 “most admired” poll really shows Barack Obama with the highest “brand awareness” and 10 other men with approximately equal “brand awareness,” all lower than Obama’s. Any other rank ordering is simply misleading (i.e., not true – according to Gallup’s own statistical analysis).
What about the 2010 results? Almost the same pattern. Gallup doesn’t give a margin of error for each of the men (Gallup gives a “maximum” margin of error of plus or minus 4 percentage points), but it appears as though Obama again has the “highest brand awareness,” with perhaps no significant differences among the rest of the men (though it’s possible that Bush in second place is significantly higher than the Dalai Lama in 10th place).
|
See above Gallup’s list of “Most Admired” Women in 2010. (Credit: Gallup) |
Among the women, Hillary Clinton’s “brand awareness” is highest, followed by Sarah Palin and Oprah Winfrey in a tie for second, followed by Michelle Obama in third, and seven other women all tied for fourth. (Again, this is an approximation, based on Gallup’s maximum margin of error.)
As we’ve learned from Newport, it’s important to keep in mind that this is a “brand awareness” list, not a “most admired” list.
Implications
This year, Gallup’s announcement makes no reference to the caveats that Newport described last year. The analysis proceeds as though the list of “most admired” really reflects “most admired,” though we know better. And the misleading rank orders are there once again, though the statistics don’t justify them.
OK…this isn’t a matter of life or death. Or even of significant mis-information. After all, who really pays much attention to the names on the list? The annual story can fill a news hole, as it did for USA Today shortly before the New Year, and then was picked up by scores of news organizations thereafter. But the information itself – whether correct or not – is simply disposable entertainment. Right?
Of course.
Still, it’s a bit unsettling when Gallup, one of the country’s top polling organizations acknowledges that:
1) its results don’t mean what its own news articles claim.
2) it gives different ranks to people, even when such ranks are not statistically justified.
3) while there are better ways to identify the most admired, Gallup chooses not to do so.
Gallup needs to serve the public better.
UPDATE: 1/15/2011 5:08 PM EST: Made small copy edits and deleted one sentence.
David W. Moore is Senior Fellow with the Carsey Institute at the University of New Hampshire. He is a former Vice President of the Gallup Organization and was a senior editor with the Gallup Poll for thirteen years. He is author of The Opinion Makers: An Insider Exposes the Truth Behind the Polls (Beacon, 2008; trade paperback edition, 2009). Publishers’ Weekly refers to it as a “succinct and damning critique…Keen and witty throughout.”