“Majority think Trump’s judge comments racist.” That’s the headline about a poll from last week. It’s not false. But it’s not the whole truth, either. To see why, let’s take a closer look at some of the issues with the data.
First, a sizable number of people — 73 percent of the sample surveyed — had either “heard a little” or “heard nothing at all” about the lawsuit against Trump University that had sparked Donald Trump’s comments about the judge overseeing the case. But the opinions of these respondents, a quarter of whom weren’t familiar at all with the controversy, are a part of the headline-grabbing “majority.”
Second, the “majority” was only 51 percent. And the margin of error in the poll was 4 percentage points. Margin of error is one common way to measure statistical uncertainty from polling. It’s a way of answering the question, “How sure are you?” In this case, the pollsters were only “sure” within 4 points — anywhere from 47 percent to 55 percent of people might think the comments are racist. In other words, it’s quite possible a minority — not a majority — held this view.
Moreover, it typically means that if the study were repeated 100 times, 95 times you would get findings within the margin of error — a 95 percent confidence interval, as statisticians would call it. Ninety-five percent of the time is not the same as all the time. Unless you’re Brian Fantana in “Anchorman,” who knows that “60 percent of the time, it works every time.”
Third, the poll focused on differences among Democrats, Republicans and independents. For example, The Huffington Post reported, “Fewer than a quarter of Republicans say the remarks constituted racism.” Is that true?
Start by asking the question, who’s a Republican? Or rather, who isn’t? In the above poll, 40 percent of respondents were classified as independent — that is, neither Republican nor Democrat. But while these independents may truly not identify with either party, it’s likely that many of them will vote for either the Republican or Democratic nominee. Consider a Gallup poll that found, as with YouGov, that a bit more than 40 percent of people identified as independents. But when Gallup asked them which way they leaned, the vast majority of them chose a side. If YouGov had asked their respondents which way they leaned (Democrat or Republican) and then asked their opinion about Trump’s comments, the results likely would have been different.
There was another interesting question asked in the poll that got much less attention: “Do you think a federal judge who is a native born American citizen but has Mexican ancestry can run fair and impartial court proceedings in a class-action lawsuit?” Here, 79 percent of Democrats, but also 69 percent of Republicans, answered “yes.” One might conclude based on this question a much smaller difference by ideology than the headlines suggest. The divide among the party responses is much more pronounced once Trump, the presumptive Republican presidential nominee, is added into the question.
Finally, any time you’re looking at a sample of people, you need to think about whether the sample you’re collecting is representative of the full population. In this case, 254 survey-participating Republicans represent tens of millions of Republicans nationwide. Where did these people come from? Click on the poll data and methodology and scroll to the last page, where you’ll find this statement: “Respondents were selected from YouGov’s opt-in Internet panel using sample matching.”
An opt-in Internet panel is a group of people who agreed to be surveyed. Why would people agree to be surveyed? Maybe because they feel strongly about an issue. Perhaps they’re not busy at work that day and want to waste some time. Or maybe they just like taking online surveys. Are people who take opt-in Internet panels representative of the full voting population? A 48-year-old Republican who is a self-selected participant may not hold the same beliefs as a demographically similar Republican who didn’t elect to participate. The survey weighs people to try to make the sample more “representative” of the broader population, but the biases still exist.
So what should you do with this information?
• Read past the headline — most people don’t. And if you can, read the actual poll results if you want the full story.
• Understand that a poll is typically based on a sampling of people. Who those people are and how they were selected can have a significant impact on the poll results.
• Know the statistical terms used in polling. Margin of error and confidence intervals typically reflect the uncertainty that comes from sampling. Opt-in participants, not to mention self-reported data, have the potential to skew results.
You don’t have to be a trained statistician to understand polls. But having some basic statistical literacy can certainly give you an advantage throughout the election cycle.
Johnson is the CEO of Edgeworth Economics and the author of “Everydata: The misinformation hidden in the little data you consume every day.”