All of the published polls released the week before the Feb. 1 Iowa Republican caucus showed Donald Trump leading the Republican field. The fact that Sen. Ted Cruz (R-Texas) ultimately won the race would seem to make an open-and-shut case that the polls were wrong.
But before we hand down any guilty verdicts, let’s look at the evidence.
The average of the six polls taken between Jan. 25 and 31 had Trump ahead with 28 percent, followed by Cruz with 24 percent and Sen. Marco Rubio (R-Fla.) with 17 percent. The actual results on Monday night were: Cruz, 28 percent (4 percentage points higher than his polling average); Trump, 24 percent (4 points lower than his polling average); and Rubio, 23 percent (6 points higher than his polling average).
Clearly, the polling missed something.
First off, the polls likely missed late-voting decisions. Iowa’s entrance poll found that 35 percent of GOP caucus-goers made their final choices in the last days of the race, after many of the survey interviews had already been completed. Most of these late-deciders voted for Rubio and Cruz — the two candidates who outperformed the polls.
{mosads}The full effect of Cruz’s final get-out-the-vote activity, and the limits of Trump’s ground organization, also seem to have escaped most of the polling samples. For example, Cruz beat Trump by 12 percentage points among evangelicals, who made up a bigger portion of the GOP electorate on Monday than they had in 2012 or 2008. But most of the 2016 polls underestimated the size of the evangelical constituency. The final Des Moines Register poll, posting a 5-point Trump lead, pegged evangelicals at 47 percent of the total electorate — much lower than the actual 64 percent on caucus night.
It also needs to be noted that the last public poll conducted before the caucuses did, in fact, show signs of shifts materializing. The Emerson College poll, conducted right after The Des Moines Register survey, found that Trump’s lead had declined from 10 points in mid-January to a single point at the end of month. It also caught the Rubio surge, showing him at 22 percent, up 8 points from two weeks before.
On the Democratic side, polling leads oddly bounced around during the weeks before the caucus, which was hard to explain then and now. Margins swung from 29 points for Clinton to 8 points for Sanders in various polls. Nonetheless, discounting outliers, most of the polls generally showed a competitive contest between Clinton and Sanders, with margins ranging from 1 to 8 points. Of the five surveys conducted between Jan. 25 and Jan. 31, four had Clinton ahead and one had Sanders on top, giving Clinton an average lead of 5 points. On primary night, as we now know, her lead dwindled to a fraction of a point.
While both major candidates on the Democratic side had big get-out-the–vote operations, it’s difficult to determine whether either had a late-breaking tactical advantage. One thing we do know is that 44 percent of Democratic caucus-goers were first-timers, and Sanders carried them by 22 points. The heating up of the Clinton email issue, just days before the voting, may have played a role in the race’s tightening, although there is scant evidence to back up that hypothesis.
When dissecting polling accuracy, it’s essential to remember three things.
First, polls are only “pictures in time.” They tell us where voters were at the time they were interviewed, not where they will be at some point in the future. The effects of events that happen after a poll is conducted — such as debates, ads, gaffes, news stories, endorsements and get-out-the-vote efforts — aren’t measured, even if they occur the day after the poll is completed. That’s why poll results that differ from election returns aren’t necessarily wrong when there is a time lag.
Second, the impact of Election Day get-out-the-vote efforts is always hard to measure in a poll taken several weeks or even just a few days before an election. As we’ve seen, that was clearly an issue in the recent Iowa GOP caucus.
Third, any poll — even if it’s carefully administered using the best techniques — is subject to sampling error. “Sampling error,” as defined by the Roper Center, “is the calculated statistical imprecision due to interviewing a random sample instead of the entire population.” With sample sizes ranging from 298 to 919, Iowa polls taken the week before the caucuses had scientific margins of error between plus or minus 3 percent to 6 percent. This, too, may have accounted for otherwise mysterious variances.
Faucheux is president of Clarus Research Group, a nonpartisan polling firm based in Washington. He has taught courses in politics at George Washington University and is the former editor of Campaigns & Elections magazine. Author or editor of seven books, including “Running for Office,” he also publishes Lunchtime Politics, a daily newsletter on polls. None of the polls used in this analysis were conducted by Faucheux or his firm.