Why the polls weren’t as wrong as you think

Voters who decided to back a presidential candidate at the last minute and an unprecedented, pandemic-induced surge of mail-in and absentee ballots across the country confounded pollsters trying to measure public opinion in the days leading up to November’s elections.

But as the slow vote tallies continue to amass, while pollsters acknowledge that their industry has progress to make in accurately gauging the mood of the electorate, most reject the early narrative that the polls — commissioned by the media, by Democratic groups and by Republican groups — were wrong.

“We just don’t know the answer yet,” said Jane Rayburn, a Democratic pollster. “We’re eager to really get the rest of the data in. I mean we’re still counting votes in some places.”

In analyses and autopsies conducted in the weeks since Election Day, pollsters in both parties have concluded that their surveys largely captured the level of support Democratic candidates led by President-elect Joe Biden achieved — but they underestimated support for President Trump by around 4 percentage points, both nationally and in swing states.

National surveys showed Biden with a clear lead of between 5 and 12 points over the week before Election Day; the average maintained by the polling aggregator FiveThirtyEight pegged Biden’s advantage at just over 8 percentage points. Biden’s final margin, once the last votes are counted in laggards like New York City, will end up being slightly more than 4 percentage points, or about 7 million votes.

Pollsters came close to nailing Biden’s share of the vote in battleground states like Pennsylvania, Michigan, Arizona and Georgia. Those pollsters only overstated Biden’s vote share in Wisconsin and Florida, and even then by only 2 points and 1 point, respectively.

But in virtually every battleground contest, Trump’s share in the final FiveThirtyEight average ran 3 to 5 points below the vote share he attained in November’s elections.

Those who conduct survey research say they will not know for certain whether any particular demographic group was under- or over-represented in their surveys until election results are finalized, and until the Census Bureau puts out a final report on the demographics of this year’s electorate next year. 

All agree that the exit polls used to estimate how various groups voted in the election show even more troubling signs of inaccuracy than did preelection surveys. The discrepancies between different exit polls mean none can be trusted.

One pollster has circulated a memo to clients comparing results of exit polls conducted by The Associated Press and Fox News with those conducted by Edison Research for CNN. The AP’s electorate was older and much whiter than the one measured by CNN’s: In the former, 27 percent of voters were senior citizens and 74 percent were white. In the latter, 22 percent were seniors and just 65 percent were white.

But it is clear that pollsters missed some voters who planned — or finally decided — to vote Republican.

“One of the big existential challenges that we face is predicting who’s going to vote, and projecting out what the electorate’s going to look like is always one of the hardest things any of us as pollsters can do. It’s equal parts empiricism and analysis and feel and emotion. It’s a delicate balancing act,” said Ben Lazarus, a Democratic data expert. “There is some segment of Republicans who probably aren’t answering the phone.”

In some cases, pollsters made incorrect assumptions about the electorate itself. Some pollsters weighted their samples too heavily to Census Bureau figures that skewed more heavily toward young and educated voters. That was especially true in states like Wisconsin, where media pollsters showed Biden running away from Trump while local surveys showed a much tighter race.

“The pollsters who were the most wrong were the ones who put too many controls on their sample frames. If you had an excess of buckets in your frame, all your buckets had a better chance of being wrong,” said Brad Todd, a Republican strategist whose firm conducts polling. “The polls that were the most wrong had too many young people and too many educated people.”

Pollsters attributed some of the discrepancy to late-deciding voters, who typically break against an incumbent in large margins. This time, several said those who struggled to make up their minds were probably more conservative voters who approved of Trump’s policies but disapproved of his comportment in office — voters who might be more likely to come home to their natural party.

“Late deciding voters voted overwhelmingly for Trump,” said Glen Bolger, a prominent Republican pollster.

Others said they discovered vast discrepancies in polling results dependent on the way certain people answered polls. Text-based polling, automated surveys and live interviews all led to startlingly different results.

“Methodology matters,” Rayburn said. “Making sure that we’re reaching people in the right way is really important.”

The underestimated levels of Republican support appeared in critical contests for U.S. Senate seats decided this year. But unlike in the presidential contest, in many of those races, a difference of 4 percentage points was sufficient to flip the result.

Nowhere was that more evident than in North Carolina, where Sen. Thom Tillis (R) pulled off a surprise comeback against the Democratic nominee, Cal Cunningham, who acknowledged a sex scandal in the waning weeks of the race.

In the final week of the contest, Cunningham held a narrow lead in each of the six polls publicly released by media outlets, of between 1 and 4 percentage points. Three of those polls nailed Cunningham’s final vote share, of about 47 percent; two more were within 1 percentage point.

But all six underestimated Tillis’s eventual share, of just under 49 percentage points, by between 3 and 6 points.

In most other states with competitive Senate contests, pollsters correctly predicted the right winner. Sens. Gary Peters (D-Mich.), Joni Ernst (R-Iowa), Lindsey Graham (R-S.C.), Mark Kelly (D-Ariz.) and John Cornyn (R-Texas) all led the final RealClearPolitics polling averages — though in each state, the Republican candidate scored a slightly higher vote share on Election Day than they had in the polls.

If pollsters in any state notched a massive whiff, it was in Maine, where Sen. Susan Collins (R) won reelection after trailing every single public survey released during the course of the race. Collins ended up winning 51 percent of the vote, 8 and 9 points higher than the percentage she scored in the only two polls conducted during the race’s final weeks.

The unprecedented growth of absentee and mail-in ballots contributed in part to rosy Democratic projections, and dismal Republican fears, of a blue wave that never crested. So many people had voted early that some campaigns stopped polling in the final two weeks of the race, convinced that any last-minute strategic shifts would not have the impact they typically would in a race’s final days.

Widespread use of mail-in voting, already common in Western and Mountain West states, is unlikely to go away in future years. Pollsters said they have to consider how to incorporate the number of early voters into their polling universes, and how to use that data to fine-tune a campaign’s get-out-the-vote efforts in a normal year that is not marred by a global pandemic.

The early voters are likely to mean at least one longstanding feature of a typical campaign: the daily tracking polls that can detect even the tiniest shifts in a closing stretch.

“Daily tracking is dead, for all intents and purposes,” Bolger said. “I don’t know about the Democratic side, but on the Republican side nobody wants to do it anymore.”