Fair political polling is a statistical balancing act

Recent history has demonstrated the risks of polling with a non-representative, unweighted or poorly weighted sample.

In 2014, several pollsters incorrectly forecast outcomes for key elections: the U.S. midterm for the Democrats (Republicans won most of the seats), the Scotland independence referendum (voters rejected it resoundingly), and Israel’s election (Prime Minister Benjamin Netanyahu was the clear winner).

{mosads}Not once, but twice, many pollsters erred in predicting the outcome of Greece’s two bailout referendums, which voters rejected. And this year, several polls failed to foretell that a majority of British voters, including unlikely voters, would support “Brexit” and force Great Britain’s exit from the European Union.

Samples often exclude portions of eligible voters as polling scientists seek to develop representative samples of voters who are expected to cast their ballots. Hard-to-reach respondents are inadvertently underrepresented while those deemed unlikely to vote are excluded by design

Unlikely voters — such as those who have not voted in a previous presidential race — which in some polling models represent up to 20 percent of the election turnout. And recent incidents of miscalculated forecasts have demonstrated the risks of their exclusion.

To account for these problems, surveys are often weighted to match a “gold standard” such as census counts. However, weighting cannot create respondents, so if certain groups are missed, no amount of weighting can ensure that their voices will be heard.  

In launching the USC Dornsife/Los Angeles Times Presidential Election Daybreak Poll, inclusivity and transparency were paramount. We made a series of decisions in designing the national probability poll intended to ensure that the opinions and preferences of all potential voters, including groups sometimes underrepresented in polls — young people, ethnic minorities, low-income earners, people who have not attended college, and those who have not voted in prior elections — could contribute to our results. Their contribution, statistically, is a direct reflection of their own prediction of how likely they are to vote.

This method does not rely on screening a sample for likely voters, and is part of our innovative approach to election polling. The Daybreak poll also is distinct from other polls not just in our inclusivity, but also in our effort to assess voter support for presidential candidates. 

Respondents are asked to rate, on a scale of 0 – 100, their certainty of voting for each of the candidates, rather than saying outright which candidate they support. This “probability” method may quantify voter uncertainty in a way that traditional questions cannot. Our poll also tracks a panel of eligible voters who respond on a certain day each week. The results, which represent a seven-day rolling average, are updated nightly at election.usc.edu. 

Our ability to connect with underrepresented voters nationwide is the result of using a sampling framework that does not exclude anyone a priori, as it is based on a sample of postal addresses. Since we ask our respondents to respond to the survey online, we provided respondents who were not already online with internet-connected tablets.

This group, many of whom are low-income, ethnic minorities, or living in rural locations, represent about 3 to 4 percent of the roughly 3,000 participants in our poll. These individuals are less likely to be asked to provide their opinion during a presidential election, as there are fewer ways they can be reached.

Polling scientists must contend with bias, such as when some individuals in a polling sample are more difficult to reach, or more willing to respond, than others. The Daybreak poll is weighted to account for that problem and ensure that our sample represents the ethnic and socioeconomic diversity of the U.S. population.

To achieve this balance, using a technique known as “raking,” some underrepresented individuals in our overall sample of 3,000 participants are assigned a greater weight than are individuals in overrepresented groups.

Some of the underrepresented groups in our poll who receive fairly high weights include young people aged 18 to 34, who make up only 2 percent of our sample, and represent about 7 percent of our weighted results. African Americans represent 9 percent of our sample before weighting and 13 percent after.

Those who did not vote in 2012 receive some of the highest weights. As a group, they represent about 19 percent of our sample and are weighted to nearly 40 percent. A high proportion of these voters have no college degree, including 30 percent who are white women and 25% who are white men. About 40 percent of these unlikely voters are low income, earning less than $35,000 a year. They tend to be younger, with 35 percent aged 22 to 34.   

Politically, this group is not tightly aligned with the country’s two-party system: about a third of these previous non-voters consider themselves independents, compared to under 20 percent of others. The rest are split roughly evenly with a preference leaning toward Democrats (31 percent) or toward the Republicans (30 percent). They are less likely to label themselves as either liberal or conservative: 36 percent say they are moderates, compared to 21 percent of those who did vote in 2012.  

The higher weights for this group are the result of adjusting to match the turnout in the 2012 presidential election. The total weight assigned to voters who said they voted for Mitt Romney or President Obama in 2012 corresponds exactly with the share of votes that the candidates received: 25 percent supported Republican Mitt Romney and 27 percent backed President Obama. Voters who report that they did not participate in that election are considered among the 40 percent of the voting population that did not turn out and thus, are weighted accordingly.  

When weighting, statisticians must decide whether to limit how much weight any one individual is assigned. This is known as “trimming” the weights.  It makes intuitive sense to ensure that one person’s answers aren’t, for example, weighted to represent the views of 50 people. However, trimming can result in a sample that underrepresents important groups of voters, such as young people, minorities, or rural voters.

After much discussion, we decided not to trim. We have accounted for this with larger-than-usual confidence intervals compared to other polls of similar size, 

Anyone who refers to the Daybreak poll should pay close attention to the confidence intervals. An interval can widen considerably when the opinion of a subgroup of voters shift – a fluctuation which is marked by the widening of the gray-shaded areas of our charts. 

For example, one of our young black voters, who represents a rare group of voters in our poll and is thus highly weighted, can create a visible jump in support for Trump in the Daybreak poll’s African American subgroup chart, and we find that this voter alone can widen the gap between the Trump and Clinton vote by just under 1 percentage point. When this happens, the margin of error widens as well.  

When this blip has occurred, affecting the African American results, the gray area of our poll charts widens, indicating an increased margin of error. This is why we recommend paying attention to the size of error margins and caution against over-interpretation of big shifts within those smaller groups of underrepresented voters.

Our poll has been highlighted as an outlier because it has frequently shown Republican presidential candidate Donald Trump with a lead over Democratic candidate Hillary Clinton and it consistently has lower numbers for Clinton than those of other polls. Some have pointed to our inclusion of less likely voters as the cause of this discrepancy because of the high weight they are assigned and their preference for Trump.

The probability of these unlikely voters casting a ballot may increase as we near Election Day, but as of this writing, about 19 percent have indicated a zero probability of casting a ballot in this year’s presidential election. The average likelihood that this group will turn out has increased only to 58 percent, compared to 93 percent among those who did vote in 2012.

Although they are weighted to represent 40 percent of our sample, our poll’s design ensures that their own low estimated likelihood to vote suppresses their influence on our projected outcome.

We are not promising that the Daybreak poll is more accurate than others. Ours is experimental in its probability methodology and use of underrepresented, unlikely voters. We plan to learn from its successes and flaws, and we hope other scientists will, too, by analyzing our publicly-available data. The science of polling can only improve with research.

Arie Kapteyn is a professor of economics and the executive director of the USC Dornsife College of Letters, Arts and Sciences Center for Economic and Social Research, where he also oversees the Understanding America Study. Jill Darling is the Survey Director of the Understanding America Survey, also at the USC Dornsife’s Center for Economic and Social Research.