Mellman: The fault is not in their models

I was a modeler before it was cool.

Our firm brought modeling to Democratic poll data in the early 1980s, created the first micro-targeting models in the middle and late 1980s, developed the first state-level Senate forecasting model for use in resource allocation in the 1990s and built state-level presidential models and used them with Monte Carlo simulations in the early 2000s. 

{mosads}I offer that history not as an advertisement but to establish my bona fides. I am not a technological Luddite.

Nonetheless, models are, by definition, simplifications of reality. The better the model, the more reality it captures, but few, particularly in the messier social sciences, encompass the whole of reality.

The proliferating Senate forecasting models teach valuable lessons. More trivially they tell us what we already know — voting habits, incumbency, candidate quality, fundraising and poll numbers all matter. More interestingly, they tell us how, on average, to weight those factors in developing a forecast.

That phrase — on average — signals both a methodological caveat and an indication of interest. It represents one of the differences between academics and practitioners. Academics are primarily interested in what happens in the main, on the average. Practitioners focus on what makes a difference at the margin, what creates the exceptions. 

Precisely because they focus on uncovering average historical patterns, there are some factors forecasting models cannot and do not consider. Among them:

Great candidates: Models can distinguish “higher quality” candidates by virtue of previous offices they occupied or the money they raise, but they simply cannot measure the spectacular personal appeal some candidates exude. North Dakota, a deep red state, voted for Mitt Romney in the 2012 presidential race by a 20-point margin. Analyzing all the variables, 538 gave Heidi Heitkamp of North Dakota an 8 percent chance of prevailing in the Senate. She won in large measure because she is an extraordinary woman with a unique personal story. Whether at national fundraisers or in local living rooms, those who met her exclaimed that they had never heard a politician like her before. In fact, she didn’t seem like a politician at all. By definition, none of that uniqueness could be captured in a model — but it’s what enabled her to beat the odds.

Bad polls: Forecasts of Heitkamp’s defeat also fell prey to bad polls. Though our own surveys showed her leading throughout the cycle, the three public polls proved far less accurate: Her deficit ranged from 2 points to 10. Aggregators hope to make up for the faults of individual polls by blending together a number larger than three. 

It doesn’t always help. In 2010, almost every one of the myriad public polls showed Senate Majority Leader Harry Reid (D-Nev.) behind. Not surprisingly, the modelers at 538 pegged his chances of winning at just 17 percent. Our own polling showed Reid ahead every step of the way and proved to be the only accurate polls in the state. Illinois Gov. Pat Quinn was similarly, and wrongly, dismissed as a result of a series of faulty polls. 

Big mistakes: I’m not talking about gaffes that consume vast amounts of press attention but rarely exert real electoral impact. Rather, I am talking about game-changing errors. Arguably two Democratic senators elected in 2010 owe their seats importantly to the horrific ways their opponents’ discussed the violent crime of rape. Other mistakes are strategic. In the very first race I polled, we narrowly defeated a House incumbent because he never realized we were coming and failed to deal with his vulnerability. We’ve lost races too, because of strategic errors. Once we attacked our opponent on a failing our candidate shared but about which we didn’t know. Eventually these big mistakes usually show up in polls, changing the forecasts, but no one can predict them.

Forecasts are useful best estimates based on history, but they don’t foretell the whole story.

Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the majority leader of the Senate and the Democratic whip in the House.