Why do the experts keep getting it wrong on elections?

“EVERYBODY WAS WRONG!”

That was the typically provocative 40-point headline on the front page of the New York Post the day after the U.S. election. The Post went on to amplify that “everybody” included pollsters, politicians, press and, especially, the “blunder pundits who blew it.”

{mosads}The failure to predict Donald Trump’s election victory has occasioned an existential crisis of confidence in so-called experts — or even the concept of expertise. One commentator wrote: “Almost all the polls and experts were wrong”; another that “the predicting experts got it wrong again.”

And it wasn’t just political experts and pollsters who got it wrong. The financial journalist James B. Stewart noted that “to the long list of pundits who called the election wrong — as well as its likely consequences — let’s add Wall Street analysts.”

Most of them had predicted a Hillary Clinton victory and a modest stock market rally, while viewing a Trump win as a “black swan” event that was barely possible and could lead to a catastrophic stock market collapse. And yet in the days since the election outcome, the market has reached record highs.

It would be reasonable to conclude after a breakdown like this that maybe fewer people should be referred to as “experts.” And yet, experts are nothing if not resilient. The same people who got it so wrong on the election outcome are now telling us what to expect from the next four years – seemingly without shame or irony. And of course, we listen to them and take in their advice, because they’re, well, experts.

But this can and should be a moment to reflect upon this failure of expertise — and to ask the bigger question: why do the experts keep getting it wrong? Because we should be clear that this failure of insight and foresight is not an unusual occurrence. As Dave Weigel of the New York Times writes: “Apart from the 2015 UK election, Brexit, the Cubs winning the World Series and Trump’s election victory, it’s been a great year for experts.”

This is not just not a persistent feature of forecasting about politics and sports.  It is consistent with the long-running trend of experts making fundamental errors of prediction and judgment.

This reflects the increasingly pervasive and questionable obsession with specialist, technical expertise. More and more we have become a world sold on depth — wedded to the concept of the ultra-specialist in business, government, medicine, education, and most other walks of life. But there are flaws in the reliance on narrowly defined expertise as an organizing concept.

First, it is becoming increasingly apparent that experts are not very good at predicting the future. As the author Dan Gardner puts it, “expert predictions are next to worthless, and you can do better.” This contention has been reinforced by the research of University of Pennsylvania professor Philip Tetlock, which shows that non-experts typically make better predictions than subject matter experts.

He explains how the dominant traits of many experts (over-confidence, binary thinking, a herding instinct and, frankly, pure hubris) diminish their ability to predict accurately in a volatile, uncertain, complex, and ambiguous era like this one.

As we gain knowledge about a domain, we become prisoners of our own stereotypes — more entrenched in a specific way of seeing the world around us and less adaptable to changing conditions and expectations. Non-experts, in contrast, draw from a more eclectic array of traditions and accept ambiguity and contradictions as inevitable features of life.

Second, experts are prone to living their lives and practicing their crafts in narrowly defined cocoons. These act as echo chambers in which everybody coalesces around a narrow range of possible outcomes. If the presidential election has revealed anything, it’s how easy it is for specialist experts to become isolated from those living lives different from their own.

It is this disconnect which is causing such an erosion of trust in elites and the institutions they control. Over-reliance upon singular expertise makes it less likely that we will successfully tackle contemporary challenges like income inequality, climate change, healthcare, education, and policing.

These are complex, multi-dimensional problems that cannot be solved by specialist experts swimming in their own lanes. To meet challenges like these, specialist expertise is often necessary, but not sufficient.

If nothing else, the recent election reminds us of the need to live our lives more as a mosaic — extending our personal and professional networks so we are exposed to people in all manner of situations, not just those who look and think like us. 

This will hopefully translate into a better job of predicting outcomes, including a presidential election.

Lovegrove, the U.S. Managing Partner at the Brunswick Group, is the author of THE MOSAIC PRINCIPLE: The Six Dimensions of a Remarkable Life and Career, which was published this month by Public Affairs.