I spent the last major global crisis in the vortex of hell, surrounded by data and models that turned out to be misleading, trying to make sense of an unprecedented worldwide financial catastrophe.
It looked a lot like what’s happening now in the grips of the coronavirus pandemic.
The same kind of model risk that played out during the financial crisis is playing out now. Just as the models the banks used in 2008 proved unreliable in predicting how severe the crisis would be, today’s models are failing to pinpoint with anything near accuracy how fast and how far the coronavirus contagion will spread under various isolating scenarios.
We’re seeing policymakers iterate their response to the virus in the same ad hoc way that we in the financial sector reacted to the global financial collapse. Just as we did, they’re leaning heavily on data for a problem the models can’t comprehend. It takes me right back to lower Manhattan, a decade after I left my position as a big-bank chief risk officer.
In the years leading up to the economic crisis, we had a lot of models — good models, sophisticated models. Our loan-loss models could derive the probability of all sorts of outcomes, from all sorts of scenarios. They were instrumental in helping us assess risk. You probably know how this turned out.
As waves of homeowners began to default at the start of the Great Recession, we used models to try to flatten the curve on our delinquency rates, trying to implement loan modification programs and other mitigation strategies. Nothing we did stopped those waves. None of it played out the way the models predicted.
The models failed, time and time again. They had been trained on data that had never seen anything like the 2008-09 crisis before.
Sound familiar?
Today, the models are forecasting hospitalizations, COVID-19 case numbers and deaths, and those predictions are proving just as skewed, if not more, than the ones we were using at Citigroup. Just a few weeks ago, for instance, Dr. Anthony Fauci and Dr. Deborah Birx, of the White House Coronavirus Task Force, shared results from models predicting U.S. mortality rates from COVID-19. Left unmitigated, models showed over 1.5 million deaths. With a full slate of mitigation measures, it showed between 100,000 and 240,000 deaths. Later models predicted levels below the lower bound of that range.
It is a stark reminder. Models, in times of unprecedented crisis, can provide a sense of comfort — a feeling that we understand the cataclysm. But that’s a false sense of security. In examining the details behind these studies, I see little evidence of their predictive quality. In fact, the enormous confidence interval pointed out by Dr. Birx only underscores the models’ fallibility.
In both crises, the models suffered from the same thing — they weren’t robust enough to predict within the episodic, fat-tail-risk events. They might work exceedingly well for normal, run-of-the-mill types of environments — mortgage lending in finance, for example, or a typical seasonal flu, in health care. But when it comes to these episodic, fat-tail risk types of outcomes – like a pandemic or a financial crisis – those conditions remain outside of the range of what our models can assess accurately.
As Dr. Birx has said, our models are only as good as the data we feed them. And with no comparable events to these crises, the models have little data that’s relevant. Still, as experts we turn to our tools.
Experts need models. It’s one of the things I tell students at the University of Maryland’s Robert H. Smith School of Business. I also tell them this: Models need experts. They’re useless without them. And sometimes, even when you have models and leading experts, like Dr. Birx and Dr. Fauci, there’s still so much you can’t know.
In those times, as in these times, the models are there to offer a guidepost, a single piece of information, that can help leaders decide what kind of public policies to consider and what kind of business decisions to make. In those situations, the best, most-informed decisions are a mix of data, modeling, experience and judgment. And they’re the hardest decisions to make.
In a severe crisis, we all want to find specific, incontrovertible solutions. We want to anchor ourselves to the alchemy that’s associated with our fancy statistical outcomes of normal distribution. But, as humans, we know the world is abnormal.
Clifford Rossi, executive-in-residence and professor of the practice at Maryland Smith. Prior to entering academia, Rossi had nearly 25 years’ experience in banking and government, having held senior executive roles in risk management at several of the largest financial services companies. His most recent position was managing director and chief risk officer for Citigroup’s Consumer Lending Group, where he was responsible for overseeing the risk of a $300+B global portfolio of mortgage, home equity, student loans and auto loans.