Mirror, Mirror on the Wall

Which conflict forecasting model is the most accurate of them all? This is yet another question posed by the Ivory Towers at the ISA conference vis-a-vis conflict early warning systems. My non-academic answer: who gives a hoot? My more refined answer: they’re all equally bad. My academic answer: has no one read Nassim Taleb’s “The Black Swan“?

Recent empirical studies demonstrate that experts, i.e., us (and our sophisticated systems and methodologies) are only marginally better than novices in our ability to accurately forecast political and economic events. Furthermore, these studies show that neither group’s forecasts are much better than random guessing. Of greater concern still is the empirical observation that experts nevertheless remain consistently overconfident of the accuracy of their own forecasts. This is compared to novices who tend to be more conservative vis-à-vis their forecasting abilities although they are equally (in)effective when it comes to accuracy. A separate study found that “somehow, the analysts’ self-evaluation did not decrease their error margin after their failures to forecast.”

Perhaps the most telling test of how academic methods fare in the real world was run by Spyros Makridakis, “who spent part of his career managing competitions between forecasters who practice a ‘scientific method’ called econometrics […]. Simply put, he made people forecast in real life and then he judged their accuracy” (1). This led to the following lamentable conclusion “statistically sophisticated or complex methods do not provide more accurate forecasts than simpler ones” (2). And so, despite the fact that “billions of dollars have been invested in developing sophisticated data banks and early warnings, we have to note that even the most expensive systems have shown a striking inability to forecast political events,” not to mention galvanize any preventive measures (Rupesinghe 1988).

The “novel” approach presented at the panel is to take several different models/datasets and to compare their forecasts. Where they agree, we know we’re on to something. Where they don’t, this begs to be explained. “Triangulating” or “control mechanism” is the fancy language used to describe this process. But don’t be fooled. This is like asking which of the Emperor’s new clothes is the most majestic of them all? Comparing different models that are incapable of producing accurate forecasts in order to decide which is correct sounds a little problematic to me. Especially when “out of close to a million papers published in politics, finance, and economics, there have only been a small number of checks on the predictive quality of such knowledge” (3).

In any case, this was the approach presented at the panel. The conclusion: all we need is a comparative analysis of conflict early warning forecasts and to write up the results in an early warning report. The challenge, according to the academic, is not accurate forecasting but who exactly we should email our reports to. The underlying assumption is that by us providing forecasts to those in position to act, they will. Hubris. If only it were that simple! Decision-making processes within large institutions like the US and UN are far more complicated! A recent study by Susanna Campbell and myself also showed there was no link between decision-making structures within the UN and input from formal conflict early warning systems. Wake up academia, policy makers don’t use our forecasts, which is probably for the better since these are inaccurate anyway.

What I really don’t understand, however, is how academics can profess to forecast conflicts and at the same time use the word “discontinuous” to describe trends in conflict. If a process experiences tipping points (or punctuated equilibria) then no econometric model however fancy can provide accurate forecasts. Talk to anyone at the Santa Fe Institute (SFI) if you’re not convinced. Or read this piece by Charles Doran on “Why Forecasts Fail” presented at the same ISA conference albeit 10 years ago!

Patrick Philippe Meier