Those who have followed this blog long enough will know that I have often been critical of those that claim to be able to forecast commodity prices. My concern was initially spiked after it appeared that many investors, companies and governments had been fooled into believing that commodity prices would continue infinitum. That sparked me to write my second book, Crude Forecasts: Predictions, Pundits & Profits in the Commodity Casino which set out to provide the evidence, lay out the incentives and offer some ways that things could be improved for the better.
I’m certainly not the only one who is skeptical of the abilities and claims of Wall St forecasters. In a recent piece for Bloomberg, Barry Ritholtz of Ritholtz Wealth Management offers 5 suggestions as to how the forecasting business could be made a whole lot more transparent and potentially more successful.
No. 1. Share the underlying model’s past performance: As Ritholtz highlights, “if the forecaster has an audited track record showing how the prognostications stacked up versus reality during the past five years, and can demonstrate how these made clients some money, that might be worth notice.”
But he rightly caveats this by saying that past performance is no guarantee of future results since “a good track record may not be repeatable; that those winning outcomes could have been the result of luck or that specific era or some other random element.”
In my book I analysed the forecasting ability of investment banks and other institutions over a 10 year period to 2016, which of course covered commodity booms, busts and financial crises. But that was only one cycle, and just one commodity. It tells you nothing about whether that institution was just lucky, or unlucky. It may of course just be survivorship bias.
No. 2. Acknowledge the unknown variables: The article highlights the importance, and the rarity of the caveat, “Not locking oneself into any single outcome because things might change is simply common sense. Unfortunately, that is a rare characteristic in too many forecasters.”
The financial media being what it is (although there are welcome exceptions such as Real Vision) rewards the confident, no doubt about it, banging on the table kind of forecast. It just doesn’t have time (nor the appetite) for a forecaster saying, “on the one hand this or that could occur.”
No. 3. Acknowledge inherent biases: According to Bloomberg Business Week, economic forecasters are “more likely to miss recessions than to predict ones that never occur.” Ritholtz rightly highlights that there are other incentives at work, “it wasn’t because the economists were necessarily bad at economics, but rather, because of basic game theory. Career risk for being wrong is very real.”
To examine what the age and experience of the forecaster has on the degree of herding, research published in the RAND Journal of Economics examined over 8,000 forecasts by equity analysts between 1983 and 1996. They found that younger analysts tend to herd more than their more experienced colleagues do. Less experienced analysts, meanwhile, are more heavily punished for getting their forecasts wrong and so they have every incentive to stick with the herd. In contrast, older analysts, who have presumably built up their reputations, face less risk of termination. The researchers also found that, contrary to expectations, making bold and accurate predictions does not significantly improve a young analyst’s career prospects.
No. 4. Use errors to make better forecasts: Accurate predictions are held up as evidence of forecasting prowess, and a great marketing platform to sell clients something else. Poor forecasts are quietly swept under the carpet, where, hopefully no one will notice. But we can all learn from failure, and so maybe we shouldn’t be so quick to sweep the evidence away.
As Ritholtz explains, “All models do is take a series of data inputs, sprinkle a little fairy dust on them, and then generate an output. But even if the model does OK, how has the forecaster used its output? Can they make money for clients with it? Alternatively, do they anchor themselves to these predictions, regardless of subsequent data? There is a specific skill to adjusting to errors and failures in order to improve. It is a skill used too little by economists and financial analysts.”
As I explain in Crude Forecasts, “Cognitive dissonance is a psychological phenomenon that refers to the discomfort felt at a discrepancy between what you already believe to be true and new information that presents itself. This is an especially big risk for forecasters of all types, but essentially means that the forecaster will either discount new information that conflicts with the stated forecast or attempt to reframe the evidence to validate the success of the forecast.”
5. Learn from the pros: Finally, the article highlights the work that Philip Tetlock has done on why so many forecasts fail, “His 2006 book “Expert Political Judgment” studied thousands of forecasts, and came to the conclusion that people simply are not good at making predictions about much of anything. With one caveat: Buried within Tetlock’s huge dataset of failed forecasts was a surprising subset of superforecasters. Those in this group stood out for their ability to make more accurate predictions than others.”
According to Tetlock pundits typically fall into categories – “foxes” and “hedgehogs”. Foxes pursue many different ends, often unrelated and even contradictory; they entertain ideas using divergent thinking (ie, looking at many possible outcomes), rather than convergent thinking, and they also don’t seek to fit these ideas into, or exclude them from, any one all-embracing inner vision.
However, many of the pundits courting the financial limelight are hedgehogs. You can easily spot a hedgehog – they are characterised by an attitude of relating everything back to a single vision, and they over simplify and come across as much more confident in their outlook on the world in order to produce a compelling narrative.
Overall, I sense that the process employed by forecasters has improved over recent years. Nevertheless, as the economic forecasting world debates the possibility of a global recession over the next few years these questions are unlikely to stray too far.
