Financial market forecasts: How to avoid buying a lemon

As much as you or I like to think of ourselves as forward-looking, we are all backward-looking and we update our perception of the world only gradually. Known as the recency bias, forecasters often give greater weight to very recent events in their forecast and let what could just be random events colour their perception of how the future will evolve.

In benign market conditions we should see institutional inertia among professional forecasters – only updating their view of the world slowly and iteratively, not wanting to appear too far from the pack or consensus. The exception to this only appearing when markets reach a peak or a trough. Then investment banks and commentators, etc, all seem to want to come up with an even more extreme prediction of where prices could go.

Researchers at the European University Viadrina Frankfurt (EUVF) analysed over 20,000 individual forecasts of nine different metal prices over different forecasting horizons between 1995 and 2011. Instead of finding the institutional inertia and forecasting herding that we might expect, they found strong evidence of “anti-herding”.

So why might some forecasters want to stray from the herd? It all comes down to incentives. Understanding them will enable you to more accurately judge forecasts on their merits.

The incentive to herd or stay away from the consensus depends upon the mix of clients (both existing and prospective) of the bank, institution or even sole pundit. Think about who buys forecasts. There are two groups of buyers. The first are those that buy forecasts regularly, perhaps as part of a subscription to a company’s analysis or for free as clients of an investment bank, for example. In the case of commodity price forecasts, examples of frequent buyers of commodity forecasts might include an oil company or a manufacturing company that regularly buys a certain small range of commodities. Given they are long-time consumers, they may have based their decision on how accurate a forecaster was over several forecasting periods.

In contrast to the regular buyer, there are also onetime or irregular consumers of commodity price predictions. This second group of buyer is more likely to be swayed by the commodity forecaster that was most accurate in the past year or so, or has been the most vocal about his or her success. This is rational from the irregular buyer’s point of view in that perhaps movements in the price of copper or another commodity only have a minor impact on their business, or maybe they only need to buy or take a view on a commodity infrequently. Either way, the cost/benefit of monitoring whether the commodity forecast they are buying has been accurate in the longer term is much higher than in the first group of buyers.

If the second group of buyers dominates (the infrequent consumer), forecasters have a strong incentive to differentiate their forecasts from the predictions of others by making extreme (or non-consensus) predictions. Even though an extreme forecast may have a small probability of being accurate, the expected payoff of such a forecast can be high, since the number of other pundits making the same extreme prediction is likely to be small. Should they be successful in their prediction, then the forecaster can capture the attention and the wallets of the infrequent consumer of forecasts.

In contrast, if a forecaster publishes a less extreme forecast, one close to the consensus forecast, then by definition there is a high probability that other forecasters will make similar forecasts. If this is the case then even if a forecaster’s price prediction is spot on, then the impact on his income and reputation will be minimal. The infrequent buyer will ask, “why pay for a forecast from an average forecaster?”

Incentives, being as they are, set up a paradox. For just as commodity prices work on reflexivity, the same could be true of the commodity forecaster. While a single or a string of successful predictions will bolster a forecaster’s reputation, this may result in future forecasts being much less extreme in order to protect their reputation. When a person or a company has their name on a forecast that may also alter the incentives; for example, a commodity research firm with a relatively low profile (perhaps it is just moving into the area) would be rational to make a wild forecast, drawing big attention from the media. In contrast, firms with a strong reputation are likely to make much more conservative forecasts, not wanting to stray too far away from the consensus.

Career concerns can also play a part too in forecasting. Just as at the level of the firm (whether a bank, consultancy or something else), you might think there is the temptation for an analyst to produce a bold prediction. If the analyst makes an “outlier” forecast that turns out to be spot on, this is likely to capture a lot of attention in the financial media, raising the prospect of the analyst being recruited by a rival firm touting a bigger salary and an even bigger bonus. However, set against this is the risk of being fired (or at least having a few rungs taken from under the career progression of a young analyst) for a bad call.

To examine what the age and experience of the forecaster has on the degree of herding, research published in the RAND Journal of Economics examined over 8,000 forecasts by equity analysts between 1983 and 1996. Equity analysts should, in theory produce reliable forecasts of future earnings of the companies that they monitor, which are then used to produce recommendations on what their clients should buy. Equity analysts face their own quandary, having to balance the interests of the buy-side (ie, their clients who prefer accurate forecasts) and those on the sell-side (other parts of the same bank they work for that might value trading commissions and large initial public offerings more than the accuracy of their analysts forecasts). Note that commodity analysts may face their own conflicting internal objectives too. From trading commissions on a commodity-related exchange traded fund, to a bank’s own proprietary trading on commodities and on to gaining profitable consulting business from a highly valued client. There is more than one incentive.

What the researchers found is that younger analysts tend to herd more than their more experienced colleagues do. Less experienced analysts, meanwhile, are more heavily punished for getting their forecasts wrong and so they have every incentive to stick with the herd. In contrast, older analysts, who have presumably built up their reputations, face less risk of termination. The researchers also found that, contrary to expectations, making bold and accurate predictions does not significantly improve a young analyst’s career prospects.

There is another factor to consider when thinking about forecasts. Again it comes down to the incentives of the forecaster, but this time the inference is more insidious. The nature of forecasting may drive out those that are best equipped to produce them. Commodity price forecasts might just be a market for lemons.

This reference to lemons comes from economist George Akerlof, who published a paper in 1970 in the Quarterly Journal of Economics called “The Market for Lemons”. Within it was a simple and revolutionary idea in which he noted that markets in which buyers possess imperfect information while sellers possess a profit motive are thin, insubstantial and low quality.

Akerlof used the example of the used-car market. Suppose buyers in the used-car market value good cars – referred to as “peaches” – at $20,000, while sellers value them slightly less. A malfunctioning used car – a “lemon” – is worth only $10,000 to buyers (and, again, assume a bit less to sellers). If buyers can tell “lemons” and “peaches” apart, trade in both will flourish. In reality, buyers might struggle to tell the difference: scratches can be touched up, engine problems left undisclosed, even odometers tampered with.

To account for the risk that a car is a lemon, therefore, buyers cut their offers. They might be willing to pay, say, $15,000 for a car they perceive as having an even chance of being a “lemon” or a “peach”. But dealers who know for sure they have a “peach” will reject such an offer. As a result, the buyers face “adverse selection”: the only sellers who will be prepared to accept $15,000 will be those who know they are offloading a “lemon”.

Smart buyers can foresee this problem. With the knowledge that they will only ever be sold a “lemon”, they offer only $10,000. Sellers of “lemons” end up with the same price as they would have done were there no ambiguity. However, the “peaches” stay in the garage. This is a tragedy: there are buyers who would happily pay the asking price for a “peach”, if only they could be sure of the car’s quality. This “information asymmetry” between buyers and sellers kills the market.

In the same way as the used-car market example, one could argue that bad forecasters drive out good forecasters. The deep uncertainty that forecasting fosters may create incentives that perversely degrade the ability to offer better predictions. Many bright individuals might be deterred from working in the sector because from a career perspective the likelihood of error is so high. You could argue that there is little incentive to contribute when the exercise is seen as a dubious one. This then creates the space for people who are less afraid of such reputational costs, which in the end only results in a less critical debate.

There is also little incentive for forecasters to improve on their predictions. Rather, the incentives are geared towards exaggerating the precision of forecasts – a form of signaling in economics parlance. Just as with second-hand car dealers, the job market and many other markets, there is an asymmetry of information. In order to correct for this, dealers, job seekers and perhaps forecasters may try to signal their trustworthiness and talents by collecting awards that convey some authority – the best second-hand car dealer in the North West, for example, or the best gold price forecaster of the last quarter. Such exaggerations satiate the cognitive preferences of governments and corporations, and generate greater media attention to the forecast itself.
For much of the private sector, public forecasts are designed to maximise marketing rather than predictive accuracy. As Philip Tetlock concluded:

…the demand for accurate predictions is insatiable. Reliable suppliers are few and far between. And this gap between demand and supply creates opportunities for unscrupulous suppliers to fill the void by gulling desperate customers into thinking they are getting something no one else knows how to provide.

For every seer, there’s a sucker. Just always think about the seers incentives before following the herd.

Find out more in “Crude Forecasts: Predictions, Pundits & Profits in the Commodity Casino”.

Related article: Just how accurate are oil price predictions?

Unlock commodity market insight now and subscribe to our email updates or

Orange juice prices: The top 10 most important drivers

1) Hurricanes

Florida has been in the eye of many storms that have hit the US Gulf. In the worst case trees are uprooted and orange groves flooded, while at best nearly ripened fruit is blown off the trees. Although hurricane forecasting has come along way over the past few decades it is still very difficult to pinpoint exactly where a storm will make landfill. As such traders in orange juice futures add a premium to prices as the hurricane season commences and especially if it looks like a storm is heading towards the state. The best month for orange juice futures prices tends to be November, right at the end of the season, as uncertainty over the extent of the damage still remains unclear.

2) Frost

Damage to orange trees occurs if the temperature drops below freezing and stays there for more than four hours. Market participants realise of course that severe frost is more likely during the winter and so the price of orange juice futures going into the end of the year should be high enough to reflect the probability of a freeze during the coming season.

3) Disease

A disease called “citrus greening” has severely hampered both Brazil’s and Florida’s orange juice industry in recent years. Greening starts at the leaves and works its way through the tree like a hardening of the arteries, blocking nutrients and water. Oranges then drop off the tree unripe and unusable. Although a genetically modified version is being developed it is likely to be many years away from being introduced.  Greening — which also hurts grapefruit, limes, lemons and other citrus — has cut Florida’s output in half over the past decade, according to the U.S. Department of Agriculture.

4) Drought

Dry weather in the main orange juice producing regions of Brazil may also cause the supply of orange juice to decline. If drought occurs in this region it typically also affects other related ‘soft’ commodity prices like sugar and coffee.

5) The Brazilian Real

The largest global supplier of orange juice is Brazil (producing over 80% of the oranges for processing). When exports of Brazilian OJ are high this can cause the price of orange juice futures to fall. As such currency movements can have a disproportionate impact on how big exports of OJ are. An appreciation of the Brazilian currency, the Real, against the US dollar is likely to be positive for the price of orange juice futures since it is now more expensive to import Brazilian orange juice into the US than it was before.

6) Planting

Oranges grow on trees that require 5-15 years to mature. And so any decision by the farmer on planting does not involves long lead times, by which time the fundamentals may be very different.  This means that farmers price expectations (i.e. whether they expect high or low prices to continue sometimes 3-5 years in the future) are vitally important in determining future supply and prices

7) Inventories

Although the commodity is frozen and not very perishable, only a small amount of inventory is carried over from one year to the next. This lack of a ‘buffer’ helps to contribute to very volatile prices.

8) Consumer tastes & health concerns

Orange juice demand is being squeezed because of a shift away from sugar laden drinks and competition from other products such as flavoured water. The drop in demand can be traced back to the early 2000’s as the popularity of low carbohydrate and high protein diets surged. Changes in eating patterns has also hit demand. With fewer people eating breakfast as a sit down meal fewer families are drinking it in the morning.

9) Substitutes

The relative price of substitute products like apple juice, tomato juice and other soft drinks will influence the demand for orange juice. Meanwhile, farmers in Brazil have been tearing up orange groves and replacing them with more profitable crops such as sugar cane. As Brazilian supply becomes more important to the overall orange juice market, it is starting to become more closely related to the sugar market, which Brazil is also the major global supplier.

10) Shipping, production and logistics costs

Transportation bottlenecks when exporting orange juice out of Brazil can delay product reaching the market resulting in higher prices. Meanwhile, higher production costs (e.g. higher fuel costs) as with all farmers either result in lower margins or need to be passed onto the end consumer.

Silver prices: The top 10 most important drivers

Natural gas prices: The top 10 most important drivers

Copper prices: The top 10 most important drivers

Livestock prices: The top 10 most important drivers

Sugar prices: The top 10 most important drivers

Cocoa prices: The top 10 most important drivers

Unlock commodity market insight now and subscribe to our email updates or

Geopolitics and technology: An unholy union of uncertainty?

“In economics things take longer to happen than you think they will, and then they happen faster than you thought they could.”

Rudiger Dornbusch

As 2017 draws to a close geopolitics and technological innovation appear (at least on the surface) to be the main drivers of greed and fear in commodity and other financial markets. The problem for investors is that the risk underpinning both geopolitics and technological development cannot be priced up neatly.

A fragile peace?

The recent arrest of more than 200 princes, businessmen and other high ranking officials in Saudi Arabia would have been unthinkable only a month ago. Now, the inherent fragility of the Saudi political and economic system may mean that the country now implodes; or indeed the cracks may by papered over for years into the future. But how do you put a price on that risk? Just because things go quiet now does not mean that the risk has gone away.

Ruptures in the price trend for many commodities are often the result of geopolitical developments. Political scientists Ian Bremmer and Preston Keat defined geopolitics as: “The study of how geography, politics, strategy, and history combine to generate the rise and fall of great powers and wars among states.” Given its importance to the running of the modern global economy, nowhere is this more vividly observed than in the battle for energy resources and, in particular, oil.

If geopolitics plays such a big role in seismic moves in the oil market and many other commodities, then if geopolitical shifts can be forecasted with any accuracy this must give forecasters an edge, right? The Good Judgment Project, set up by Phillip Tetlock, set out to answer the first part of this question. They explored the profile of the best out of hundreds of forecasters who made over 150,000 predictions on roughly 200 events during a two year period. Forecasters were asked a multitude of questions, such as: Will the United Nations General Assembly recognise a Palestinian state by September 30th 2011? Will Bashar al-Assad remain president of Syria through to January 31st 2012? The researchers found that forecasters can be good at spotting changes, but only over long timescales.

The problem with geopolitical events is that they tend to be binary outcomes (although clearly not always). They either happen in the future or they don’t. This contrasts with what we might term “market” or “economic” risks which are more dynamic. There are three main problems with binary outcomes: first, they offer little information advantage for investors to play with; second, they are hard to predict and, third, they offer few easily identifiable markets that might benefit from a particular outcome.

Even if you have fantastic foresight about how a geopolitical event is likely to develop, the next problem is decoding what the impact is likely to be on a range of different markets. All too often pundits focus on the immediate effect; for example, based on whichever candidate wins an election. However, they forget to draw the dots as to how the “narrative” could change once the geopolitical uncertainty of the political event falls away.

Even if you could correctly forecast that the regime of a particular oil producing nation would be toppled within a given year, you wouldn’t be able to know the exact path that oil prices would go as a result. You could at least add a risk premium to your forecast, but even that might not be correct. It is after all the risk of a sharp spike in prices that gets people’s attention.

Batteries now included?

Take technology. How do you price up the likelihood of further innovations in shale extraction technology both in the US and elsewhere in the world? Or how do you price up the likelihood that the recently launched Tesla truck will usurp conventional combustion engine powered vehicles and so put a dent in transport demand for oil?

Implicit in any forecast of commodity prices is an assumption of how technology could evolve and how its adoption will affect commodity prices. Commodity prices provide the incentive for new technology, yet also influence commodity production and consumption. Innovations, once introduced, may lead to higher yields from agriculture, more oil being extracted from offshore wells and deeper mines to extract more metals and minerals – all of which could eventually lead to rising commodity supplies.

High commodity prices may also lead to innovation on the demand side too. High energy prices, for example, may discourage consumers from using a particular energy inefficient product. This acts as an incentive for companies to redesign their products to become more energy efficient and less resource intensive. However, just because a technology might appear to be negative for demand doesn’t mean it has to be bad for prices, at least not in the short to medium term. For example, if oil producers are worried about the growth in electric vehicles they may decide to postpone large scale, multi-decade, multi-billion dollar investments. If they get it wrong and electric vehicles don’t take off as fast as they expect, then oil prices may rise sharply if there isn’t enough supply to meet demand.

And remember, don’t forget about rebound effects. If an innovation results in an energy intensive product (transportation for example) becoming cheaper or more accessible consumers are likely to want to consume more of it. Every improvement in technology has a rebound effect.

It’s the uncertainty over how current technology can be utilised and how technology could evolve that makes forecasting so difficult. Technological developments of all sorts involve a large dose of serendipity. The philosopher Karl Popper perhaps best describes the struggle to anticipate future innovations: “The course of human history is strongly influenced by the growth of human knowledge.” Popper also wrote:

But it’s impossible to “predict”, by rational or scientific methods, the future growth of our scientific knowledge because doing so would require us to know that future knowledge and, if we did, it would be present knowledge, not future knowledge.

Yet to forecast the price of oil, lead or cobalt into the next decade we need to make some assumption about how technology will make it easier to extract these commodities and how technology will change the demand for these commodities. Note that no one predicted the invention of pig iron or imagined how it would affect the nickel market, neither did anyone anticipate the introduction of hydraulic fracking and how it would turn the market for oil on its head.

Just noise?

Yet, a question is whether it has it always been such? Times are always uncertain and is now really any different? As can be heard almost any time someone frets about the future, there is the issue of hindsight bias. Someone says, “Things are uncertain, not like it was in the past.” The first part of that statement is accurate; the second is not. The future is always uncertain, whether it is the future we face right now or the future that people faced a century ago.

The answer for investors then may be to unplug from the noise and just ask themselves two questions: first, where are we in the cycle?; and second, what lessons from history can we draw for what might come next?

Related article: Narrative economics

Related article: Batteries now included: You’ll meet a bad fate if you extrapolate

Related article: What lessons does rhodium have for commodity investors?

Related article: Where do we stand in the commodity cycle?

Unlock commodity market insight now and subscribe to our email updates or