Information or manipulation? High quality data is crucial to commodity markets

According to Jan Hatzius, Goldman Sachs’ chief economist, forecasters face three fundamental challenges: first, it is very hard to determine cause and effect; second, that the economy is always changing and, third, that the data that they have to work with is pretty bad. This article looks at the last of those challenges in more detail.

Basic statistics and forecasts about oil reserves, production, consumption and stocks ought to be a matter of routine. You stick a gauge at the end of a pipe and measure the amount of liquid flowing through, right? However, it’s not that simple, and the problem isn’t limited to just oil, but to all commodities.

Doubts about the reliability of energy statistics were a major part of the “energy crisis” that erupted during the 1970s. As late as 1968, the US reportedly had four million barrels per day of spare production capacity. Meanwhile, thousands of wells across Texas and Louisiana were being operated for fewer than ten days per month. But by March 1972, spare capacity had dropped to zero; every well was at maximum production, domestic output was falling and politicians spoke of an energy crisis.

The oil embargo, announced in October 1973, intensified the sense that something had gone badly wrong, leaving the US unprepared. Politicians and the media blamed a conspiracy between domestic producers and OPEC for engineering the crisis to drive up prices and profits. Congress held hearings amid a sense the statistics and forecasts prepared by oil and gas producers and the US Department of the Interior had been inaccurate or manipulated. One outcome of the crisis was the creation of a new US Department of Energy, and within it a new Energy Information Administration (EIA), in 1977, to produce more accurate and independent data. Another was the creation of the International Energy Agency (IEA) in 1974, to gather better statistics and bring greater transparency to the international energy markets.

Improvements in data collection and forecasting in the US, led by the EIA, have by and large quelled controversy about domestic US oil production, consumption and stocks; but that doesn’t mean they are free from error or revision. According to a study by the Wall Street Journal (WSJ), annual estimates of global crude demand by the IEA in the seven years until 2016 were underestimated by an average of 880,000 barrels per day. And there is little evidence that the demand forecasts from others are any more accurate. The EIA also underestimated global demand – by an average of 2.3 million barrels a day.

Demand is much harder to estimate than supply. Unlike supply, which can be estimated from the pre-announced expansion plans of a relatively small number of companies, estimating demand involves billions of consumers worldwide and many millions of companies of all sizes.

Revisions to oil supply estimates are typically much smaller than for demand, and are often about correcting overestimates for crude production. The IEA’s supply data has been revised down 60,000 barrels a day on average over the seven years to 2016, according to the WSJ’s analysis. That means the oversupply usually ends up being smaller than initially thought. The history of data discrepancies underscores how oil markets often trade on incomplete data.

The information collected in many other parts of the world remains much less comprehensive and accurate. Two major sources of uncertainty are the deliberate secrecy of the major oil producing countries and poor data collection in emerging markets.

In late 2016, OPEC producer Iraq published an unprecedented level of detail about its oil producing activities. Instead of providing just one figure for output and one figure for how much it was exporting, the Iraqi authorities released detailed data about the crude oil output at each of its 26 oilfields and detailed export figures. This “transparency” was a calculated move to prove to outside observers, and other OPEC members in particular, that secondary source estimates of its oil production were way too high. In doing so, it was also making the case that Iraq should not be subject to output cuts.

The opaqueness of oil data from OPEC producers is nothing new. As with any oligopolistic organisation, there is an incentive for individual OPEC members to misinform. This can take the form of under-reporting the amount they are producing and the size of their reserves, but it can also result in outright disregard for pre-agreed production cuts – the higher the price of oil, the greater the incentive for an individual member to break the agreement. Remember, each OPEC member is a sovereign country, meaning that they are not legally obliged to commit to or honour any agreement.

The oil market is by far the largest, most liquid and most important commodity market in the world. If such big revisions are made in the oil market, then imagine how difficult it becomes to estimate demand and supply and then forecast prices in much smaller markets such as lead, live cattle or lithium.

Opaqueness can also be a feature of major commodity consumers too. Despite China’s undoubted influence on global commodity markets, developments in its economy continue to remain opaque and hence so too are its implications for commodity prices.

In the past week, the Financial Times published an article revealing the fake gross domestic product data routinely released from many northern Chinese regions. Although it has long been believed that authorities “smoothed” the economic growth figures, it is now clear that many provinces artificially boosted growth figures between 2012 and 2016, masking a real downturn, and last year covered up a genuine recovery.

Unreliable data makes it difficult to assess risk, which raises the probability of some internal shock. Statistics are never completely accurate, especially when trying to estimate activity in far flung parts of the world. An analyst trying to figure out where the price of copper is going next has to first look at what could happen to copper demand growth. Will it slow, increase or even fall? For this you need to look at what businesses consume lots of copper, not just in the US, Europe or China but also in emerging economies where the quality and frequency of data may leave much to be desired.

Even if there were copious amounts of statistics available in real time and covering all aspects of demand and supply, that doesn’t mean that the data is the best and the final estimate. Think about population growth for a minute, which is thought by many to be the most predictable of all variables that could affect future demand for commodities. Ignoring the uncertainty about the future, even the baseline from which you could take a forecast is uncertain and subject to frequent revisions. As Dan Gardener remarks in his book “Future Babble”:

…demographic facts like this are based on available research, and when new research suggests the established “fact” isn’t accurate, it has to be changed. Between 1951 and 1966, the official estimate of the world’s population in 1951 changed 17 times.

Uncertainty about what is happening to commodity supply and demand, and thus the overall commodity balance, is not without cost. The uncertainty is likely to increase the risk premium that commodity producers use to decide on whether to invest or not. This increases the cost of that investment, perhaps resulting in delays or cancellations. The cost of which will only be felt during the next upturn in commodity prices when, yet again, producers may be caught in the dark.

Better data on commodity demand and supply outside of the US must be the priority if policymakers want energy and other commodity markets to operate more smoothly. The activities of large commodity producers are particularly opaque. Better data on consumption in fast growing emerging economies will also help to give a more accurate assessment of demand prospects.

Some private companies and often large institutional investors deploy people to count cocoa stocks in the Ivory Coast, use infrared cameras to monitor oil levels in storage tanks in the US or set up cameras to film coal stocks at Japanese power stations. All are set to determine the inventory fluctuations and price discrepancies through which they and their clients can profit. This kind of inside edge is outside of the scope for all but the wealthiest people. Nevertheless, some participants in the commodity markets are fighting back. Using a combination of technology, crowd sourcing and social media, they are beginning to change things for the better.

Samir Madani, a consumer electronics entrepreneur, saw an opportunity to combine real time data with his passion for oil markets. Traditionally, oil researchers and forecasters would talk about their expectations for supply and demand. However, without knowing how much is being exported and how much is going into storage, the overall supply and demand estimates are of limited use. I interviewed Sam for my latest book and he explained he launched the not-for-profit website Tanker Trackers to help increase market transparency by combining crowd-sourced data on crude tanker movements and official data published by government agencies. “I love real-time data and felt that the average Joe had no insight into what’s happening out at sea,” Sam told me as he outlined his vision of how he could help disrupt the market to the benefit of “ordinary” traders:

Two-thirds of the world’s oil is transported by sea and instead of hearing what crystal-ball-polishers think the price of oil would be end of the year due to supply vs demand, I felt that I could do something about it.

It’s here where crowd-sourced information on tanker loadings can help increase market transparency, hopefully allowing traders and physical market participants to make better decisions. For example, knowing that one week there was an unusually large spike in tanker loadings in Saudi Arabia destined for the US means that a trader can deduce that in approximately 45 days time there could be a big jump in crude inventories in the US.

Whichever way, the pursuit of better and more open data is likely to reduce uncertainty in commodity markets and potentially reduce price volatility too. Robert McNally, founder and president of energy consulting firm The Rapidan Group and the author of the book “Crude Volatility”, believes that much more should be done to force or incentivise the energy industry to become more transparent (the same arguments are just as valid for other commodity markets):

The oil market is turbulent enough, patchy and incomplete data make the problem worse. Energy data reporting should be lawfully compelled, timely and comprehensive. Upstream governments should require industry to disclose and validate field-by-field production and reserve data. Doing so would reduce surprises, manic hoarding, and price volatility. Downstream, figures for production, storage, net trade, and refining stocks and flows should be comprehensively reported, enabling much better implied demand estimates.

Data may be the new oil that powers the the global economy, but the former needs to improve for investment and innovation to really power progress.

Unlock commodity market insight now and subscribe to our email updates or

Probably the best podcasts on macroeconomics, markets and geopolitics

Over the past year I have become a complete and utter convert to podcasts. Tired of spending most of my day staring at a screen they are the perfect means to keep on learning but also giving your eyes a rest. I am constantly amazed at the quality of many of the podcasts out there. Sure many of them have a team of researchers providing background information but still to produce weekly podcasts of such a high standard is amazing.

The podcasts listed below are the main ones I follow to learn about broader macroeconomic and geopolitical trends and sometimes much more focused editions that look at the detail of a particular market or trend. Under each podcast below I have listed some of my favourite episodes (most likely with a commodity slant to it) and why I found it so useful. Hope you enjoy.

Adventures in Finance: From the team behind Real Vision, Grant Williams (@ttmygh) is a great host and interviewer. They interview some of the most famous investors and hedge fund managers, but also lesser known investors and analysts. They get you to really think about what is behind the story, not just the headline message.

Recommended episodes

– Number 34 – Jim Rogers: The Investment Biker Rides Again. Legendary investor Jim Rogers who is normally associated with commodity markets but as he explains looks for value anywhere and everywhere is a really interesting guy. He shows how important non-consensus thinking has been in his career.

– Number 14 – Feeding the Future: Debt, Demographics & Dinner Plates. Former Soros Fund MD Alan Boyce is an expert in agricultural commodity markets and outlines some of the factors that are far from common knowledge.

– Number 44 – Mind Games: Harnessing Market Psychology. Guests on this episode are Peter Atwater and Ben Hunt. They talk about how understanding market psychology is so important. Key insight for me is how private knowledge transforms into common knowledge or at least the perception.

Oddlots: From the team at Bloomberg (@TheStalwart and @tracyalloway) this is a weekly podcast that delves into everything from why the Beanie Babies craze took off to how algorithms impact every aspect of our lives.

Recommended episodes

– Why Wheat is the Worlds Most Exciting Market Right Now (17th July 2017). Their guest is long time agriculture trader Tommy Grisafi who explains what was happening to the wheat market in mid-2017 following a period of adverse weather. A really useful primer on what the main differences are between the various wheat markets in North America.

– The Incredible True Story of the Real Life ‘Trading Places’ (3rd March 2017). Delves beneath the surface of the so called ‘Turtle Traders’, novices who were given brief training in commodity trading by commodity speculator Richard Dennis.

Macro Voices: Former hedge fund manager Erik Townsend (@ErikSTownsend) discusses whats been moving the markets over the past week and his views on whats going to happen next. Very useful insights into see how someone else thinks about markets. Like many of the other podcasts listed here Macro Voices invites some great guests on the show to discuss things as diverse as Saudi Arabia, retail, cryptocurrencies and commodity prices. Sign up for the weekly email newsletter too that comes out following the show. On my podcast list for a longtime, but thanks to @MiningStockEdu for the upvote.

Recommended episodes

– Jack Schwager: Trading Futures (3rd Feb 2017). Although I’ve read a couple of Schwager’s books (where he interviews top traders) this episode was useful to hear from the man himself on the main lessons to takeaway in both technical and fundamental analysis.

– Rick Rule: Deep dive into pre-revenue resource investing (13th Jan 2017). Great insight from the commodity investor on how important it is to do your research on early stage resource companies.

Futures Radio Show: Podcast from futures trader Anthony Crudele (@anthonycrudele). Topics range from technical analysis, fundamentals and technology. One of the most easily understood of the hosts, Anthony keeps things very accessible and easy to understand.

Recommended episodes

– Episode 19: Developing a relationship with your market. Anthony interviews one of the most prominent oil and energy traders on Twitter @chigrl.

– Episode 140: Forecasting oil and gas markets: Anthony interviews Anas Alhajji (@anasalhajji) about the factors affecting energy markets and some of the mistakes that people make in forecasting.

Peak Prosperity: From Chris Martneson (@chrismartenson) at the blog Peak Prosperity. Many of the episodes are a bit doom and gloom (societal and financial collapse, etc.) but even that can be nice to imagine sometimes, or at least prepare for. Worth checking out Chris’ books too.

Recommended episodes

– Art Berman: Do not get used to todays low oil prices (7th May 2017).

– Michael Shermer: The importance of skepticism (3rd August 2016).

FT Alphachat: A healthy dose of skepticism from the FT Alphaville team. Definitely worth following @izakaminska and @cardiffgarcia.

Recommended episodes

– Dan Drezner on the economics of ideas (29th September 2017). Important episode on how ideas are being popularised by corporations and political parties.

– The making of the crisis in Venezuela (17th March 2017). Economist Ricardo Hausmann examines how the country got into the mess its in and what may happen in the future.

– Michael Mauboussin (@mjmauboussin) reflects on 30 years in the markets (27th Janaury 2017). I’ve read a number of books by the investor (definitely recommend “Think Twice”) but this is the first interview I’ve heard. Interesting perspective on how things have changed in the time he has been involved in financial markets.

Other podcasts worth a mention include…

Palisade Radio: Only recently discovered this one but seems to be very useful to understand investing in resource companies directly.

Monocle 24’s The Foreign Desk: Global-affairs show featuring interviews with political leaders and in-depth analysis of the big issues of the day. Only just discovered this one but will be checking it out. Thanks to @archie_hunterMB for the suggestion.

This list is certainly not complete. There are only so many hours in the day to listen to the shows and discover new ones. If there any others that you recommend be added to the list please let me know through Twitter.

Unlock commodity market insight now and subscribe to our email updates or

Most popular posts of 2017 on Materials Risk

Below are the 10 most popular posts from Materials Risk during 2017. If you haven’t already seen it my second book, “Crude Forecasts: Predictions, Pundits & Profits In The Commodity Casino was published this autumn. Do check it out. If you are interested in my other book recommendations check these out too.

Many thanks for your support during 2017. Here’s to a successful 2017.

Happy New Year


1. Cocoa prices: The top 10 most important drivers

2. Where do we stand in the commodity cycle?

3. Just how accurate are oil price predictions?

4. When will the 5 year agricultural bear market come to an end?

5. Batteries now included: You’ll meet a bad fate if you extrapolate

6. What lessons does rhodium have for commodity investors?

7. Everyday stable prices: How one entrepreneur is helping farmers plan for the future

8. How monetary policy and commodity markets collide

9. Narrative economics

10. The mocha hedge: Long planting cycles point to moribund soft commodity prices

Unlock commodity market insight now and subscribe to our email updates or