Positioning analysis in commodity markets: An interview with Mark Keenan

I recently had the opportunity to talk with Mark Keenan. Mark is Managing Director, Global Commodities Strategist & Head of Research Asia-Pacific at Société Générale. He is also the author of the book, “Positioning Analysis in Commodity Markets – Bridging Fundamental & Technical Analysis”.

In this interview you will learn why it’s important to be aware of the objectives of all market participants in commodity markets and why using methods like positioning analysis offer very good insights into whose doing what, and what happens when they do too much of it and how much positioning then relates to other factors like price, curve structure and fundamentals.

What first got you started in commodity markets?

I think like perhaps many of us I didn’t study what I ended up doing, much to some peoples disappointment, but it’s turned out ok. I studied chemistry and biochemistry so I was also drawn into the world commodities because I felt I had some edge in understanding things like the chemistry of oil and how its refined, metals, and how they are produced, including the various smelting and refining processes, and by extension in agriculture the genetic modification of crops. I liked all of that, understood it and I figured commodities were real. I studied science and that was real, and so I thought that if I’m going to get into finance, this is a suitable place with some degree of overlap.

The thing that really got me into modelling and trading it was that I was quite ‘quanty’ at university where even the chemistry that I did, was quantitatively driven. I worked on the human genome project which is not so different from what I do today, which is looking for patterns and sequences in vast amounts of data. Specifically, at the time when I started out, it was the birth of computerised trading in so far as all the CTA models and trend following algorithms were starting to proliferate and develop. This was much in alignment with the growth in software that became available at the time to test and build strategies such as TradeStation and Metastock. I spent my early days endlessly building and programming algorithms and of course commodities were one the main markets that leant themselves well to that approach.

The “real” nature of commodities and a background in modelling were the two main things gave me my background in commodities and seeded my interest in the asset class.

What gets you excited about talking about commodities?

I think it’s important to make a distinction between talking about markets and talking about commodities. The thing that is so fascinating for me about commodities is that they are so very different from the other financial assets. You have such a vast array of objectives among the many different market participants which adds huge insight into behaviour. Very simply, if you look at the futures markets, which is the main point of access to commodity markets for most people, you have consumers and producers which are essentially “hoping” to lose money with their derivative hedging positions and make money in the physical world; you’ve got speculators hoping to make money with directional bets, but in reality, many are playing relative value positions, and collectively all of these groups are trading across a forward curve, so the point or location in time at which they are doing it, also varies considerably. So that’s hugely interesting, and distinct from the relatively uniform objective of participants in the equity market for example, space where pretty much everyone is seeking to make money out of their position.

I think also that uniquely in the commodities space, there is often a mismatch between analysis and research and the actual price response despite often considerable consensus in a view that make the asset class tradeable. By that I mean that you often come across situations, where you’ll see that many people are bullish sugar for example, and because of transport dynamics, including port and vessel logistics and many physical characteristics of the market; everyone may think the market is going to go up, but they may not be able to effect that view. Simple they might not be able to buy all the sugar that they need now, because they just don’t have the budget or the credit lines or the vessels to do it. By doing research and understanding what the underlying fundamentals are, because of these structural hurdles, there is often a slower response in terms of how the price catches up with the view. This allows positions to be taken from an investment, trading or speculative perspective that have time to make money.

As many of these logistical constraints force shifts in fundamentals to persist for some period of time, prices often become very serially correlated. Interesting mismatches between research and analysis and the eventual price response occur that are often monetizable due to these relationships not being contemporaneous (like in many other asset classes). I like that disconnect, that’s what you can get your teeth into, and that’s what can make money.

Even for speculators, I would say that 70% are CTA driven in the commodities space which generally means they have a rules based strategy or they are technically driven and are hence often constrained in how flexible and responsive to a view they can be. For other types of investors, whilst they may think a commodity may go to price x they are often subject to enormous restrictions in terms of their trading and investment mandates and can’t really effect the view either. Lastly on and almost on a daily basis, I come across people that have a view on a particular commodity, whether its natural gas, gold or crude oil and they don’t really understand how to capture that view. They are simply not familiar with the access instruments, or they are not comfortable with the instrument like futures or options.

Collectively many of these properties and dynamics, combined with the innate diversity across the complex make commodities fascinating assets to trade in a financial framework. Their sensitivity to their own endogenous idiosyncratic price drivers, but also to outside macroeconomic variables combined with how well they lend themselves to data analytics make them fascinating to talk about and to study. They are always changing and there are always new ways of looking and thinking about at things, fundamentally, technically and for a sentiment/behavioural and positioning perspective – that’s what I like.

How should traders and investors use that blend of approaches – fundamental, technical and positioning?

I get that all of the time when I’m talking to people who are just entering the markets. Should I be a technical or a fundamental trader? And what I hope to try and do is to break down many of the preconceived barriers or notions about what types of analysis works and to not think about research or trading in silos. I don’t think that makes sense anymore, or whether it ever did. A blend of a variety of different analytics is essential, especially with the growth in technology, and the fact that everyone with a website can now plot reasonably advanced technical analysis for all commodities, as distinct from 20 years ago when they had the point and figure charting on the wall of the NYMEX exchange, and price data was very hard to obtain. Now, with it being much more accessible, I think a more balanced exposure to different ways of analysing the market is very sensible, with the challenge very simply lying in knowing when you need to look at fundamentals, when technical analysis is important and when the slightly more esoteric things like sentiment and behavioural patterns and positioning can help. This is what makes the difference between a good trader and an excellent trader.

You say that the individual can produce relatively sophisticated analysis but isn’t the playing field still weighted against them when they are up against commodity trading funds?

That’s an interesting question. I think most certainly the physical market participants including the trade houses have a phenomenal window into the actual commodity market and in many instances influences over the movement and storage of commodities all over the world. This affords them deep insight into how commodity prices are going to behave. They do however tend to be less familiar with the investment world, which has a significant interest in commodity markets and is growing by the day, with how prices are behaving. They have very little understanding of how investors behave and what they want out of commodity markets. I’m constantly sitting down with trade houses, that might be bearish soybeans for example, and I say I’ve been talking to a pension fund that’s just increased their commodity exposure, which means they will be buying soybeans. Often they can’t reconcile this and therefore don’t understand why they still buy them. They’re not familiar with commodity indices, diversified exposure and hedging against inflation and all the other objectives.

It’s important to be aware of the objectives of all market participants, as the market continues to grow and become more diversified – especially now with computerised trading strategies and algorithms. A lot of the physical world is very nervous about this; they don’t understand what this might mean for prices.

In short, with so many different market participants, and a fascinating blend of different strategies, using methods like positioning analysis offer very good insights into whose doing what, and what happens when they do too much of it and how much positioning then relates to other factors like price, curve structure and fundamentals. This can help all market participants enormously – especially individuals.

How should someone think about a market where money managers are holding a large net short position? That it could unwind and so see prices rise sharply, or that money managers have typically done well in the past and so prices are likely to stay lower for longer?

There are a few factors to take into account; firstly it is very important to have some view as to whether the market is range bound or whether its trending – be that up or down – since this can alter the interpretation of positioning quite significantly.

Secondly, one of the things I have found interesting, is that money managers and commodity speculators since 2008 have had a very difficult time making money. If you track hedge fund indices based on the performance of commodity hedge funds, they have been in near linear descent. Anecdotally, we also continuously hear reports of hedge funds winding down. The question therefore of what to do when a manager has large position lies in whether to copy them or to bet against them.

One of the things in the book that was addressed is how to measure where money managers are making money. Understanding this is very important as it can help shape behaviour. For example, hearing that fund managers are very long natural gas for example – a market that they’re generally very profitable in, could suggest that not going against this trend would be sensible.

Perhaps the most important thing that I think is critical is that positioning analysis cuts traders up into various buckets and then they have to behave accordingly to what bucket they are in. For example in the money manager bucket they have to close out their positions, they can’t go to delivery, they can’t play in the physical market because if they could they would be in a different category. So when you see a very large short position that means very simply that they are going to have to close that position out. And if you see that in the context of a very low price suggesting that those money managers have made some money building up that short position and prices have fallen, that can be a very interesting buying opportunity for consumers as there is a high risk that those prices move higher as a function of short covering. And likewise on the other side when you have a very large long position and prices are very high, that might be a very good time for a refiner to sell some of their products. Simply using the position of money managers, whether they are right or wrong in their direction can really help you time your trading and hedging more effectively. We’ve found that many of the clients that we’ve spoken to and showed them these very extreme positioning its been very helpful.

What was your objective behind writing the book?

My objective very simply was to define positioning analysis as an area of research that is a very powerful framework for understanding how commodity prices behave. And I think that positioning data is probably the most underutilised set of data in the commodity markets. Over the years through building models I’ve realised that it is incredibly powerful so the objective behind this book was to set out numerous strategies and models that show different ways of thinking about how positioning data firstly bridges fundamental and technical analysis and how changes in positioning patterns inter-plays with price, curve structure, inventories, cancelled warrants in the metals sector, seasonal factors and many others. It’s fascinating as you’re identifying how people behave in the context of a wider variables and that really helps trading.

People have said to me that one of the biggest hurdles to using positioning data is that it is lagged. But my response to that is that every bit of data is lagged. The fascinating thing to me is when I ask them what you mean by the word lagged. They seem to infer that someone has the data before them, that in some ways that it has been corrupted in some way. When you look at inventory that people often attach even more importance to that is lagged and that is data that is most definitely corrupted because the people that control the inventory and have built storage facilities they know pretty much before the market participants. If you take energy as an example when the Dept. of Energy release their inventory numbers all the oil market has a very good idea of what’s going to happen, yet people don’t talk about that as being lagged. Whereas with positioning data every single person in the world, except the US government gets it at the same time. When you look at it like that that allays some of those concerns.

So my objective with the book was to break this data apart and blend it with all these inputs, that we often look into isolation, and try and tie them together into very simple models. The look to see what happens when these variables reach extremes and when they move in trends. What does that mean for prices, curve structure and behaviour patterns. Certainly over the years I’ve found these relationships to be extremely powerful.

There’s a chart in your book that shows Bloomberg search terms for positioning, and while its clearly become more popular in recent years it’s still not mainstream.

I was trying to explain positioning to people that are not in the market and on a very basic level its like gossip. Everyone is interested in that on some level. If you go to a technical analysis in commodities and you say to him that copper inventories are at a ten year low, he or she is not going to really care about that information. If you go to a fundamental analyst and you explain that the MACD lines have crossed over last week they are not going to know what that means potentially. But if you go to both of them and you say guess what all the consumers did last week? They are not going to say we’re not interested. That’s useful information. How you cut it and deliver that information is crucial. As you say there is a big gap between technical and positioning analysis and positioning analysis fills that well.

Any plans for a future book?

People have asked me to extend the book and do a more advanced version with all the latest data – a sort of advanced positioning analysis that might build on this.

One of the charts in the book is based on news flow. One of the areas we work a lot here at SocGen is building news flow indices to do all sorts of things, to assess geopolitical risk, the weather, trends in fundamentals and price and so on. I think that news flow analytics is doable now because of the computer power available now and the integrity of news databases. Perhaps my second book will revolve around news flow analysis as a new type of momentum signal. It’s quite fascinating when you look at the correlation between a basic technical trend following model for example – one where the signals are generated not by changes in price, but by changes in news flow, where people talk about copper going up or down for example. When you look at that dynamic, it tends to give you a very different return profile.

Thanks for your time today Mark. Where can people find or follow you?

You can follow me on Twitter @markjskeenan

Unlock commodity market insight now and subscribe to our email updates or

Soybean prices: The top 10 most important drivers

1) Weather

The weather has a significant impact on crop yields and thus overall agricultural production. The wrong type of weather at the wrong time in the planting cycle, even if not prolonged or extreme, can also adversely affect the production of soybeans. The weather can also affect the shipment and logistics of transporting crops to market.

Heavy rain in the north and west-central regions of Brazil (where most of the worlds crop is grown) makes working conditions in the field more difficult and reduces the planting pace, in turn hurting the soybean output. Meanwhile, adverse weather conditions in the south of the country (where most of the crop is shipped from) could affect the supply of soybeans out of Brazil while delays could also increase wastage, further reducing supply.

2) Protein demand

The importance of China and other markets in Asia as a market for soybeans has been driven by an explosion in demand for meat as consumers switch from a diet dominated by rice to one where pork, poultry and beef play an important part. Soybean meal (a product of soybeans) is the most important protein source used to feed animals and represents about two-thirds of the total world output of protein feed-stuffs.

3) Chinese demand

China imports around 60% of global soybean imports, around 90% of its requirements. Chinese production of meat surged 250% between 1986 to 2012. However, China is unable to produce enough animal feed itself, hence the need to import soybeans from the United States and Brazil.

4) The US Dollar

Like most internationally traded commodities soybeans are priced in US dollars. At its most basic a decrease in the value of the US dollar relative to a commodity buyer’s currency means that the purchaser will need to spend less of their own currency to buy a given amount of the commodity. As the commodity becomes less expensive demand for the commodity rises, resulting in an increase in the price and vice versa.

A weaker dollar can also act as a disincentive to producers to increase output. For example, a depreciation of the US dollar against the Brazilian real can reduce profit margins for a soybean farmer in Brazil. All of the farmer’s revenues will be received in US dollars, which will now buy less real, but some proportion of the costs will be denominated in real and will remain constant (at least in the short term). Therefore, the prospect of a lower profit margin acts as an incentive to decrease the supply of soybeans.

5) Tariffs

Given soybeans position as one of the most important sources of food governments may use this as a key element of trade policy with other countries. In 2018 for example China announced that it would increase tariffs on imports of soybeans from the US. The impact of such a tariff would be to increase the price of soybeans in China, but for the price to fall in the US.

6) The price of other crops

Corn and soybean compete on multiple grounds. They compete in the cooking oil industry. They compete in the animal feed industry. They also compete in the biofuel industry. Therefore, the production of one does affect the other. In general, if the production of corn falls, soybean prices are expected to rise.

7) Stock levels

Stocks (otherwise known as inventories) act as form of buffer for both producers and consumers of soybeans. Typically, falling stock levels occur if demand increases faster than supply, resulting in higher soybean prices. Falling stock levels may, however, make the soybean market more vulnerable to an unanticipated disruption to supply or a sudden increase in demand.

8) Speculation

Hedge funds operating in agricultural markets including soybeans may result in prices moving faster than they otherwise would in the event that there is deficit or surplus in the market. Speculators play a vital role then in signalling to farmers how much they should be planting while also allowing the transfer of risk.

9) Energy costs

Higher energy costs imply higher costs of production for soybeans and higher costs of transporting soybeans to market. Energy makes up a significant part of operating costs for most crops. This is especially true when considering indirect energy expenditure on fertiliser because the production of fertiliser is extremely energy intensive, requiring large amounts of natural gas.

10) Speculation by funds

Over the short term at least the positions held by large commodity funds can have a significant impact on soybean prices. If positions in the soybean futures market become extremely overbought or oversold then, as in other commodity markets news that counters this view can result in a sharp upswing in price volatility in the market.

Previous episodes

Gold prices: The top 10 most important drivers

Silver prices: The top 10 most important drivers

Natural gas prices: The top 10 most important drivers

Copper prices: The top 10 most important drivers

Livestock prices: The top 10 most important drivers

Sugar prices: The top 10 most important drivers

Cocoa prices: The top 10 most important drivers

Palladium prices: The top 10 most important drivers

Unlock commodity market insight now and subscribe to our email updates or

Firming up fuzzy forecasts

What can the forecasting profession do differently? Well, like most things in life, change will only happen if you demand it. Sharp words about failed predictions are essentially forbidden where they are most needed. So for all the incentives pushing experts to pump up their predictions, there must be a countervailing incentive to tone it down. In reality, there is little accountability for predictions, and while big calls that go bad should damage the reputations of those who make them, they seldom do.

Given how pervasive and influential commodity price predictions are, there is a surprising lack of data into how accurate the forecasts have been and which forecasters have the best track record. According to Philip Tetlock and Dan Gardner – authors of “Superforecasting: The Art and Science of Prediction” – there is a lack of accountability when it comes to financial forecasts. “Every day, the news media deliver forecasts without reporting, or even asking, how good the forecasters who made the forecasts really are,” say Tetlock and Gardner. They continue:

Every day, corporations and governments pay for forecasts that may be prescient or worthless or something in between. And every day, all of us – leaders of nations, corporate executives, investors and voters – make critical decisions on the basis of forecasts whose quality is unknown.

Governments, businesses, investors and individuals don’t demand evidence of accuracy before deciding whether to accept and act on a prediction. Forecasts are routinely made but the results are almost never tracked. As noted earlier, prominent forecasters build reputations not because of their accuracy but because of their skill at telling a compelling story with conviction.

Predict if you want, and rely on predictions if you really need to, but keep a tally of the predictions. Although my review of oil price forecasts over the past ten years was relatively easy to carry out, many of the predictions that pundits make are much more difficult to gauge. Part of the problem is the fuzzy language in which many predictions are often expressed, making it difficult to tell if the forecast was right or wrong even after the event.

Karl Popper famously made the observation that the usefulness of a prediction was related to its potential for falsification. “It will rain in London in the future” is a statement that is 100% accurate, but useless when it comes to telling us which day we should carry an umbrella. A statement that it will rain at 10.30am tomorrow is much more useful; it will rain or it will not. If it doesn’t, then we can examine what assumptions were used in the forecast that turned out to be false.

Forecasts are often expressed using ambiguous words like probable, possible and risk, for which there are no agreed definitions, making it impossible to score them afterward. The US Intelligence Community has famously struggled with the lack of precision in the meaning of words that are commonly used to express likelihood and chance since the 1960s. Sherman Kent, often described as the “father of intelligence analysis”, was a CIA analyst that recognised the problem of using imprecise statements of uncertainty. Particularly, Kent was jolted by how policymakers interpreted the phrase “serious possibility” in a national estimate about the odds of a Soviet attack on Yugoslavia in 1951. After asking around, he found that some thought this meant a 20% chance of attack, while others ascribed an 80% chance to the phrase. Most people were somewhere in the middle.

Investor and author Michael Maubossin recently posted a survey to gauge how people view probabilistic language. Take ‘real possibility’ as an example. To some people this meant a probability of just 20%, but to others it meant an 80% likelihood. Next time you hear someone say there is a ‘real possibility’ of this or that happening to a particular market you know they are just trying to make a forecast that no one can hold them to account for.

Remember, hits and misses don’t come with labels. It’s often a matter of perception whether a forecast is deemed to be a hit or a miss, which makes language important. The more ambiguous the wording is, the more a pundit’s prediction can be stretched. And since we want hits, that’s the direction in which things will tend to stretch. As Dan Gardener describes in his book “Future Babble”:

When the notoriously vague Oracle of Delphi was asked by King Croesus of Lydia whether he should attack the Persian Empire, the oracle is said to have responded that if he did he would destroy a great empire. Encouraged, the king attacked and lost.

The same confusing probabilistic terminology are used by pundits, and are then often also used by many in the financial media to imply something is much more likely to happen than it actually is. We saw an example of that in a previous chapter when Goldman Sachs suggested that because of limited spare capacity oil prices “could lead to $150-$200 a barrel oil prices” – many in the financial media interpreted “Could” as “Will”.

In other cases, relatively specific forecasts are matched with an unspecific time frame, which also makes it difficult to score them for accuracy. There is a maxim among professional analysts that cynically confirms the problem: “always predict a price, or a time frame, but never both”. However, in recent years, some commodity market forecasters have been pushed to quantify their forecasts by making specific price predictions over specified time horizons. Many have also embraced uncertainty by offering forecasts in the form of a probability distribution, rather than a point estimate, which is a much more useful and realistic way to think about the future. Commodity market forecasters are catching up with weather forecasters and the US Intelligence Community in trying to estimate the likelihood of a whole range of outcomes, not just the central one.

Percentage forecasts are an important step forward, but the commodity market is still lagging behind in terms of measuring forecast accuracy after the event. The problem with percentage forecasts is working out whether they were accurate even in retrospect. Tetlock and Gardner call this problem “being on the wrong side of ‘maybe’”. To understand the problem, imagine a weather forecaster who says that tomorrow there is a 70% chance of rain. The forecast also implies there is a 30% chance it will not rain. If it doesn’t rain, the forecast was not necessarily wrong in a statistical sense but it is still likely to be criticised by anyone concentrating on more than just the most likely outcomes.

Although assigning probabilities to particular scenarios is an improvement, it is far from being a panacea. It can give an impression of faux certainty, that all possible outcomes are knowable in advance and have been captured by the forecast. Known as “Knightian uncertainty”, probabilities cannot be assigned to different outcomes because the existing distribution of possible outcomes is unknowable.

The danger with faux certainty is that it might lead market participants to believe something is more likely than it actually is. Consumers of forecasts might think that all of the possible outcomes have been captured by the commodity forecaster and then act based on this supposed “evidence”. Remember, pundits see uncertainty as something that threatens their reputation. Often forecasters will make assumptions in their models that lower the perception of the degree of uncertainty in future prices.

Again, the solution to firming up fuzzy forecasts is to track performance over time. This would weed out those forecasters unable to capture the range of possible outcomes accurately and the over-confident from the accurate. Meteorologists pioneered the solution to the probability forecasting problem and the solution was published by Glenn Brier of the US Weather Bureau in 1950. Brier published a careful methodology for comparing a set of forecasts expressed as probability distributions with eventual outcomes, and scoring forecasters on a standard scale from zero (complete accuracy) to 2.0 (perfect inaccuracy). The most accurate forecaster is the one whose forecast probability distributions get closest to the distribution of actual out-turns over time. If a forecaster predicts there will be a 70% chance of rain, they should be correct approximately 70% of the time.

Verifying accuracy is obviously much easier for weather forecasts, where thousands of fresh forecasts are issued every day and can be compared with thousands of outcomes. Verification is more difficult for subjects like commodity prices but, given how frequently prices are forecast, it is not impossible and would be highly desirable.

Brier scoring price forecasts could also bring important benefits for commodity markets. The aim would not be just to identify the most accurate forecasters, those most worth paying attention to, but improving the accuracy of all forecasts by subjecting them to rigorous analysis after the event. Weather forecasts have improved enormously over the last fifty years because they have been subjected to rigorous analysis. It is far less obvious that forecasts for commodity prices and other financial markets have become any better.

In “Black Box Thinking”, Matthew Syed describes how the airline industry actively promotes the sharing of mistakes and failures in order to help propel the safety of the industry forward. Other professions are not so great at looking at past mistakes, learning from them and improving their processes and models to make the future better. Syed describes how psychotherapists gauge whether their treatments are effective, not by observing the patient with objective data over a long period of time, but by observing them in the clinic. As well as being prone to all kinds of biases, both from the patient and the psychotherapist, there is no feedback on the lasting impact of the treatment and hence no opportunity to learn – from success and from failure.

There is no reason why learning from mistakes and failures should not be part of the job description of the analysts involved in commodity forecasting. Capturing data is not a problem (whether that be prices, futures market activity, etc), information on market events are well publicised (for example, changes in interest rates, currency movements or political instability). All it needs is a change in attitude.

Unlock commodity market insight now and subscribe to our email updates or