Adaptive Markets Hypothesis — Andrew Lo on Efficiency and Evolution in Finance

AuthorAthenarium / Posted on

Adaptive Markets Hypothesis — Andrew Lo on efficiency and evolution in finance
Adaptive markets, rationalizing animals, and evolutionary finance — Andrew Lo pushes academic finance in a much-needed direction.

Skip ahead

The Efficient Market Hypothesis

Following the Space Shuttle Challenger disaster in 1986, it took the Rogers Commission several months to fault NASA, Marshall Space Flight Center, and contractor Morton Thorkiel for “poor engineering management”. Yet, shortly after the disaster itself, the stock market had already “singled out [Thiokol for manufacturing] the faulty component”.

How could traders without any scientific expertise, insider knowledge, or crystal ball anticipate the Commission’s conclusion? For many academics and practitioners, this was evidence in favor of the wisdom of crowds and the Efficient Market Hypothesis (EMH).

According to the hypothesis, prices in an efficient market “fully reflect all available information”. If not, rational, profit-seeking participants would use their information to exploit and in turn eliminate mispricing. It follows then that historical prices cannot provide the speculator with new information about future prices.

The Efficient Markets Hypothesis, along with Optimal Portfolio Theory, Capital Asset Pricing Model, and Black-Scholes formula, represented a new age in academic finance. Indeed, many papers and events lend credence to the wisdom of crowds, and the markets’ ability to aggregate information efficiently. 

But as Andrew Lo reminds readers in Adaptive Markets: Financial Evolution at the Speed of Thought, the Efficient Market Hypothesis is actually about “what information is available to market participants” and “how prices fully reflect that information”. Debate and skepticism on both fronts remain heated to this day.

Perfectly efficient is impossible

Indeed, we know that in contrast to what EMH predicts, the invisible hand is not always accurate or instantaneous in its incorporation of information. Cobweb pricing models, for example, predict overshooting and undershooting around some equilibrium price and quantity. Markets can collapse altogether, or spiral out of control.

What’s more, efficient markets as an idea is somewhat paradoxical. As Stanford Grossman and Joseph Stiglitz argue, perfect markets eliminate the incentive to uncover mispricing. If you cannot make abnormal profits, why bother (especially when price discovery is expensive)? But if nobody gathered information, the market could not be perfectly efficient.

“There is a fundamental conflict between the efficiency with which markets spread information and the incentives to acquire information.”

Sandford Grossman & Joseph Stiglitz. (1980). On the Impossibility of Informationally Efficient Markets.

Fear, pain, and pleasure

I find it ironic that EMH dogmatists are unwilling to incorporate information from other models into their worldview. A complete theory of market behaviors must account not only for the wisdom of crowds, but for the “madness of mobs” too, Lo writes. EMH struggles to explain the manias and panics that happen from time to time. 

Indeed, a new theory should recognize our capacity for fear, envy, and hubris, and desire for pleasure seeking and pain avoidance. Daniel Kahneman and Amos Tversky made some headway with Prospect Theory, suggesting that many of us are risk averse in the face of potential gains, and risk seeking when it comes to loss avoidance. Risk-seeking in fear of losses may explain, for example, why traders like the London Whale continue to double down after repeated losses and bad bets.

Emotion, framing, and context affects decision-making as well. This is so even when the underlying math is unchanged. Lo’s research found, for example, that physiological responses are higher in inexperienced traders than experienced traders during volatile or news-breaking market events. (Unsurprisingly, emotional traders tend to earn poorer relative performance as well.)

“Winning money—and not even very much money—had the same effect on the brain as a cocaine addict getting a fix, or a patient given an injection of morphine. In each case, dopamine is released into the nucleus accumbens, reinforcing the behavior. With sufficient repetition, the action associated with the dopamine release becomes a habit. In the case of cocaine, we call it an addiction. In the case of monetary gain, we call it capitalism.”

Andrew Lo. (2017). Adaptive Markets: Financial Evolution at the Speed of Thought. 

Shotguns and maladaptations

A more complete theory of markets, Lo believes, should draw conceptually from biology and ecology. That is, to see how bounded rationality, investor sentiment, strategic variation, cumulative selection (by way of market forces), and incremental learning can lead to rich adaptations and behaviors in financial markets.

Indeed, Joseph Schumpeter long ago talked about the evolutionary process of creative destruction in economics. Armen Alchian, likewise, wrote about uncertainty, evolution, and economic theory; and adaptation via trial-and-error search and a “blanketing shotgun process” in the 1950s. 

We also know about adaptation under environmental pressure from competition strategy as well. Some organizations or strategies are so specialized that they cannot survive outside their specific niche. It’s like asking a poker champion to box professionally, and vice versa.

The rationalizing animal

Traditional economic models, however, assume that we are perfect, rational optimizers. They say that while history, institutions, and culture may affect our preferences, it does not shape the mechanism we use to calculate our best response. But we know, of course, that there are very real limits to calculation and optimization.

Even chess grandmasters tend to look four to five moves ahead in most positions. They rely on theory and principles to make moves, and calculate deeply only when necessary. And that’s just for the game of chess, a closed system with perfect information and infinite permutations.

What then should we expect of rational investors in a more nebulous market system? To arbitrage, we have to identify mispricing when we see them. But “our rationality”, Lo writes, “is biologically too limited for the Efficient Markets Hypothesis to hold at all times and in every possible context.”

Indeed, as George Akerlof and Robert Shiller emphasize in Animal Spirits, heuristics and narrative-making is central to everything human. This is true not only in politics and children stories, but in economics and finance too. People rely on trial-and-error, norms and institutions, and feedback to find their way under complexity and uncertainty.

As Lo writes:

“Remember, we humans are not so much the “rational animal” as we are the rationalizing animal. We interpret the world not in terms of objects and events, but in sequences of objects and events, preferably leading to some conclusion, as they do in a story. Our ability to choose an optimal behavior appears related to our ability to come up with the most plausible sounding explanation of the world: the best narrative.”

Andrew Lo. (2017). Adaptive Markets: Financial Evolution at the Speed of Thought.

The Adaptive Market Hypothesis

This brings us finally to Lo’s Adaptive Market Hypothesis. The hypothesis views financial market dynamics as a function of learning and adaptation alongside ever-changing institutions and environments. As markets select for and replicate successful ideas and variants, less successful mutations fade into obscurity.

What survives in the marketplace then is a product of continuous evolution via competition, adaptation, and innovation. The Adaptive Market Hypothesis realizes, Lo writes, “that despite the evolutionary pressures to maximize, … an evolutionarily successful adaptation doesn’t have to be the best; it only needs to be better than the rest”.

Note that success or fitness in this context is not necessarily what is smart, but what has staying power and replicability. As Akerlof and Shiller lament:

“Capitalism fills the supermarkets with thousands of items that meet our fancy. But if our fancy is for snake oil, it will produce that too. … This theme in the minor key has been forgotten. Yes, capitalism is good. But yes, it has its excesses. And it must be watched. … We need to realize that the stories people tell themselves about the economy exaggerate.”

George Akerlof and Robert Shiller. (2009). Animal Spirits: How Psychology Drives the Economy, and Why It Matters for Global Capitalism.

Lucky Munger and chess-in-flux

Conventional wisdom in finance tells us that we cannot beat the market, and that we must accept higher risk for greater rewards. It suggests that investors like Charlie Munger are either extraordinarily lucky, risk-seeking, or both. The Adaptive Market Hypothesis disagrees. Opportunity and risk depends instead on the environment and what everybody else is doing. It leaves plenty of room for shenanigans and tomfoolery.

Adaptive reasoning requires us to distinguish between macroclimates and microclimates, between systematic and idiosyncratic risk. Indeed, risk and value undergo distortions when people everywhere converge on collectively foolish ideas. Boom-bust cycles, from the dotcom mania to the subprime bubble, illustrate this well — like an infectious disease that tears through a field of monoculture crops every so often.

As Lo explains: 

“The Adaptive Markets Hypothesis shows that differences in behavior come down to reproduction and elimination, given the environment. … When reproductive risk is idiosyncratic, the optimizing behavior of the economist tribbles can persist because there aren’t any system-wide shocks that can wipe out an entire subpopulation that exhibits the same behavior. … In the case of systematic risk, … there’s just one coin flip for the entire generation …. If they all behaved in the same way, … [then] the entire population suffers the [same] consequences.” 

Andrew Lo. (2017). Adaptive Markets: Financial Evolution at the Speed of Thought.

Reflexive feedback loops

In this way, manias, panics, and imitation are examples of adaptations or maladaptations to changing niches and environments. Lo describes, for instance, how the once lucrative business of high frequency trading has since been watered down by new entrants and competition.

Ideas of maladaptation are not dissimilar to George Soros’ theory on reflexivity. He describes how fallibility, reflexivity, and positive feedback can push prices away from theoretical equilibrium. Financial bubbles reoccur and persist for this very reason. EMH “has certainly proved inadequate”, Soros writes — preferring instead to focus on reflexive feedback loops.

Thinking adaptively, however, does not imply the rejection of EMH. Rather, it sees it as a very special case of market system dynamics. In a predictable, stable environment, learning and adaptation might push behavior towards the optimum. Indeed, it is easier to play chess well when the rules aren’t forever changing. But how often is this true of the economy?

“An efficient market is simply the steady-state limit of a market in an unchanging financial environment. Such an idealized market is unlikely to ever exist in practice, but it’s still a useful abstraction … Markets do look efficient under certain circumstances, namely, when investors have had a chance to adapt to existing business conditions, and those conditions remain relatively stable over a long enough period of time.”

Andrew Lo. (2017). Adaptive Markets: Financial Evolution at the Speed of Thought.

New frontiers

Of course, the Adaptive Market Hypothesis, much like the EMH, is only a start. While promising, it is still early days before we achieve a general theory on market dynamics. It is one thing to say that markets behave as they do because they are efficient or adaptive. It’s another thing to make good predictions from it. 

While mutation, selection, and evolution are instructive ideas, Paul Samuelson warned many decades ago about the dangers of forced analogies. Yet it is also equally short-sighted to disregard the mental models and deep principles that cut across different fields. Un-mined riches for economics, I believe, remain in the fields of biology, psychology, and ecology.

The trick, of course, is in knowing where the parallels stop. So with all that said, Andrew Lo’s Adaptive Markets is, I think, a much-needed step in the right direction. It’s a reminder of the long road ahead for the field. To future students in finance, this prospect should excite you.

Sources and further reading

Recent posts