The Wisdom of Crowds — James Surowiecki on Ant Mills and Financial Spirals

AuthorAthenarium / Posted on

The Wisdom of Crowds — James Surowiecki on ant mills and financial spirals

Smart crowds everywhere?

According to James Surowiecki in The Wisdom of Crowds, expert phone calls on the game show Who Wants to be a Millionaire answer around 65 percent of their questions correctly. The audience poll, by contrast, has a correct answer rate of 91 percent. The generalist audience sounds impressive, no? To Surowiecki, smart crowds are everywhere.

One instance is the 1986 space shuttle Challenger disaster. In a supposed case of market efficiency, investors faulted Morton Thiokol within days of the explosion, driving its stock price down. Indeed, Thiokol had made the O-ring seals that failed during the mission. But it took the Presidential Commission six months to confirm that the market’s hypothesis was correct.

For the more mundane, Francis Galton found that crowds were better at guessing the weight of oxes than individuals. Then there was Dr John Craven, who located the lost submarine by crowdsourcing ideas from his team and applying Bayesian search theory. To Surowiecki, these are examples of “collective wisdom”. That “many [can be] smarter than the few”. 

Conditional wisdom of crowds

But we’ve also seen groups of all sorts make dumb decisions. Gargantuan enterprises are often slow, bureaucratic, and political. They talk too much and do too little. Small groups, while nimble, might lack the resources, know-how and scale to move the needle.

When then might we expect to see some wisdom from the crowds? Surowiecki argues that there are four requirements for the group or system to satisfy: (1) diversity, (2) independence, (3) decentralization, and (4) aggregation.

That is, people must bring a variety of opinions or ideas to the problem. And their views must not be totally influenced or warped by their peers around them. Of course, open-mindedness to evidence and debate is healthy. The system or solution device must also be decentralized in the sense that it can tap into the local knowledge of each individual or group. But it must also contain some aggregation device to collate and coordinate disparate contributions into collective direction.

Now, we’re not saying that a handful of drunkards at your local bar can solve complex equations better than an engineer. Quality matters. What Surowiecki is saying is that large, abled groups that satisfy the four conditions above can outdo the lone genius on general problems.

Surowiecki draws inspiration from the behavior of social insects. He writes, for example, about the astonishing ability of bees to find nectar, sometimes up to several kilometers away from their hive. Bees rely not on rational calculation, but on trial-and-error search. Their scouts do a little dance when they find promising resources, alerting others to the location. In this way, simple, dependable rules, applied in numbers, provide the hive with a viable strategy for finding food.

Human innovation is not all that different. Surowiecki notes, for instance, that there were “literally hundreds” of automakers in the early twentieth century. Inventors were not yet sure about the ideal form of cars, and so were trialing all manners of types. The battle between petrol, steam, and battery power was not yet won. And it took some time for manufacturers like Ford to standardize the process, and for industry to consolidate.

This process of search and selection is a common feature in the advance of  science, technology, and organization. Here, the wisdom of crowds stems from the diversity of trying things differently, to see what works and to converge on what does. It is a complex mixture of experimentation, adaptation, and replication. 

“The early days of the business are characterized by a profusion of alternatives, many of them dramatically different from each other in design and technology. As time passes, the market winnows out the winners and losers, effectively choosing which technologies will flourish and which will disappear.”

James Surowiecki. (2004). The Wisdom of Crowds.

Ant mills, financial spirals, and independence

Ant behavior, like bees, depends not only on environmental signals, but on the surrounding ants. Simple response rules like ‘follow the ant in front of you’ can help one find useful things to do. In some instances, however, the ants can lock themselves in a death spiral — where ants are following one another in a “circular mill”. This is self-organization gone awry.  

We can find ant mills in financial markets too. In Irrational Exuberance, Robert Shiller describes the investor’s tendency to herd during periods of uncertainty. Unsure investors find comfort in the crowd and the prescription of advisors and gurus. Many fund managers, likewise, prefer to underperform as a group than to risk their career on a contrarian bet. 

Ant mills and Wall Street show us why independence is important to the wisdom of the crowds. The ant’s follow-rule works when other ants are acting (not following). Similarly, the herding rule in finance is fine if there exists a sufficiently large pool of independent actors who are courageous enough to correct mispricings. 

As Surowiecki writes:

“Once each individual stops relying on his own knowledge, the cascade stops becoming informative. Everyone thinks that people are making decisions based on what they know, when in fact people are making decisions based on what they think the people who came before them knew. Instead of aggregating all the information individuals have, the way a market or a voting system does, the cascade becomes a sequence of uninformed choices, so that collectively the group ends up making a bad decision.”

James Surowiecki. (2004). The Wisdom of Crowds.

Networks, decentralization, and aggregation

Decentralization, the third element, has received more attention recently. Indeed, despite the absence of any central planning, decentralized networks, from beehives and ant colonies in biology, to the Internet and Wikipedia in technology, appear to function rather seamlessly.

In human systems, Surowiecki says decentralized networks like cities, economies, and markets benefit from two advantages. The first is specialization, which is, of course, a fundamental source of labor productivity. The second, as Friedrich Hayek writes in The Use of Knowledge in Society, is access to local, unorganized knowledge (or tacit knowledge).

Central planners cannot hope to amalgamate all the information necessary for complete and efficient decision-making. Can you imagine governments trying to set price levels for every good and service in the marketplace? While the price system we have today is imperfect, it does a reasonably good job of responding to little changes in demand and supply.

Without maintenance, connection and aggregation, however, networks will degenerate into disarray. Surowiecki writes that “two years after September 11, the [U.S.] government still did not have a single unified watch list that drew on data from all parts of the intelligence community”. One can imagine what this implies about the network’s coverage and adaptiveness.

Centralization and aggregation of the firm

To emphasize Surowiecki’s point, for decentralized networks to function well, it must contain mechanisms to evaluate and aggregate local knowledge effectively. This is why the market economy price system, Google search engine algorithms, and academic peer-review process work as well as they do. 

Similar arguments apply to the nature of the firm. Hierarchy and bureaucracy often removes the person closest to the problem from actually solving it. Surowiecki notes, for instance, that “the design of a new headlight [at General Motors] had to be considered in fifteen different meetings”. What’s more, “the CEO of the company sat in on the last five of those”!

Many of us, I think, have worked for executives and policymakers who are very consultative in their approach to problem-solving. They believe, and rightly so, in the diversity and wisdom of the crowds. Many of them, however, are terrible listeners and aggregators. So despite all their good intent, they destine themselves to circular chatter and meetings. 

“Perhaps the deepest problem with the rigidly hierarchical, multilayered corporation was—and is—that it discouraged the free flow of information, in no small part because there were so many bosses, each one a potential stumbling block or future enemy. … Paradoxically, in trying to make the decision-making process as inclusive as possible, companies actually made top executives more—not less—insulated from the real opinions of everyone else. Before any decision could be made, it had to make its way through each layer of the management hierarchy.”

James Surowiecki. (2004). The Wisdom of Crowds.

The madness of crowds

Of course, there are countless books about individual irrationality. To wisdom-of-the-crowd advocates, however, this does not prevent collective rationality. This seems reasonable, so long as the errors cancel out. On the other hand, political and financial history reminds us that crowds can and do succumb to mania, folly, and extremism. 

The problem, as you know, is that errors don’t always cancel out. Groups are not always diverse or independent. Systematic errors and converging behaviors arise frequently. The author points to the persistent trend of household under-saving as one example of this. Here, what is an individually suboptimal decision is a collectively poor outcome, too.

Another example is the recurring phenomenon of financial bubbles. Surowiecki writes, for example, about the bubble for bowling stocks in the late 1950s. Market prices reflected expectations that bowling would become the new national pastime. (The bubble, of course, lost much of its steam by the mid 1960s as the bowling fad cooled.)

Additionally, what seems individually sensible is not always collectively wise. As George Soros reminds in his principle of reflexivity, rational speculators will partake in mania if it is profitable, propping up the bubble in turn. The iterated prisoner’s dilemma and the tragedy of the commons, likewise, speak to a larger class of crowd coordination problems.

“Bubbles and crashes are textbook examples of collective decision making gone wrong. In a bubble, all of the conditions that make groups intelligent — independence, diversity, private judgment—disappear. … [it follows that] the more investors who refuse to buy stocks just because other people are buying them, the less likely it will be that a bubble will become inflated. The fewer investors there are who treat the market as if it were Keynes’s beauty contest, the more robust and intelligent the market’s decisions will be.”

James Surowiecki. (2004). The Wisdom of Crowds.

Wisdom, democracy, and dysfunction

What then should we make of our most important crowd that is democracy? The average voter, like you and I, do not have the time to research every policy in detail. Experts, likewise, have only deep domain expertise, not a complete view. Can we trust the diversity and independence of crowds, and the democratic system to aggregate local knowledge appropriately?

Surowiecki agrees that this is a difficult question. For one, who’s to say what’s right or wrong? Rarely can we find genuine consensus on policy and the best use of scarce resources. We are left with the standard “Churchillian” response that democracy is the least-bad of alternatives.

The author notes, however, that while “the decisions that democracies make may not demonstrate the wisdom of the crowd”, “the decision to make them democratically does”. But this is not necessarily wisdom in the traditional sense of the word. Indeed, democracy works in part not only because our founders and reformers were wise (and that citizens, for the most part, continue to respect our democratic norms and institutions), but because every system before it did not survive.

Ultimately, Surowiecki’s book is a reminder that society is as flexible and knowledgeable as it is foolhardy and dysfunctional. The question then is not so much about the wisdom of crowds, but about the arrangements and incentives that organize us. Institutional history, much like the evolution of science, is a long series of failure, learning, and adaptation. And if you look for diversity, independence, decentralization, and aggregation, you may uncover a thing or two about the group.

Sources

Recent posts