The Drunkard’s Walk — Leonard Mlodinow on Randomness & Life

The Drunkard’s Walk — Leonard Mlodinow on Randomness and Reasoning — *“A random walk in two dimensions with two million steps.” (Source: Wikimedia Commons, 2011)*

Motions of randomness

When the nineteenth-century botanist Robert Brown glimpsed at the tiny worlds under his microscope, he saw something peculiar. The grains of pollen that he suspended in water were moving. At first, Brown believed he had chanced upon the essence of life. But that wasn’t the case. Even the inorganic materials, from asbestos to antimony, displayed the same jittery motion when he peered at them.

This phenomena was later modeled by a young Albert Einstein in 1905. He showed how the molecular interactions between water and grain might produce such ‘Brownian’ motion. As the physicist Leonard Mlodinow explains, the water molecules follow a sort of ‘drunkard’s walk’. They “fly first this way, then that, moving in a straight line only until [they are] deflected by an encounter.” “It is only when pure luck occasionally leads to a lopsided preponderance of hits from some particular direction… that a noticeable jiggle occurs.”

Today, scientists “recognize the thumbprints of the drunkard’s walk in virtually all areas of study”, Mlodinow says. They include the “foraging [patterns] of mosquitoes…, the chemistry of nylon…, the motion of free quantum particles,… [and even] the movement of stock prices.” The meteorologist Edward Lorenz showed, likewise, through his study of atmospheric convection, how unpredictable weather patterns can be. Seemingly small and innocuous events can swell together into something consequential. Such randomness, as we’ll see, extends far into society, from wine-tasting to book publishing.

“Much of the order we perceive in nature belies an invisible underlying disorder and hence can be understood only through the rules of randomness. As Einstein wrote, “It is a magnificent feeling to recognize the unity of a complex of phenomena which appear to be things quite apart from the direct visible truth.””
Leonard Mlodinow. (2008). The Drunkard’s Walk.

Our ill-fitted intuition

While we humans are a product of random mutations and cumulative selection, our intuition for probability leaves much to be desired. When Apple added the random-shuffle feature to their music players, many users complained. They claimed the feature was defective because the songs they had just listened to would sometimes repeat itself again. While users clamored for a random shuffle, what they really wanted was a ‘smart shuffle’. They didn’t like the same song playing back-to-back. To meet their demands, Apple had to make the shuffle “less random to make it feel more random” to users.

Similarly and more poignantly, I think, is the “illusion of control” in everyday life. In their famous study, psychologist Ellen Langer and her colleagues asked Yale students to assess their ability to predict coin flips. Strangely, many of them believed that coin-prediction was a skill to be learned. As Mlodinow recounts, “one quarter reported that their performance would be hampered by distraction,” while “forty percent felt that their performance would improve with practice.” What’s more, the students who lucked into an early streak of correct guesses exhibited more confidence in their ability than those who did not. The results showed, of course, that there was no skill involved in guessing. But if people are fooled by the illusory patterns of coin flips, what then of more complex systems?

The conjunction fallacy

In another classic experiment, Daniel Kahneman and Amos Tversky asked students to picture a “31 year old, single, outspoken and very bright” woman named Linda. Linda was a philosophy major and cared deeply about social justice, having participated in antinuclear demonstrations. Following this account, Kahneman and Tversky asked participants, amongst other things, to rank the following statements in order of their likelihood: (1) Linda is active in the feminist movement; (2) Linda is a bank teller and is active in the feminist movement; and (3) Linda is a bank teller.

Surprisingly, most participants ranked the second statement as more likely than the third. Keen readers will know, however, that this violates the law of probability that states that “the probability that two events will both occur can never be greater than the probability that each will occur individually.” Indeed, the likelihood that Linda is both a bank teller and a feminist cannot be greater than Linda being a bank teller. The Linda problem is also known as the conjunction fallacy today. It describes our tendency to conflate satisfactory or coherent explanations with truth and likelihood. While extra details can boost a narrative’s vividness or credibility, it will also subtract from its plausibility. The same is equally true of hyper-detailed prediction models that rely on thousands of assumptions. Forecasters beware.

Mind-filling illusions

In more egregious cases, we substitute reality for what we want to see. Mlodinow points, for example, to table-turning during the early 1850s. This practice had people sit around a table in an attempt to commune with spirits and the dead. Here, the table would serve as a medium, moving and tilting to the fancies of the specter. Many scientists like Michael Faraday took notice of the spookiness. They observed, however, that willing participants were tugging and nudging the table unconsciously. Sometimes, their coaxing would combine and culminate in perceptible movement—feeding immediately into the participant’s hope or expectation of spiritual contact. This was reinforced further by their shared experience of the illusion. Perhaps this can explain in part why séance has almost always been in vogue for much of human history.

As William Calvin reminds in How Brains Think, “what [we] see under normal circumstances owes its stability to a mental model that [we] construct.” Our mind is always guessing, filling, and inferring, trying to make as much as it can from the sparsity it receives. Our imagination will provide shortcuts to some conclusion if that is the only path we’re able or willing to see. And so it is easy to “interpret ambiguous evidence in favor of our ideas”, writes Mlodinow. “The human brain has evolved to be very efficient at pattern recognition. But as the confirmation bias shows, we are focused on finding and confirming patterns rather than minimizing our false conclusions.”

Wine tasters and witch doctors

While table-turning is usually harmless, such conflations extend far and wide into our social and professional lives. Mlodinow takes aim in particular at wine connoisseurs. Referencing a study on the psychology of novice and expert wine talk, he notes that “in 1 out of 3 taste challenges, these wine gurus couldn’t distinguish a pinot noir with, say, ‘an exuberant nose of wild strawberry, luscious blackberry, and raspberry,’ from one with ‘the scent of distinctive dried plums, yellow cherries, and silky cassis.’” Likewise, “when [the experts were] asked to match wines with the descriptions provided by other experts, the subjects were correct only 70 percent of the time.”

Sure, the connoisseurs are better than the novices. But one cannot help but wonder what percentage of wine labels at our liquor stores are incorrectly and undeservedly rated and priced as an inferior make. Perhaps we are being unfair to the wine experts. Many bankers, politicians, executives, economists, consultants, salesmen, psychics, shamans, and witch doctors seem to display the same confidence, authority and expertise that they can’t possibly possess either.

Seuss, Rowling, and Orwell

These problems also seem to manifest themselves in domains like art curation and filmmaking where it is difficult to distinguish between signal and noise, and where judgment is subjective and the environment is noisy. In publishing, for instance, it might surprise you to hear that Theodor Seuss Geisel’s “first children’s book, And to Think that I saw It on Mulberry Street, was rejected by twenty-seven publishers”, Mlodinow notes. J.K. Rowling’s manuscript for Harry Potter, likewise, “was rejected by nine.” Even George Orwell was initially turned down for Animal Farm because publishers believed it “impossible to sell animal stories in the United States.”

Clearly, the people who succeed are the people who persevere. But we are also left to wonder how many superstar authors, artists, athletes, and actors in-waiting were destined to obscurity because they faced just one too many rejections and setbacks. This tells us that the details and contours of history are sensitive to accumulating perturbations. In another worldline, somebody else in the shadows may have taken Rowling, Seuss, and Orwell’s place. Such a world would appear eerily similar yet strangely different.

As Mlodinow writes:

“It is easy to believe that ideas that worked were good ideas,… and that ideas and plans that did not were ill conceived… But ability does not guarantee achievement, nor is achievement proportional to ability… A lot of what happens to us… is as much the result of random factors as the result of skill, preparedness, and hard work… We ought to identify and appreciate the good luck that we have… [and] appreciate the absence of bad luck, the absence of events that might have brought us down, and the absence of the disease, war, famine, and accident that have not—or have not yet—befallen us.”
Leonard Mlodinow. (2008). The Drunkard’s Walk.

Errors, blunders, and outliers

Moreover, it is important, if we can, to pay attention to our rates of error and false positives. We otherwise risk misinterpreting what lies before us. True, some procedures like DNA testing at a crime scene have a less than one-in-a-million chance of matching some randomly selected person. So when a positive match shows up at the place of crime, it is hard to ignore its implications. We should not forget, however, that the likelihood of human error is higher. The Philadelphia City Crime Laboratory, for example, learned this the hard way when it discovered “that it had accidentally switched the reference samples of the defendant and victim in a rape case. The error led the laboratory to issue a report that mistakenly stated that the defendant was a potential contributor of what the analysts took to be “seminal stains” on the victim’s clothing.”

Relatedly is the issue of statistical significance. As Mlodinow explains, “even with data significant at, say, the three percent level, if you test a hundred nonpsychic people for psychic abilities—or a hundred ineffective drugs for their effectiveness—you ought to expect a few people to show up as a psychic or a few ineffective drugs to show up as effective.” That’s not to say, however, that we should do away with statistical studies. It is simply a reminder that the possibility of type I (false-positive) and type II (false-negative) errors are possible. This is especially so when the sample size is small and the experiment is difficult to replicate. As Mlodinow reminds, “sometimes those patterns [and outliers] are meaningful. Sometimes they are not.”

Grecian chance

While these aren’t revolutionary ideas anymore, why are we so bad at probabilistic reasoning? Why do we struggle to distinguish between chance and agency? Perhaps we shouldn’t be so harsh on ourselves. The theory of probability, after all, at least by historical standards, is relatively new. The Ancient Greeks, for instance, were as obsessed with gambling as we are today. While dice were yet to enter Athens, the Greeks played games of chance using bones from animal carcasses. So why did their geometers fail to stumble upon the laws of probability

Indeed, it is easy to forsake just how many building blocks are necessary for us to arrive at such a system. For one, the Ancient Greeks did not possess a number system that made arithmetic easy. They relied at the time on “a kind of alphabetic code”, Mlodinow notes. What’s more, concepts like the number zero did not arrive until the next millennia. And it would take several more centuries until “people came to recognize addition, subtraction, multiplication, and division as the fundamental arithmetic operations.”

Logics and mystics

But even if the ancient scholars had possessed the conceptual tools, they may have rejected its applications on philosophical grounds. “Many Greeks [at the time] believed that the future unfolded according to the will of the gods”, Mlodinow explains. Their scholars and philosophers, likewise, “insisted on absolute truth, proved by logic and axioms.” Socrates noted, for instance, in his dialogue Theaetetus, that any “mathematician who argued from probabilities and likelihoods in geometry would not be worth an ace.”

So perhaps it is not that surprising that humanity had to wait until the sixteenth century to ferment the foundations of probability theory. But even then, many seminal works—like that of Gerolamo Cardano’s Book on Games of Chance—went unpublished and unnoticed for some time. As Mlodinow observes, mathematicians like “Cardono worked at a time when mystical incantation was more valued than mathematical calculation.”

Savant’s problem

It would be wrong, however, to assume that modern society is far improved in its handling of chance today. For example, Marilyn vos Savant was criticized and mocked after publishing her solutions to the Monty-Hall problem and Two-Boys problem. [1] What’s noteworthy is that she was derided not only by scores of general readers but by mathematicians too. Apparently more than 10,000 people wrote to her to explain why she was wrong. A few rude scientists went as far as to call her a ‘goat’. Even the leading mathematician Paul Erdös proclaimed that her solution was “impossible”.

Of course, Savant was not wrong. Yet “when presented with a formal mathematical proof of the correct answer, [Erdös] still didn’t believe it and grew angry.” According to Mlodinow, “only after a colleague arranged for a computer simulation in which Erdös watched hundreds of trials that [demonstrated the result] did Erdös concede he was wrong.” Similarly, Savant had to share thousands of school experiments and a large survey to demonstrate the results empirically to the readers that disagreed with her. (I should also mention that pigeons are actually able to solve the Monty Hall problem. For these winged rats, no simulation or proof is necessary—just good ol’ operant conditioning.)

Ghosts and blunders

Clearly, our experience with and intuition for probability is unrefined. So, unlike the experts that berated Savant, we ought to be more humble in our approach. Like the grains in water under a microscope, each of us are on a drunkard’s walk too. Rarely can we be certain of where we’re going or where we’ll end up. “We should keep in mind”, Mlodinow adds, “that extraordinary events can happen without extraordinary causes. Random events often look like nonrandom events, and in interpreting human affairs we must take care not to confuse the two.” We otherwise risk screaming ‘ghosts’ or ‘the impossible’ when just the opposite is correct. As Martin Gardner famously writes, “in no other branch of mathematics is it so easy for experts to blunder as in probability theory.”

Footnotes

[1] If you’re not familiar with it, the Monty-Hall problem goes as follows: “Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what’s behind the other doors, opens another door, say No. 3, which has a goat. He then says to you, ‘Do you want to pick door No. 2?’ Is it to your advantage to take the switch?”

Sources and further reading

Mlodinow, Leonard. (2008). The Drunkard’s Walk.
Krugman, P. (1995). The Self Organizing Economy.
Thomas, Lewis. (1974). The Lives of a Cell.
Dyson, Freeman. (1988). Infinite in All Directions.