Where Good Ideas Come From - Steven Johnson

Good ideas and the process of innovation

What are the patterns and sources of innovation in society? In Where Good Ideas Come From, author Steven Johnson draws on historical case studies and analogies from biology to tackle this very question. In it, he outlines seven common ingredients in the development of good ideas:

The adjacent possible
Error and selection
Exaptation
Slow hunch
Serendipity
Liquid networks
Platforms

This post will summarise the main lessons that I took from Johnson’s book, focusing briefly on each of his seven themes.

Taking the long zoom

Before diving in, Johnson says that it’s helpful to take a “long zoom” approach to problem-solving. This involves looking at problems from different scales and disciplines. For example, it’s difficult to explain the biodiversity of coral reefs through the study of coral genetics alone. The answer requires us to look also at the interactions between organisms in the ecosystem.

Hourglass scaling

Many patterns, like competition and cooperation, for example, appear on many scales. Johnson likes to think about these dynamics through an “hourglass” model (see below). It helps to keep such parallels in mind when thinking about the sources and patterns of innovation, both in biology and society. The long zoom and hourglass scaling, in my opinion, shares many similarities with Edward Thorp’s thinking models in A Man for All Markets; and Richard Dawkin’s hierarchicial reductionism in The Blind Watchmaker.

	The long zoom
Nature	1. Global evolution 2. Ecosystems 3. Species 4. Brains 5. Cells
Culture	5. Ideas 4. Workspaces 3. Organisations 2. Settlements 1. Information networks

The adjacent possible

The first theme in Johnson’s book is the adjacent possible. It’s a reminder that potential innovations, while vast in possibilities, are bounded by the starting conditions and path trajectories of existing ideas. The adjacent possible describes the “limits and the creative potential of change and innovation”. And as we explore new frontiers, the boundary of possibilities grows.

Sunflowers and incubators

Much of cultural and evolutionary history is the result of “gradual but relentless probing of the adjacent possible”. Johnson illustrates the point with sunflowers as an example. While the atomic constituents of sunflowers have always existed on Earth, sunflowers did not come about spontaneously. It took eons of ‘trial and error’, through mutation and selection, to bring about its current design.

Modern innovations follow a similar process. Johnson describes, for example, how Stephane Tarnier’s invention of the infant incubator — which reduced infant mortality drastically — was inspired by the chicken incubators he saw at the zoo. Both the sunflower and incubator were a result of existing processes that enabled the adjacent possible.

The multiple

Related to the adjacent possible is what Johnson calls “the multiple”. It’s the observation that many break-throughs of recent time have had variants of it “lurking somewhere in its origin story”. For example, Dean Von Kleist and Pieter van Musschenbroek had invented the Leyden jar (early-form capacitors) independently of one another between 1745-46. We find similar stories in the steam engine, the radio and other inventions across history. As Johnson puts it: “Good ideas are not conjured out of thin air; they are built out of a collection of existing parts”.

Error and selection

The process of human invention, much like evolutionary biology, involves a “steady, persistent accumulation of error”. We traverse the phase space of adjacent possibilities, selecting paths that work and eliminating those that don’t. For example, Alexander Fleming’s discovery of penicillin came from an accidental infiltration of mould after leaving his laboratory’s window open. Many other inventions, like the pacemaker, radiography and plastic, came from responses to “generative mistakes”. Whether we’re talking about evolutionary biology or economic development, error, and selection play an important role in innovation and adaptation.

Outliers and inconsistencies

We have a tendency and desire to attribute errors or outliers to experimental quirks or dismissible anomalies. Johnson recalls how astronomers Robert Wilson and Arno Penzias had initially mistaken their discovery of cosmic background radiation for “meaningless static”. It took time and new perspectives for them to realise the importance of their discovery.

In The Character of Physical Law, Richard Feynman also pointed out that many discoveries are the result of accumulating inconsistencies. Great scientists, like Albert Einstein and James Clerk Maxwell, developed their ideas by gathering the known laws and studying their theoretical and/or experimental inconsistencies.

“Two brilliant scientists with great technological acumen stumble across evidence of the universe’s origin—evidence that would ultimately lead to a Nobel Prize for both them—and yet their first reaction is: Our telescope must be broken.”
Steven Johnson, Where Good Ideas Come From, 2010

Mutation, selection and risk

Johnson cites an interesting study from Susan Rosenberg, who found bacteria to increase their rate of mutation under stressful environments. While mutation is risky, the risk-reward ratio under hostile conditions produces the “pressure to innovate”. Bacterium strategy is an interesting parallel to business and warfare, in which hostile competition create an arms-race for innovation and survival.

Exaptation

The third theme is another idea from evolutionary biology: the process of exaptation. Exaptation refers to a trait with some function that evolves to serve another function over time.

Prehistoric birds, for example, may have first evolved feathers for warmth in cold climate. While incremental mutations in feather design are accidental, nature selects for designs with utility. As such, the feathers of their descendants eventually became suited for flight.

Human ingenuity too is full of exaptations. For a quick example, Johnson describes how Sergey Brin and Larry Page exapted Tim Berners-Lee’s hypertext format to develop Google’s PageRank algorithm.

Slow hunch

Great ideas, Johnson says, tends to emerge in a “partial, incomplete form”. This is the slow hunch.

Darwin’s epiphany

Slow hunches are fragile. They require time and connections to develop into full ideas. For example, Howard Gruber’s review of Charles Darwin’s notebooks found that many elements of Darwin’s big idea, like the role of variation and selection in evolution, existed long before Darwin’s collective breakthrough in 1838. In those preceding months, Darwin probably “had the idea of natural selection in his head, but at the same time was incapable of fully thinking it”. Darwin’s achievement, in Johnson’s view, was more of a slow hunch than a sudden epiphany.

Berners-Lee’s web

Another example is Tim Berners-Lee’s development of the World Wide Web. Berner-Lee’s says there was no moment of brilliant inspiration. His gradual realisation came from the arrangement and rearrangement of ideas “in an unconstrained, weblike way”. It was “a process of accretion”, as opposed to “linear solving of one problem after another”.

A commonplace book

Since hunches come and go, the way to cultivate them, in Johnson’s opinion, is simple: “write everything down”. Darwin for example took notes rigorously — quoting others, drawing diagrams and interrogating ideas on paper. And he’d review his notes frequently.

Similarly, John Locke maintained a commonplace book to “facilitate reflexive thought”. This habit of preparing commonplace books can help one to balance the “tension between order and chaos” – between “methodical arrangement… [and] new links of association”. (The Athenarium too is a commonplace book of sorts!)

Johnson summed it up nicely:

“Reading and writing [are] therefore inseparable activities… [By] keeping an account of your readings, you made a book of your own, one stamped with your personality… Each rereading of [your] commonplace book becomes a new kind of revelation. You see the evolutionary paths of all your past hunches: the ones that turned out to be red herrings; the ones that turned out to be too obvious to write; even the ones that turned into entire books.”
Steven Johnson, Where Good Ideas Come From, 2010

Serendipity

Another theme is serendipity, which refers to the “accidental” discovery, connection, or recombination. It refers to the completion of hunches or the finding of an adjacent possible once overlooked.

How do we improve the likelihood of serendipity? Many great thinkers, Johnson says, like to walk and sleep on their problems. For example, scientist Otto Loewi kept paper at his bedside, jotting down notes that came to him from his dreams. Similarly, mathematician Henri Poincare and physicist Richard Feynman liked to go for walks to think about their problems. Bill Gates is a more recent example. He takes annual retreats just to read and think.

John also cites the work of Kevin Dunbar’s How Scientists Really Reason. Dunbar observed that most useful ideas, contrary to popular thinking, did not emerge from isolated instances of brilliance, but from regular laboratory meetings. “Productive analogies” that are conducive to serendipity were more likely to emerge from these informal meetings.

Liquid networks

New and good ideas depend also on social networks. Networks can help individuals and organisations to uncover the adjacent possible and to generate something worthwhile.

The quality of these network, Johnson says, depend on two important preconditions: (1) the network size and density of connections; and (2) the plasticity of the network (i.e., it’s capacity for adaptation and reconfiguration).

“Liquid” networks, to use Johnson’s terminology, are networks that strike the right balance between order (e.g., solids) and disorder (e.g., gases). It helps society to preserve useful ideas, and individuals to transform hunches into complete ideas.

“In thinking about networked innovation this way, I am specifically not talking about a “global brain,” or a “hive mind.” … This is not the wisdom of the crowd, but the wisdom of someone in the crowd. It’s not that the network itself is smart; it’s that the individuals get smarter because they’re connected to the network.”
Steven Johnson, Where Good Ideas Come From, 2010

Primordial innovation

Innovation becomes more likely when we can combine and recombine ideas in mass. That is, to “maximize the combinatorial power”. Johnson analogises this to primordial innovation in biology, where the sheer volume of connections and collisions over an immense time scale led to the remarkable emergence of life.

Conversely, static and isolated networks are less likely to produce an adjacent possible of value. To pursue good ideas, we have to nurture our networks and immerse ourselves in them. Is it no surprise then that universities and technology clusters, for example, tend to spring up here and there?

Search and selection

I’d also add that the quality of our search and selection ‘functions’ for innovation are of great importance. As we’re limited in time and resources, we have to search, select and develop carefully. As noted in Garry Kasparov’s Deep Thinking, chess grandmasters do not consider every possible permutation. Experience, theory and good principles can help decision makers to narrow their focus.

Information spillover

Related to liquid networks are the ideas of positive externalities and information spillovers. Humans like to share, diffuse and replicate good ideas. After all, the success of our civilization is founded on the accumulation and transmission of knowledge over time and generations.

Johnson argues that networks provide better conditions for innovation than organised hierarchies. Networks spread the responsibilities for creativity and decision-making; enable greater connections between individuals and groups; and foster greater information spillover. This is in contrast to trickle-down creativity and information in rigid hierarchies.

Platforms

This brings us to the final theme: platforms. Here, Johnson is thinking about Darwin’s Paradox. That is, to ask why resilient, innovative and diverse ecosystems can arise from nutrient-poor environments. Coral reefs, for example, account for around 25 percent of marine life species, but only 10 percent of our planet’s surface.

Research suggests that these “ecosystem engineers” or “platform builders” (i.e., the polyps and corals) are efficient recyclers of energy and nutrients. They create an environment for interconnected complexity and diversity to thrive. While competition in nutrient-poor environments is rife, the density of “inventive collaborations” enable coral ecosystems to thrive.

We’ve seen innovation platforms emerge in social systems too, from research institutions to the internet and social networks. Believe it or not, the developers of YouTube had originally intended the site as a video-based online dating service. It has since exploded into a platform of incredible diversity. As Johnson puts it, platforms are not just about the flow of information and resources, but about the recycling, transformation and recombination of processes for new and improved functions.

“There are good ideas, and then there are good ideas that make it easier to have other good ideas.”
Steven Johnson, Where Good Ideas Come From, 2010

Apple’s coffeehouse

While many revere Steve Jobs and Jony Ive for their creative genius, Johnson attributes part of Apple’s success in product design to their development cycle. Many companies use an assembly line model for product development. This linear chain, while efficient, is often less conducive to customer-centric innovation.

By contrast, Apple’s model, in Johnson’s view, is more akin to a “coffeehouse model”. It’s “messier and more chaotic”, in which major functions, from design to sales, meet regularly to design their products. According to Johnson, Apple’s internal liquid network and design platform allowed the company to reach the adjacent possible before their competitors could.

Competition and collaboration

There are plenty more lessons from Johnson’s book, but I’ll end the post here for brevity with a final takeaway: that innovation, whether we’re talking about biology or economics, is as much about collaboration and connection as it is about competition. Johnson encourages readers to see the “complex interdependencies of the tangled bank” alongside the “war of nature”, and the “symbiotic connections of an ecosystem” alongside “the survival of the fittest”. Those who oversimplify Darwin’s work tend to focus only on competition.

References

Steven Johnson, Where Good Ideas Come From, 2010. Visit Johnson’s blog at < https://stevenberlinjohnson.com/ >

Where Good Ideas Come From — Steven Johnson on the Natural History of Innovation