The Voltage Effect — John List on the Five Vital Signs of Scale

AuthorAthenarium / Posted on

The Voltage Effect — John List on the Five Vital Signs of Scale

Voltage effects and vital signs

How do we know whether a start-up or policy has any legs? How do we know if it is scalable? These are important questions because progress depends on the reach of our technologies, policies, and institutions. As behavioral economist John List writes in The Voltage Effect, “all too often, promising ideas collapse at scale.” They experience a “voltage drop”. What we need to investigate, he says, is “the voltage effect”, or the science of scaling. 

Having worked with many organizations, from high schools to ride hailing companies, List says that there are no defining traits to guarantee scaling. But there are roughly five “vital signs” that an idea, company, or policy must pass to have a chance. They are: (1) false positives; (2) audience representativeness; (3) unscalable ingredients; (4) cost traps; and (5) negative spillovers. The prospects for scaling are diminished if one or more of these factors are unmet.

Skip ahead

False positives

The first danger is a false positive. That is to believe your idea or startup is scalable when it is not. When good ideas are rare, and the likelihood of misjudgment is high, false positives are likely. Consider, for instance, as List writes, “that between 94 and 99 percent of burglar-alarm calls turn out to be false alarms, and that false alarms make up between 10 and 20 percent of all calls to police.” Scalability faces a similar challenge.

Indeed, the conclusions we draw from a small-sample study or survey may sometimes be a statistical fluke. It may not be reflective of the target population. Say you’re at the hospital, for example, and your X-ray scan shows something that looks like cancer. Most of us, I suspect, are not jumping immediately to invasive surgery. Your doctor might recommend a second opinion and independent replication of the results before proceeding.

Confirmation and bandwagon bias

In government and entrepreneurship, however, statistical errors are not the only problem that we have to address. Heuristics and behavioral biases can reinforce the false positives we see. Confirmation bias, for instance, describes our tendency to “filter out or ignore information that is inconsistent with our expectations or assumptions.” If we are committed to a passion project or policy proposal, acceptance that it will not fly can be emotionally unpalatable.

The risks of false positives are further amplified when there are bandwagon biases as well. Herding is dangerous, List says, because it “relegates the selection of ideas to a few individuals rather than an entire team of free thinkers.” Society may scale the wrong idea when a charismatic celebrity, politician, or authority latches onto it. Confirmation bias in turn makes it hard for people to change their minds.

The winner’s curse and duper effects

Sometimes, false positives are also a product of the winner’s curse. Consider, for example, an auction for some item of ambiguous value. Here, the winner is most at risk of overpaying by virtue of being the highest bidder. This assumes, of course, that the winner has no unique insight or advantage. And in real life, where things are more complex and unpredictable, such curses are not all surprising. Just ask your friendly venture capitalist. 

In the worst case, false positives are the result of outright fraud. Investors flocked to Elizabeth Holmes and Theranos not for proven science and early results, but for her charisma, narrative, and promise. Social proof and bandwagon effects propelled the company’s valuation to $10 billion. Even the likes of Rupert Murdoch and Larry Ellison wanted in. But as it turned out, Theranos was a false positive of the highest order. 

Audience neglect

Let’s assume then that the idea or startup is not a false positive. The second test is the danger of overestimating your audience. That is to assume incorrectly that your early adopters or sample results are representative of the market you hope to serve. List says it helps at this stage to think like a comedian. “For a comedian to kill it, she has to know her audience. Jokes that bring down the house in one setting… [may] fail in another.”

Kmart learned about representativeness the hard way. When a Kmart store manager in Indiana started the retailer’s first Blue Light Special in 1965, it was a huge success. Customers loved it. So Kmart decided to roll it out everywhere. But in doing so, they made a fatal error. Rather than having each store decide their selection of discounted wares, the decision came indiscriminately from their corporate office. “So when a summer heat wave hit Sarasota or a rainy spell hit Seattle, managers in Florida and Washington no longer had the autonomy to tailor the promotion to better serve their customers”, List writes. While Kmart had a great idea, they butchered its scalability by neglecting the nuances in their customer base. As Sam Walton explains in Made in America, to think like your customers, you have to “think small” and “think one store at a time.”

Representativeness and selection bias

The lesson here is to determine whether or not your sample results or early adopters are representative of the whole. When McDonalds’ launched their luxurious Arch Deluxe, the burger flopped. They forgot that the people who signed up for their focus groups and raved about the burger are not necessarily representative of the average patron.

Similarly, some academic studies are not scalable because researchers frequently “harvest their campus community for study participants”, List reminds. Beer chugging college students may not represent those outside WEIRD (Western, educated, industrialized, rich, and democratic) communities. We must sample and extrapolate with care.

Unscalable ingredients

Not only that, we have to “take into account the representativeness of the situation.” Does the idea or enterprise depend on the chef or the ingredients? This is the difference between Kentucky Fried Chicken and a Michelin starred restaurant in France. World class chefs, after all, do not grow on trees. They face a replication constraint and must strategize accordingly.

Many enterprises, however, fall somewhere in between. Some elements are scalable while others are not. Consider the book publisher Penguin Random House, List suggests. While their distribution network makes it easy to expand to new stores, they are limited ultimately by the quality of their content. The supply of good ideas and writers imposes a constraint.

Cost traps

Related to this is the fourth issue of cost traps. Most of us are familiar today with the concept of scale-economies—that the average cost of production per product falls as sales volumes grow. We have to remember, however, that size alone may not confer advantage. Retailers that expand too quickly, for instance, may find themselves inundated with lacklustre locations and talent. Higher margins prove ephemeral when quality drifts and bureaucracy sets in.

Of course, that’s not always the case. As William Green explains in Richer, Wiser, Happier, companies like Amazon and Costco use scale economies to great effect. The idea is simple: keep prices low to encourage greater spending and word-of-mouth. Growth, in turn, translates into higher revenues and scale savings, which allows for even lower prices and more customers. Through scale economies, a virtuous cycle ensues.

Negative spillovers

This brings us then to the complicated fifth test of spillovers and unintended consequences. The most obvious example, of course, is the global economy. It cannot scale without limit. Our activities produce waste, heat, pollution, degradation and other negative externalities—and the risks may very well be existential if we do not reverse course in time.

Negative spillovers and counterforces manifest themselves in many micro settings as well. List recalls, for example, an education program that sought to improve child performance via parental engagement. While the program was successful in lifting outcomes, they neglected the fact that for parents to participate, time had to be taken away from other siblings.

That’s not to say the program was right or wrong. It may have generated real returns to society on a net basis. But we risk overstating the benefits when we do not take the spillovers and unintended consequences into account. And the issue can be costly if the program is rolled out nationwide without a full accounting of direct and indirect costs.

Similarly, List recalls how California had tried to lift student outcomes in the 1990s by reducing classroom sizes. The initiative makes sense in theory. Smaller classrooms means each student gets more attention from their teachers. That’s a no-brainer. How could it go wrong? 

What they neglected was the shortage of high quality teachers that would ensue. And if a school is forced to hire unenthusiastic, ill-fitting educators to fill a role, what’s the point? As List laments, while the “initial small-scale results were promising…, the benefits [at scale] turned into vapor.”

General equilibrium effects

That’s not all. Scaling requires us to look for “general equilibrium effects” that may undo everything we are trying to achieve. List recalls, for instance, when he worked with Uber as their chief economist on a project to improve driver earnings. The obvious approach, of course, is to raise base fare rates. Easy peasy, right? 

Earnings did improve for a few weeks. But as more people learned about the higher fare rates, they too opted in to become Uber drivers. And as the pool of Uber drivers expanded, it became harder for each driver to secure passengers for trips. The gains in pay rates were offset by increases in competition.

A similar issue occurs when managers in competitive markets believe they can get a leg up by lowering prices. They forget that competitors can respond with the exact same thing. Suddenly, everyone is back at square one with lower prices to boot. The point is that when it comes to scaling, we have to step through the consequences.

Peltzman effects

Sometimes, however, the spillovers and unintended consequences are counterintuitive. List points in particular to the Peltzman effect. The theory on risk suggests “that we make different choices depending on how secure we feel in any given situation.” The effect was named after economist Sam Peltzman, who observed “that auto safety regulation [had] not affected the highway death rate.” As List explains, “drivers felt safer because of the legislated measures… So they took more risks while driving, and in turn had more accidents”—undermining the intent of the legislators. 

The Peltzman effect presents scaling troubles in many industries. For one, moral hazards in insurance and financial markets, remains a thorny issue. When people do not have skin in the game, distorted behaviors arise. The subprime crisis reminds us how bad products and incentives at scale can threaten the entire house of cards.

What else?

In the opening of Anna Karenina, Leo Tolstoy writes that “happy families are all alike; every unhappy family is unhappy in its own way.” The same is true of scaling, List writes. “Scaling is, in the end, a weakest-link problem.” Any one of these five tests, from false positives to negative spillovers, are enough to bring an idea or enterprise down. 

The Voltage Effect, however, is curiously light on the interplay between scale and competition. In Competition Demystified, Bruce Greenwald emphasizes, for example, that it is not the absolute but relative scale economies that matter. So if you’re a mouse among elephants, scaling will be a challenge—especially if you have nothing to differentiate yourself with.

List also does not discuss the scaling phenomena that researchers observe in the natural and social world. Geoffrey West and colleagues show, for instance, how network effects and increasing returns can explain the superlinear scaling effects of income and innovation that seem to emerge in larger urban city populations.

So when it comes to scaling, competition and macrostructures matter too. Readers would do well not to ignore them. What The Voltage Effect provides, however, is a checklist that entrepreneurs and policymakers can use to investigate the inner scalability and constraints of their ideas. It serves not as a complete toolkit but a starting point for scrutiny. And in a later post, we will discuss some scaling strategies that List believes are worth considering.

Sources and further reading

Latest posts