4

Innovation at the Technological Frontier: Three Policy Icons and a Common Approach to Uncertainty

We have argued that deep decarbonization depends on innovation at the technological frontier. In this chapter, we look at three iconic cases of successful frontier innovation. We first examine the success of the California Air Resources Board (CARB) in orchestrating deep reductions in transportation emissions by tightening standards for combustion engines—a success that laid a foundation for the state’s zero-emissions vehicle mandate of the 1990s. Second, we consider the development of scrubbers, a technology for controlling SO_2, the leading cause of acid rain and other environmental perils, in the context of a pioneering cap-and-trade system of pollution permits. Last, we discuss radical clean energy innovation through the Advanced Research Projects Agency (ARPA-E) of the US Department of Energy (DOE).

These examples cover the gamut of policy strategies that have also been proposed as approaches to the climate crisis. The case of ARPA-E is typically taken to illustrate the view that markets are too preoccupied with short-term calculations to invest in radical innovation; for that the state must step in. The case of scrubber development is often taken to point in the opposite direction: the solution to complex environmental problems is beyond the reach of government bureaucracy; only markets are fit for the task. The success of CARB is taken to confirm a middle ground position: empowered by the ability to threaten the closure of a large (state) market, the descendant of the New Deal administrative state can set rules to force change.

There is something to say for each of these strategies as a way of grappling with market failures. But we contend that when it comes to the mechanics of any of these approaches—how to ensure that the state picks truly promising research areas and projects; how to identify feasible goals for technology-forcing programs and correct errors when they occur; and how to create the right background conditions for the market, ranging from trading rules to technological innovations that the market will not produce itself—experimentalist governance is indispensable as a means of addressing the uncertainties that frustrate conventional approaches. To ignore this need for experimentalism is to run the risk of repeating the same errors that turned the promising Montreal model into the failed Montreal mold.

All three of the cases we examine are from the United States. This is not for want of instances from elsewhere; there are plenty in the literature, and chapter 5 discusses some of them. But even those who are most open to programmatic innovation might cast a dubious eye on the relevance of foreign examples. Could it really happen here? What about the US reflex—sometimes justified, and sometimes not—to see collaboration with industry as regulatory capture rather than a path to problem-solving? And what about the baroque and ossified formalism of US administrative law that allows—and perhaps even encourages—reluctant firms to challenge new standards in court with good chances of success? With three powerful homegrown examples, we show that these objections can’t be as robust as frequently assumed—just as the examples in chapter 3, where we outlined the theory of experimentalism, demonstrate that this mode of governance is highly effective under the right conditions. The space for innovative collaboration and experimentalist governance is a lot greater than assumed, even in the United States.

What Was Miraculous in the California Miracle?

Our first case begins with smog, an unhealthy urban pollutant primarily the result of combustion emissions from motor vehicles and industrial operations. Since the 1960s, California—whose cities had (and still have) some of the worst air quality in the nation—has pioneered successful regulatory efforts to limit the emissions of smog precursors and other important vehicular pollutants. Because of the state’s trailblazer status, the 1970 Clean Air Act allows California, alone among the states, to set its own emissions standards, provided they are at least as protective as the national ones; under the statute, other states may (and in fact many states do) adopt the (unaltered) California standard.

By these mechanisms—demonstrable leadership in its home market and easy followership by others—California has set the pace, especially in recent decades, for the reduction of vehicular emissions in the nation as a whole. Smog levels in California have dropped by 60 percent since the 1960s, even though the number of cars on the road has doubled.¹ Today, thirteen other states plus the District of Columbia have adopted California’s standards; about a quarter of all the vehicle miles traveled in the nation occur in states that follow California’s rules.²

CARB, which has regulatory authority for protecting the quality of the state’s air, is governed by a board of twelve members, appointed by the governor, plus two legislators (one each from the state’s assembly and senate). By statute, its board must include at least one member with experience in automotive engineering, one with training in chemistry or meteorology, one with expertise in medicine or health, one with experience in air pollution, and two representatives of the general public. Each of the remaining six members represents one of the state’s air pollution control districts.³ By design, CARB thus reflects views from a variety of stakeholders—informed by technical expertise—and creates a forum where lessons learned in one control district can be rapidly transferred to others.

Here we focus on one particularly iconic period in CARB’s history: the effort in the 1990s to create a market for zero-emissions vehicles. CARB took that goal to entail the development of electric vehicles, but it soon learned that the technology was not ready at the time. How it responded to that reality reveals why CARB works so well in its core regulatory mission: catalyzing and steering efforts to clean up California’s air. We tend to think of electric vehicles as an innovation of today rather than three decades ago—in part because electric vehicle technologies (notably batteries) are presently so much more advanced—yet it was CARB’s 1990s’ regulatory innovations that redefined ultraclean vehicle requirements and set the stage not just for the current revolution but also for extraordinary improvements in pollution control for internal combustion engines. In effect, CARB confronted exactly the same challenge that the Montreal Protocol parties faced when they sought to cut ODS to zero. They knew the ambitious goals they wanted to achieve, and among at least some of the parties, there were powerful incentives for success. But nobody knew how to get there.

In 1990, partly in response to amendments to the Clean Air Act in that year, CARB introduced the low-emission vehicle (LEV) and zero-emission vehicle (ZEV) programs. Together they established the nation’s most stringent standards for the reduction of exhaust emissions from gasoline and diesel vehicles of smog precursors, such as nitrogen oxide (NO_x), nonmethane organic gas, and carbon monoxide.⁴ The standards in both programs were formally amended a number of times (the LEV standards in 1999 and 2012, for example) and continuously revised between formal amendments. The Clean Air Act set extreme punishments for states that failed to clean their air—at the limit, the federal EPA could impose onerous restrictions on economic activity—but those vague punishments were conditional on effort. States had to show, through State Implementation Plans, that they were making progress; the dirtiest states had to demonstrate the most effort. Moreover, inside California, state political leaders were aligned around cleaning the air and faced electoral punishment if they failed to make efforts. Penalty defaults were numerous.

Both of CARB’s vehicle programs induced and codified major changes in technology and emissions. The LEV program’s ambitious standards drastically reduced smog emissions by inducing incremental innovations to the gasoline- or diesel-burning internal combustion engine that have had, over time, cumulatively radical effects. A new car sold today in California emits 99 percent fewer pollutants than a new car in the 1960s. The ZEV program—designed to encourage volume production of cars with “no exhaust of evaporative emissions of any regulated pollutant”—helped induce the development of affordable batteries that could meet range and chargeability standards akin to conventional vehicles as well as appeal to consumers—an extraordinarily ambitious goal given the existing technology.⁵

By the standard wisdom, LEVs and ZEVs were classic examples of what CARB does best: identifying a serious problem and moving to fix it through the imposition of strict rules that force changes in industry. It uses science to set goals and then forces industry to align. It demands the seemingly impossible of industry, but because industry needs the California market—as well as other markets that often follow California’s lead—it finds a way to make the impossible possible. Having done that, other jurisdictions follow the miracle—in short, the “California effect.”⁶

But this characterization is misleadingly incomplete. Access to a crucial market mattered to automakers, to be sure. The threat of exclusion created a powerful penalty default—imposing a price for the failure to cooperate. Yet by themselves, these incentives would no more have guided actors to workable solutions than the incentives created by commitments and sanctions under the Montreal Protocol would have yielded results without the guidance provided by the TOCs. California’s success hinged on the ability of regulators to couple regulatory standards to the ongoing evaluation of critical and rapidly changing technologies, such as pollution control equipment and electric vehicles. In other words, penalty defaults and strict standards are necessary but not sufficient conditions; to work, they need to be applied against a background of continuous adjustment and review. In practice, the LEV and ZEV programs succeeded because, as a leading commentator put it, the regulator was “committed to a continuous process of implementation oversight and regulatory review,” and was correspondingly willing “to adjust and change the basic program.”⁷ When one looks closely at the history, the outcome of this collaboration between regulators and industry was extremely different from what CARB imagined would be the case in 1990. As with essentially all complex regulatory programs, the more ambitious the effort, the harder it is to know ex ante what is achievable.

In this crucial function—linking standard setting to the continuing investigation of technological development—CARB obtained its broad authority by the feasibility requirement in the Clean Air Act’s Section 202(a), which specifies that the EPA and other regulators must demonstrate that available technologies provide cost-effective ways of meeting any proposed standard. A line of court decisions, including International Harvester v. Ruckelshaus and especially NRDC v. EPA, suggested that it was the long lead-in times for new requirements that assured feasibility because they made it possible to correct unworkable rules.⁸ “The time element in the EPA’s prediction,” the court in NRDC acknowledged, “introduces uncertainties in the agency’s judgment that render the judgment subject to attack. [But] the presence of substantial lead time for development before manufacturers will have to commit themselves to mass production of a chosen prototype gives the agency greater leeway to modify its standards if the actual future course of technology diverges from expectation.”⁹

The need to link standard setting to active engagement with technological development was also reinforced by the widespread recognition within CARB, from the outset, of the uncertainty of the overall situation and hence the need to learn rapidly to stay abreast of possibilities. A December 1995 briefing to CARB, Making ZEV Policy Despite Uncertainty by the RAND Institute for Civil Justice, put this directly: “Enormous uncertainty and risk suggest a search for near-term policies that, a) enable learning, b) are not susceptible to disaster, and c) can be tailored as new information is obtained.”¹⁰ Learning and the corollary policy updating were central to those within CARB from the early days of the LEV and ZEV mandates.

Another important element of effective collaboration between the regulator and regulated entity was greatly facilitated by deep changes in the organization of the automobile industry. Until the 1980s, the major automobile companies were vertically integrated. They developed and produced key components in-house in order to avoid the risk that a powerful supplier could withhold delivery of a crucial component unless the supply contracted was renegotiated in its favor. But as markets became more volatile and the direction of technological development more uncertain, the risks of owning captive suppliers increased dramatically; a shift in the trajectory of development could make in-house capacity irrelevant, while a shift in the level or composition of demand could make it superfluous. Beginning in the 1990s, automakers divested internal suppliers, and purchased more and more components and subsystems from independent manufacturers. Technological innovation also shifted—away from the automakers themselves, and toward coproduction between the supplier and buyer.

An unintended side effect of this restructuring was to shift the balance of power in relations between the regulator and firms in the industry. Vertical disintegration created important new actors, primarily equipment suppliers, with interests in reducing the levels of polluting emissions, even if this improvement was in tension with the final assemblers’ goal of minimizing the overall costs. These interests arose because emissions requirements meant greater demand for vehicle components and thus more opportunity for suppliers—especially for suppliers with the greatest capacity to connect innovation to products. Thus component manufacturers and suppliers frequently approached CARB to pitch new emissions control technologies. By demonstrating the superior performance of their products, they influenced emergent standards and created markets for their innovations.¹¹

This reconstitution of interests reduced the information asymmetry that traditionally advantages producers in negotiations with relatively ignorant regulators. Indeed, suppliers and component manufacturers developed many of the newer emissions control technologies, such as turbochargers and electric superchargers. The first LEV standards, in the late 1990s, were shaped by the demonstration of an electrically heated catalyst that drastically reduced emissions.¹² Over this period, CARB did not conduct its feasibility assessments in a vacuum; rather, the assessments were more and more the result of near-constant conversations with both component manufacturers and automakers. Since virtually all the automakers belonged to trade associations that lobby regulators and raise the specter of oligopolistic coordination across the industry, CARB emphasized communications with the automakers themselves, and it regularly recruited experienced engineers from the auto industry to build its capacity to engage in detailed technical dialogue with component and carmakers.¹³

This dynamic atmosphere of innovation and oversight produced standards that were technology forcing without being technology determining. Continuing exchanges between firms and regulators set performance standards with an eye to what was feasible, but those standards did not mandate the use of particular technologies to achieve them. Once the feasibility of a certain reduction in emissions had been demonstrated, each firm was free to find its lowest-cost path to the required result that met a particular standard—for example, the quantity of volatile organic compounds emitted per volume of fuel consumed or mile driven. Indeed, automakers usually devised their own means for meeting new standards. For instance, although CARB staff projected “that gasoline vehicles meeting the more stringent [LEV] standards would require the use of emerging new technologies such as electrically-heated catalysts (EHCs) and heated fuel preparation systems,” only one manufacturer ever used an electrically heated catalyst; others used (and further developed) their expertise in systems integration to realize “improvements in combustion control in the engine itself to reduce engine-out emissions.”¹⁴ The real contribution of the joint exploration of new technological possibilities, in other words, was to demonstrate that daunting, technology-forcing standards could be met—not to determine, and still less to impose, required solutions.¹⁵

The frequency and scope of these technological inquiries varies according to circumstance. The general rule is that the more uncertain the technological foundation of a proposed standard, the more extensive and frequent the review—and the more interdependent the technological changes, the more frequent the review of the whole system’s potential performance. CARB subscribed to this philosophy, and other institutions that help orchestrate innovation at the frontier do the same—from the technical assessment panels of Montreal to ARPA-E. When CARB initially proposed amendments to the first LEV standards in 1996, the agency produced a staff report that identified “four basic aspects of current emission control systems that vehicle manufacturers have been improving to achieve low-emissions levels, [namely] more precise fuel control, better fuel automation and delivery, improved catalytic converter performance and reduced base engine-out emissions levels.” The report listed nineteen potential low-emission technologies that would become available when the new standards took effect, and predicted which technologies manufacturers would use to meet the new requirements.¹⁶

As an illustration of this continuous, joint assessment, consider CARB’s biennial review of the ongoing technological feasibility of the ZEV program targets.¹⁷ These reviews, and the related assessments to which they gave rise, proved essential to the effective pursuit of the ZEV mandate despite the uncertain technological developments on which it depended.

The first review, conducted in 1992 under the pressure of various procedural deadlines, was limited to perfunctory approval of the initial ZEV requirements.¹⁸ CARB staff members continued consultations with emission control suppliers and the volume vehicle producers, conducted their own tests of components and prototype vehicles, and reviewed the findings of the United States Advanced Battery Consortium, a collaborative research project supported by the large auto manufacturers.

At the second biennial review, CARB again found that the LEV and ZEV requirements were feasible and cost-effective but extended the review process in two ways.¹⁹ First, given the centrality of battery development to the ZEV program, CARB established an independent Battery Technology Advisory Panel to assess candidate technologies through visits and follow-on discussions with leading developers of advanced batteries and their customers.²⁰ Second, in response to questions raised during the review, CARB staff members organized a series of public workshops and other forums, from May to November 1995, in which interested parties ranging from electric utilities to environmental groups and auto manufacturers discussed key ZEV issues, such as electric vehicle infrastructure and the marketability of ZEVs. In the course of these discussions, proposals to modify the ZEV mandate were introduced, and CARB convened a public meeting to consider these and other possible modifications in anticipation of the third biennial review the following year.²¹

On the basis of these consultations and a review by the Battery Technology Advisory Panel, CARB concluded that ZEV technology would not be available in time to meet the 1998 requirements.²² Battery prices had not fallen as quickly as expected; the industry feared that the premature introduction of ZEV models at high prices and their potentially unreliable long-term performance would produce consumer resistance, thereby complicating the future sales of ZEVs.²³ Acknowledging these concerns, CARB removed the ZEV requirements for the 1998 through 2002 model years, but left the mandate of a 10 percent share of ZEVs in the 2003 fleet in place.²⁴ It did not give up the goal completely, on the theory that the underlying technological conditions that made prompt change impractical might shift in the future. (For reference, only in 2019 did electric vehicles of all types account for nearly one-tenth of new vehicles in California.)²⁵

In exchange for this relaxation of the rules, CARB entered into a memorandum of agreement with each of the large-volume manufacturers. These memorandums bound the manufacturers to continue developing low-emission cars; produce cleaner cars nationwide and participate in advanced technology development and demonstration projects; share propriety development information with CARB; and commit substantial funds to support the United States Advanced Battery Consortium’s research. As under the original LEV program, noncompliance was sanctioned by substantial damages.²⁶ This approach to relaxing one set of rules while keeping others in place helps illustrate how penalty defaults operate best—within an environment of constant adjustment and review. The automakers were failing to meet ZEV rules through no intended fault of their own, but they were doing better at meeting LEV rules than expected. The regulator, suitably informed through peer review, fine-tuned the location of the goalposts and penalty zones.

The 1998 and 2000 biennial reviews recognized that battery development for ZEVs continued to disappoint. Yet LEVs were developing faster than expected. As a result, CARB created new vehicle categories—the partial-zero-emission vehicle (PZEV) in 1998 and advanced technology PZEV. Cars that met the criteria for these new categories—mainly hybrids and plug-in hybrids—were less polluting than any full-scale internal combustion engine vehicles that previously had been qualified to use the state’s roadways. CARB therefore allowed manufacturers to meet their ZEV requirements by, for example, counting five PZEVs as equivalent to one ZEV. It formalized such substitutions as the Alternative Compliance Path in the 2003 amendments to the ZEV mandate.²⁷

As experience unfolded, CARB kept adjusting the targets so that the regulatory requirements and understanding of technical feasibility changed in tandem. In the 2007 biennial review, CARB delayed increases in the fuel cell requirement, judging the technology insufficiently mature for application. In time, battery technology and control systems caught up with CARB’s ambitions—partly because of large and rapidly growing sales of lithium-based batteries for consumer electronics. By 2011, there were seventeen thousand electric car sales, and that number tripled in 2012 to fifty-two thousand. In 2013, there were sixteen ZEV models available from eight auto manufacturers—nine of them purely battery operated. Currently nearly every major automaker has a ZEV for sale in California, and some—for instance, GM—have decided to shift decisively away from internal combustion toward electricity.

As CARB repeatedly shifted rules to reflect its assessment of feasibility, many of the automakers challenged its authority to adopt technology-forcing requirements that they thought infeasible. CARB successfully defended itself by showing that its administrative process engaged the industry and remained aligned with the latest technical information.²⁸ Far from freezing development, uncertainty about whether the automakers would prevail in their efforts to steer the content of the regulation was a prod to innovation; neither individually nor in a consortium could firms run the risk of halting their own research, while at least some competitors were likely to plunge ahead. (And those that plunged in the wrong direction soon found themselves punished. BMW, for example, pushed hard with fuel cells while CARB interpreted ZEV rules in electric terms. No market for BMW’s hydrogen fuel cell cars ever emerged in California.)

This continuing innovation generated new ideas and technologies, which in turn allowed CARB to keep tightening performance standards as well as better defend itself against legal challenges that questioned its authority and technical competence.²⁹ Similar challenges have been raised about CARB’s authority in other domains—such as in the regulation of carbon emissions that affect interstate commerce—and CARB defended itself with a similar playbook that it honed in the 1990s and 2000s: arguments about grounding its work in impartial, technical competence backed by transparent peer review and within the scope of regulatory deference.³⁰

As technology has improved over the last decade—this time unexpectedly to favor electric vehicles even as some of the original equipment manufacturers and suppliers expected fuel cells to excel—CARB has changed the rules again. In January 2012, CARB approved a new emissions control program for model year vehicles 2017 to 2025—a new scheme called the Advanced Clean Cars Program that mandates more ZEVs with greater controls of the pollutants that are precursors to smog, including soot, and focuses on cutting emissions of greenhouse gases. The new scheme also includes an overcompliance rule that allows manufacturers who overcomply with some rules to earn credits that can offset a portion of their ZEV requirement from 2018 through 2021. For regulators, this approach not only offers flexibility to manufacturers—a key demand from some in the industry—but also creates incentives for more advanced experimentation that can be used by CARB to reveal places where future rules can be tightened. Supernormal gains from doing better turn the penalty default on its head.

Comparing today with 1990, the ZEV mandate predictably did not operate exactly as originally envisioned. Uncertainty was too profound in 1990 for anyone to know how the system could, would, or should unfold. Precisely because it kept responding to new and often unexpected developments—some even exogenous to the whole auto industry, such as lithium batteries—the program achieved its goals of radical reductions in harmful air pollutants. At first these improvements came mainly through making internal combustion much cleaner; later they relied on hybrids; in time, full electric vehicles have come to play a more central role. Along the way they underwrote the explosive growth of hybrid electric vehicles in the early 2000s.³¹ In turn, the development of hybrids has translated back into the substantial development of pure battery electric vehicles in recent years. As of 2010, California had 2.2 million electric vehicles on the road, 80.8 percent of which are PZEVs (mostly hybrids), 17.6 percent are advanced technology PZEVs, and 1.6 percent are ZEVs.³² And today, those shares are shifting quickly in the direction of full electric models.³³

From the experimentalist perspective, the ZEV and LEV programs show how rules coevolve between the regulator and supplier. Specific targets for a reduction in ozone precursors and other air pollutants set an initial, high-level goal. CARB served as a central coordinating body for an iterative learning process structured around biennial reviews and many other forums. On the basis of pooled reports discussing extensive field experience implementing the regulatory mandates, CARB periodically—and sometimes substantially—adjusted its requirements. It set rules that forced massive investment in new technology—and competition among suppliers—but did not pick winners. A penalty default—the threat that uncooperative behavior could ultimately result in exclusion from a leading market as well as civil penalties for failure to meet the program’s substantive goal—existed at the outset. All of these features, not just any one of them, were essential to make the program a success in pushing forward the technological frontier.

The Misunderstood Miracle of Sulfur Control: Before and after the US Sulfur Emission Trading Program

SO₂, often just called sulfur, is one of the most noxious air pollutants. When the federal government adopted the 1970 Clean Air Act, the first sustained nationwide effort to control air pollution, it was already known that SO₂ was a major source of respiratory disease, an aggravation for asthma, and a cause of numerous other ailments, including in children. SO₂, it would be learned by the 1980s, was also the chief cause of acid rain, which killed forests and other fragile ecosystems. The Clean Air Act created a list of National Ambient Air Quality Standards (NAAQS), which named SO₂ as one of the inaugural six pollutants to be regulated. The EPA, also created in 1970, was charged with keeping the NAAQS up to date and setting standards for “clean air” by looking to the science of public health, without consideration of cost.

At the time, most SO₂ came from coal-fired power plants. The United States has always burned a lot of coal for power, but when the first oil crisis hit, the country increased its dependence on coal, nearly all of which came from US mines; it shut plants fired with costly and insecure oil, and built more coal plants to keep up with the rising demand for electricity. By the 1980s, coal accounted for about 60 percent of all domestically produced electricity. Thus the regulation of SO₂ implicated a well-organized, important, and politically powerful industry: electricity. Controlling sulfur would require accommodating powerful economic interests.

Five decades later, US emissions of SO₂ have fallen tenfold, from 31 million tons annually to about 2.8 million tons today. In the conventional account, this tremendous success was due to the timely recognition that government was not skilled at administrative intervention—or the command-and-control prescription of technology, as it was and still is called. Though it was armed with the NAAQS, a mandate to clear the air, and the authority to impose technology standards under the Clean Air Act, the EPA nonetheless failed to achieve satisfactory results from the 1970s through the 1980s. Those failures had become apparent by the 1990s, when Congress, at the urging of powerful environmental groups and business interests worried about cost, authorized creating a national trading scheme with amendments to the Clean Air Act.

It took five years to get the new market up and running, but once in operation it worked like a charm. Prices for sulfur permits were less than half the level that had been expected. The conventional story attributes this drop to the power of market forces in finding inexpensive solutions that administrative action could not.³⁴ This success only further entrenched the celebration of markets as heirs to failed bureaucracies such as the Soviet State Planning Committee, known as Gosplan. A retrospective by one of the most active NGOs, the Environmental Defense Fund, is titled “How Economics Solved Acid Rain.”³⁵ Writing in 2002, the Economist, looking back at the history, called the sulfur market “the greatest green success story of the past decade.”³⁶ (The 1990s also included the ramp up of the Montreal Protocol, creation of the UNFCCC, signing of the Kyoto Protocol, and Rio Earth Summit—so the pronouncement was a bold one.)

The story we tell here differs dramatically from this standard account. Administrative action was working quite well in forcing needed technological innovation. Without new technology, it would be impossible to make deep cuts in emissions. The breakthrough came with scrubbers—devices that chemically remove sulfur from flue gases before being pulled up a smokestack. The technology was developed and tested from the early 1970s through the 1980s, long before the sulfur market existed. That innovation emerged through collaboration between industry (led by the largest and most motivated firms), the EPA, equipment vendors, and a research institute created by the utility industry known as the Electric Power Research Institute (EPRI). That collaboration mirrored the patterns we observe in CARB. Hardly an effort in heavy-handed command and control, it was a collaborative system that ran multiple experiments in parallel, constantly adjusted means and ends, and chose technologies based on performance.

Ironically, regulation was coming under sustained attack by many US political and business elites just as this experimentalist approach was realizing its most profound benefits of innovation and dynamic efficiency; only by the later 1980s was it clear which scrubber systems would perform best, and whether it would be possible to operate those systems at high reliability and rising efficiency. The sulfur market “worked” in part because a regulatory and innovation system that operated according to experimentalist principles had already supplied the critical innovations that were diffusing into service as more power plants learned which scrubbers to install and how to operate them. When the sulfur market began operations around 1995, it helped allocate resources efficiently in the short term across known technologies—a process often called static efficiency. Even in that respect, however, its contribution was more limited than it appears.

Another major contributor to lower-than-expected sulfur prices was a sharp increase in the availability of less polluting sources of coal—an option that emerged from fortuitous changes in other bodies of regulation not tied to the trading system. Sulfur prices were low, relative to expectations, not because markets were superior to bureaucracy in finding dynamic solutions. Instead, unexpectedly low prices appeared because no analyst at the time anticipated how all of these technological and coal market changes would interact to make pollution control easier than expected.

Although today the sulfur trading system has a hallowed place in the story of the triumph of markets over administration, in fact the period of market success—when prices were lower than expected and stable—lasted only about eight years. So long as there was little need for major technological innovations, and no need to make plant- and location-specific choices, the nationwide market helped achieve modest gains. But this approach failed to achieve the mission of the Clean Air Act: clean air everywhere. A national market allowed pollution hot spots to emerge; places where it was cheap to pollute attracted more polluters.

At the same time, new science identified new pollutants and new local interactions between pollutants, which meant that effective pollution control would require addressing multiple pollutants from multiple sources simultaneously in particular places. Managing these complex interactions required more “administration” and the construction of custom-built, narrow “markets”—ones responsive to local hot spots, but by their nature unable to realize the efficiency gains of nationwide trading. These problems linger because their solutions demand more active regulatory collaboration, not less government. The market’s eight good years show what trading can do, but also—especially in the case of severe environmental harms—what it cannot.

MAKING SCRUBBERS WORK

In theory, the logic of the Clean Air Act was simple. It mandated that the EPA set tough, science-driven NAAQS—standards that every locale would be required to meet, ideally prior to the nation’s bicentennial celebration in 1976. Each state that had places in so-called nonattainment would be required to adopt a state implementation plan, which the EPA would review and approve.

Controlling pollution from processes deeply embedded in the modern industrial economy would prove to be a lot like today’s problem of carbon dioxide, of course: extremely difficult and not amenable to ambitious solutions on a rapid schedule. Indeed, the dirtiest places didn’t clear their air until well after 1976, and many are still in nonattainment today. But the Clean Air Act did gave the EPA a powerful weapon: it could reject a state implementation plan and dictate industrial policy in nonattaining areas. In reality, that weapon was politically and practically hard to wield, and wound up being reserved for egregious cases. Thus began a long negotiation that continues today between the EPA and the state regulators on which it must depend for the frontline work. While state regulators set local rules and checked progress, the EPA set goals and threatened penalties for lack of effort. In the words of William Ruckelshaus, the revered first head of the EPA who served as head again during part of the Reagan administration, federal weapons were essential to effective state action: “Unless they have a gorilla in the closet, they can’t do the job. And the gorilla is [the] EPA.”³⁷

Scrubbers became an essential component of sulfur reduction only as other, apparently less costly control strategies proved infeasible. One strategy was to use less sulfurous coals. While all coal contains sulfur, its exact sulfur content varies widely—from less than 1 percent to about 10 percent. By far, the greatest concentrations of coal-fired US power plants were in the Midwest and South. Local coals there were generally high in sulfur. More distant coals were thought to have lower sulfur, but moving coal long distances in the 1970s was prohibitively expensive because of tightly controlled railroad tariffs. Local coals, especially at midwestern and eastern mines, also benefited from regulatory protection.

Beyond these factors, horrific mining practices—later pilloried as “mountaintop removal”—were tolerated at the time, and mining companies had few responsibilities to protect the local landscape. Nor were they required to post bonds for the restoration of sites when they were done. With no serious demand for low-sulfur western coal, there was no supply. The federal government owned vast mineral rights for western coals, but essentially none of that resource was leased until the 1980s, when the Reagan administration put politicians who favored the West in charge at the Department of Interior and a gold rush of western coal leasing began.³⁸ In the early 1970s, when minds were first focused on the problem of cutting sulfur from coal-fired power plants, the option of coal switching was not available at scale.

A second option was to clean the sulfur from the coal before combustion. That was tried and soon shown to be impractical as well. Also impractical at the time was to gasify the coal, which might have made it easier to remove gaseous sulfur before combustion; some demonstration programs dismissed that idea quickly. (Today, gasification is much better understood, thanks partly to a lot of innovation in coal-rich China, where gasification is a way to use coal as a feedstock for making chemicals. Coal gasification might be practical in the United States too, if the cost of natural gas had not plummeted when fracking and horizontal drilling unleashed massive new supplies, and utilities stopped building new coal plants.)

The impracticality of these two options meant that power plants and their utility owners had to face the reality that large amounts of SO₂ would be released from coal combustion. Compliance with NAAQS, which set limits on local concentrations of air pollutants, required breaking the link between sulfur released during combustion and what the local community experienced as sulfur pollution.

The third option was simply to build taller stacks, thus embodying the age-old maxim that the “solution to pollution is dilution.” This approach would break the link between sulfur released from coal combustion and local concentrations of sulfur in the air, where it would cause harm to the local populations, by getting more of the sulfur to leave the immediate areas. Industry instantly calculated that taller stacks would lower local pollution, and they responded by retrofitting and building new, taller stacks. It was later learned that this practice actually worsened the problem of acid rain, which was the result of sulfur (and other pollutants) traveling days downwind and converting chemically into acids while aloft.

In the 1970s, acid rain was still a matter of scientific debate, not a top national priority, and unrelated to the immediate problem of compliance with NAAQS that were written entirely in terms of local air pollution. Nonetheless, environmentalists were alarmed by stack rising, which they saw as a ploy to avoid regulating pollution, and organized quickly to block the practice as an abdication of responsibility. In the 1977 amendments to the Clean Air Act, rules supported by the increasingly powerful environmental community barred taller stacks unless plants installed modern pollution control equipment too. Those amendments also included a scheme known as the “prevention of significant deterioration” that made sure that even the clean areas wouldn’t get a commercial advantage by getting dirtier. It put industry on notice that new and existing power plants, even in cleaner parts of the country, would need to install pollution control equipment. The solution to pollution was no longer dilution.

What remained was a fourth option, and it became the most important for industry: pollution control equipment. For sulfur, that meant scrubbers. Coal plants are extremely complex machines, and their economic value comes from operating the machine as an integrated, reliable system. In reality, a plant is at least two systems that operate in an uneasy marriage. One is the boiler, which often takes hours to start and heat up. Inside the boiler, plant operators worry about temperatures and pressures along with the propagation of the flame across the fuel as it is injected. The boiler makes steam that is sent to a turbine, where another system functions; its operators have their own set of worries about the chemistry of the water, variation in temperature and the pressure of the steam, and the reliability of the turbine itself.

Each system, if it fails or gets out of synch with the rest, can shut down or throttle back the plant. Pollution control equipment adds one or more additional systems, more operations that must be synergized, and more things that if they operate unreliably, curtail plant output (and thus cost a lot of money). One of these is the electrostatic precipitator—a machine that sits in the flue gas after the boiler and shunts the gas over an array of electrified rods. The electricity charges particles, which stick to the rods. Periodically the rods shake, and the accumulated particulates drop to the ground (and hence don’t go up the smokestack). After the electrostatic precipitator comes the scrubber.³⁹

While everything in the front of the coal plant runs according to the principles of mechanical, combustion, and electrical engineering, the scrubber is a chemical engineering system. (Suspicions across these disciplines are legendary among operators of power plants. One plant designer, during a visit by one of us to a big coal plant, declared that “chemical engineering is the work of the devil.”) The fear is that anything new bolted to an already-finicky engineering system will lower the reliability and raise the costs. In a scrubber, the sulfur in the flue gas reacts with water, reactive chemicals (e.g., limestone), and some catalysts to make prodigious quantities of some by-product (e.g., gypsum).⁴⁰ The engineering quickly demonstrated that all the components of a scrubber system would work. Bench-level lab work and small experiments generated a wealth of ideas about how to efficiently integrate scrubber components, but generating system knowledge in an industry where reliability is central required testing on actual power plants at scale, and collaborating with highly motivated owners that were willing to risk downtime and expense in exchange for knowledge.

The motivated owners were the firms with the biggest coal-fired power plants and most potential regulatory exposure, such as Detroit Edison, Southern Company, the Tennessee Valley Authority, Commonwealth Edison, and Southern California Edison. The early ideas for how to run scrubber chemistry came, in part, from the EPA’s laboratories at Research Triangle Park, which did bench science and ran small pilots. Research Triangle Park has been historically critical to the EPA’s pollution regulation because it gives the agency its own direct access to the latest science. The Tennessee Valley Authority—a government-owned corporation created during the New Deal to provide irrigation, electric power, and support for economic development in its region—also played an important role, including by running three ten-megawatt test plants at its Shawnee, Kentucky facility.

These test plants served as way stations where ideas that worked in pilots could be tested at modest scale before deployment at commercial coal plants sized in the hundreds of megawatts or larger. The Tennessee Valley Authority was also host to one of the first full-scale demonstrations, alongside Detroit Edison (which carried out a pilot project in River Rouge, Michigan, followed by a full-scale demonstration in nearby Saint Claire) and several others. Around the mid-1970s, depending on how you count, there were between four and seven major contending scrubber strategies. Each demonstration was located at a plant with an owner motivated by a desire to understand and shape emerging regulations, and if successful, use the results from scrubber experiments to control pollution at a lower cost. Each utility feared that the failure to make serious efforts to understand and control pollution would wake the gorilla in the EPA’s closet.

No utility would have been foolhardy enough to undertake this kind of experimentation in isolation from its regulators and peer companies. The risks that any single project could end in costly failure—while a similar one nearby in the industry succeeded—were simply too high. Active collaboration among utilities and with regulators was thus a precondition for exploration of the emerging technology.

Collaboration with the environmental regulator in particular was essential because plant operators wanted to show regulators what was in fact feasible (and hence should shape the standards). But collaboration was also important because the systems under test sometimes fail, causing more pollution. It is prudent to secure the regulator’s forbearance for such well-intentioned accidents in advance—in return, for example, for the right to observe the process in operation. Over time, all the utilities learned that the consent of the environmental regulator would be extremely important for yet another reason: extremely costly plant upgrades could inadvertently be treated by regulators as creating a “new” plant—and under the Clean Air Act, depending on how regulators interpreted the law, new plants were controlled much more aggressively than old ones. The utility regulator was essential, too, because these experiments at scale on actual plants were expensive. With a regulator’s blessing, they would be allowable research expenses (and therefore passed through to ratepayers without harming financial flows for stockholders) or, in some cases, investments (and thus capitalized into the rate base from which the utility and its investors earned a guaranteed rate of return).

Collaboration among utilities, meanwhile, was indispensable for all firms to learn the lessons of each plant experiment. Assuring that flow of information required engaging, beyond the firms themselves, two additional groups of actors. One was the cluster of equipment vendors and engineering contractors that could see across the totality of the industry—firms such as Peabody Process Engineering (which built the River Rouge pilot and then applied that knowledge to the full-scale Saint Claire plant) and Bechtel (which was the general contractor for many coal-fired power plants and also operated the Shawnee test facility). The other and perhaps more important group of actors was the industry itself, embodied in a newly created research collaborative: EPRI.

EPRI was itself the product of a penalty default. Fearing onerous federal regulation after the 1965 blackout, the electric industry set up a nonprofit collective research arm to develop its own solutions to collective problems.⁴¹ The institute invited regulators and public interest stakeholders onto its advisory councils to help steer the research. Scrubbers were one of the first demonstrations of EPRI’s essential role in collaborative research and information pooling. EPRI helped design demonstration projects and organize performance reviews by industry peers so that lessons from those projects about the cost and efficacy of different experimental scrubber systems became common knowledge in the industry; it also helped with “technology transfer” as firms absorbed the new knowledge into plant-level operations. (Using the same methods, the institute has played a central role in helping improve the performance of the US nuclear power industry.)

Through collaboration under the shadow of the penalty default, decentralized experimentation, and centralized learning through peer review, the scrubber was adopted as the best-available control technology for SO₂. Scrubbers in fact became a lot more reliable, and more acceptable to cautious plant operators and utility executives. New plants installed them. Even in extremely clean areas of the country, some plants installed scrubbers thanks to the prevention of significant deterioration (PSD) rule. Knowledge about scrubber reliability and efficiency led to refinement in regulatory rules; means and ends coevolved. In the early 1970s, installing a scrubber meant cutting sulfur pollution by 70 percent. With the 1977 Clean Air Act amendments, that number rose to 90 percent, and some local air pollution rules (under review by the EPA through state implementation plans) demanded even higher scrubber efficiencies—goals that were achievable at costs that would have been unthinkable prior to industry demonstrations.

It was not command-and-control decisions by remote bureaucrats that produced these results but rather an institutionalized process of collaborative experimentation and peer review involving firms, the industry’s research institute, and state and federal regulators. In a word, the process was a case of experimentalist governance through and through.

Perhaps most striking is that the industry—organized through EPRI—needed knowledge about scrubber performance that didn’t sit neatly within established ways of organizing knowledge inside the sector. Chemical engineering mattered, of course, but the best scrubber systems turned out to be those that involved reacting sulfur with limestone—both because limestone was cheap and available almost everywhere, and because good chemical control could convert the product of limestone-sulfur reactions into gypsum, which was valuable (as wallboard) or easily discarded. Through the early 1980s, alternative systems based on chemical reactions with Mag Lime (containing large amounts of magnesium oxide) were thought to be viable, but they faltered on the lack of cost-effective supplies and the expense of waste removal. Only by the mid-1980s was the limestone tower sprayer scrubber system widely accepted as best under almost all conditions.

Ironically, this collaborative approach to technology forcing was well established by the 1980s. Scrubbers were its most iconic success, and with their introduction, sulfur emissions were already declining quickly. From 1970 to the late 1980s, emissions had dropped by about ten million tons annually. But the successful continuation of collaboration and translation of the results to standards required the willingness of the EPA to continue to invoke the threat of a penalty default, and Congress to keep the Clean Air Act framework in place. The rising tide of deregulation ultimately made constancy impossible. In the first two years of the Reagan administration, EPA head Ann Gorsuch tried to eviscerate the agency she headed. When Ruckelshaus took over again in 1983, normalcy returned and collaboration continued. Yet the interlude presaged what was to come for much of the George W. Bush administration and all of the Trump administration.

THE TURN TO MARKETS: NEW ENVIRONMENTAL CHALLENGES AND NEW POLITICAL FORCES

In the late 1980s, the political stars aligned for another big change in federal law on air pollution—what became the 1990 amendments to the Clean Air Act, adopted with overwhelming bipartisan support. This amendment, the first since 1977 and the last Congress has ever passed, allowed for the codification of new missions—in particular, efforts to control acid rain. And it allowed for the act to be updated to include a new suite of policy approaches: pollution markets.⁴²

From the 1970s on, the problem of acid rain was known in theory, and evidence was accumulating that fragile ecosystems, especially along the Eastern Seaboard downwind of industrial pollution, were deteriorating. Similar evidence appeared in Europe as well, made visible by Waldsterben, the dieback of forests concentrated in central Europe that accelerated in the late 1970s and helped galvanize public opinion in favor of pollution control. Many hypotheses were offered for these dramatic declines in ecological health, but the exact causes were unknown. In 1980, the last full year of the Carter administration, Congress created the National Acid Precipitation Assessment Program to study the problem over a decade, and report on the impacts, costs, and solutions.⁴³

Meanwhile, political momentum for action grew and crested throughout the Reagan administration. Unrelated to air pollution policy, during the 1980s, the government also instituted reforms in coal leasing (which opened bigger supplies of low-sulfur coal in the West) and implemented a 1980 railroad deregulation law that lowered rail rates by 60 percent over the decade (which made it much cheaper to ship low-sulfur coal from the West to power plants around the country).⁴⁴ Had the country continued to deploy scrubbers and invest in better scrubber innovation, it is plausible that sulfur emissions would have come down just as quickly, if not faster, than required by the new limits inspired by worries about acid rain. But the politics of pollution control were shifting, and building a bipartisan consensus required a shift to markets.

The idea of using markets to control pollution was not new. The theory dated back at least to the 1960s, and in the 1980s, a coalition of academics, environmentalists, and industrialists helped articulate to policy makers the benefits and modalities of market-based pollution control.⁴⁵ A large lead-trading program had demonstrated the practicality of cutting compliance costs through market strategies—albeit in the context of a rapid phaseout of a pollutant for which substitutes were already well-known.⁴⁶ Under the 1977 amendments of the Clean Air Act, there were limited opportunities for the trading of pollution offsets and credits.⁴⁷ Despite the limits of this experience, retrospective studies showed that markets usually saved money.⁴⁸ In 1987, there was a major legislative push to amend the act to address acid rain by requiring more extensive use of scrubbers. That failed, in part, because technology mandates were seen as costly intrusions by government.⁴⁹ At about the same time, there was bipartisan support in the late 1980s for the greater use of markets for achieving environmental goals.⁵⁰ For those who were skeptical of government effectiveness and yet still interested in pollution control, markets were the way forward.⁵¹

The market solution relied on a cap-and-trade system. At the time, US emissions from large power plants were about twenty million tons of SO₂ per year.⁵² The National Acid Precipitation Assessment Program had not reported its results yet, so it was impractical to do a cost-benefit analysis.⁵³ The best, highly uncertain estimates suggested that the costs of emissions reductions would rise steeply as abatement exceeded ten million tons (a cut of power plant sulfur pollution by about half). The cap would apply starting in 1995, and big polluters that acted early could earn valuable credits that they could bank for future phases of the program when it was expected that the costs would be higher. To avoid creating too many enemies, emission permits were given away mostly for free to existing polluters.⁵⁴ Because firms knew that grandfathering was going to be the likely rule, in the late 1980s, when elements for a national trading system were being debated, the continuous decline in sulfur emissions actually stopped. Polluters saw more pollution as an advantage, and made no additional efforts to cut emissions until the baseline (1990) was agreed and an economic advantage from further cuts was written into law.

Analysts of cap-and-trade systems have looked at details of the sulfur market design with an eye to how they create economic efficiencies. Early action, which would lower emissions promptly and create extra credits that could be banked for a later phase, allows for greater efficiency in the timing of emission controls. Detailed monitoring requirements—through the continuous emission monitoring systems equipment that all large polluters were required to install—would make sure that behavior was transparent. Avoiding new requirements for equipment mandates would allow for technological efficiency. Putting the whole country under a single bubble would maximize the gains from trade. The free allocation of permits to emit, if necessary to avoid political blocking by utilities and coal suppliers, was acceptable because it would not harm economic efficiency; the giveaway just affected the allocation of policy costs in ways, hardly surprising in politics, that benefited incumbents.⁵⁵

ACHIEVING STATIC EFFICIENCY

For about eight years, the national sulfur market worked. Prices were relatively stable, and the total costs were less than half of the original expectations.⁵⁶ In the conventional view, markets created this tremendous success.

But assigning causality is, as always, tricky. Broadly speaking, the sulfur market allowed generators to choose among three options to comply with caps on emissions: buy credits on the marketplace (or equivalently, use their own credits that had been banked), install and operate scrubbers, or switch coals. Two out of those three options were made possible by actions outside the cap-and-trade program.⁵⁷ Indeed, the most important innovations in sulfur removal—scrubbing—predated the cap-and-trade system; it was the product of experimentalist governance. And while advocates for market-based policies often claim that these approaches encourage technological innovation, careful analysis of this history has shown that regulatory approaches forced a lot more innovation (and patenting) compared with more market-driven approaches.⁵⁸

Because firms were free to make these choices, the market helped achieve some static efficiencies. In political terms, however, these choices actually created substantial harm because they made learning and adjustment politically harder to achieve. By fixing pollution caps at the outset, they locked in the expectation that pollution control costs would be high. When participants learned that compliance costs would be much lower than projected, it was impossible to adjust—to make the actual effort align better with what society was willing to pay for pollution control—without changing the statute. The political preconditions for an efficient market—high credibility with clear limits on pollution—interfered with its efficient operation.

But these questions aside, even as the cap-and-trade system entered its salad days, another fundamental limit on the utility of markets was coming to light.

THE LIMITS TO MARKET-BASED APPROACHES TO SOLVING COMPLEX, LOCAL POLLUTION PROBLEMS

The general problem was that national sulfur controls—creating wide and efficient markets—were mismatched with the real environmental problems, which often involved local mixtures of many pollutants. Concretely, acid rain was the result not just of SO₂ but also its combination in particular air sheds with NO_x that turned to acid. Solving the acid rain problem would thus require addressing NO_x as well. And since coal-fired power plants were not the only sources of those pollutants (half of such oxides came from vehicles, for example), any effective strategy would need to address those other sources of pollution in addition to large stationary coal-fired power plants. Moreover, just as the 1990 Clean Air Act amendments were being adopted, it became clear that there were many other regional air pollution problems, such as smog and haze.⁵⁹ Over the years, still more regional pollution problems would become understood too, such as emitted mercury. These were the real problems that needed solving; the sulfur market, by design, focused on something different: nationally averaged emissions.

To be fair, this mismatch was discussed—if not fully appreciated—when the national sulfur program was created. Ideas were floated, for example, to address SO₂ and NO_x together in a single market, but they were rejected as too complex. Complexity, it was rightly thought, would undermine the ability of the market to send simple and clean price signals. During the period of the most active reduction—the golden age of the sulfur market—the fictions needed to make the market work were not too debilitating. The total pollution was tumbling. And the counterfactual for comparison—the speed at which emissions would have declined if the experimentalist system of collaboration had just stayed in place or had been perhaps expanded, as contemplated in 1987, with requirements to use scrubbers more widely—was unobservable.

But once the market had helped achieve quick reductions in pollution, the disconnect became readily apparent. Subsequent attempts to revise the market solution revealed that when pollution problems are highly complex and poorly understood, it is impractical to create a credible market design that can respond to the problem at hand. There were many attempts to do just that; all failed, and market participants soon learned that faute de mieux, the old methods of the Clean Air Act—regulation—remained as the key tool for controlling pollution. The sulfur market was no longer credible, and prices tumbled to zero.⁶⁰

One cause of the failure, already feared by many environmentalists when the nationwide market was created, was that trading based on national standards created hot spots. Trading might lower costs for abating sulfur overall nationwide, but by design the effort would differ across plants. In places where pollution control was relatively inexpensive (e.g., at plants located closer to cheap, western, low-sulfur coal or plants of more recent design that had scrubbers installed), the emissions would tumble. The emission credits thus generated would be sold to dirty places and used to keep emissions high or let them go higher.

One retrospective study has shown this fear was justified: western, newer plants did cut more than older plants, which tended to be located near cities. The cost of those higher local sulfur emissions (where they caused harms to human health) were five times the static efficiency gains from the sulfur trading system overall.⁶¹ Looking back, it seems that even by the standards of a single national goal—the easiest way to create a national sulfur market—the sulfur trading program was less efficient and possibly value destroying compared with a plausible experimentalist alternative that would have made the best efforts to apply scrubbers. Alas, much of this story has been pieced together only recently, in part because of an improved ability to measure local health effects from sulfur and other pollutants.

The other cause of failure is the one that actually undid the sulfur trading program: the complex interaction between sulfur and other pollutants, including NO_x, in the creation of regional pollution. The level of control needed to vary with the level of downwind pollution and ecological sensitivity. This problem arose during the Bush administration, which embraced markets on ideological grounds, crafting a complex scheme that would allow for the trading of multiple pollutants with varied weights depending on geography. Creating this kind of a market would require a lot more information than a simple national cap, and much of that information, such as the exchange rates between pollutants, was either unknowable with precision or varied in such complex ways that it would be impractical to design a market aligned with the exact, changing nature of emissions and pollution impacts. Nonetheless, the Bush administration made a formal legislative proposal to do just that: the Clear Skies Initiative. When that initiative failed politically, the administration attempted to achieve the same outcome through administrative action in the form of the Clean Air Interstate Rule.

Ideologically committed to using markets, the Bush team created ever more complex contortions that never could align incentives with the real nature of the environmental problem and control at hand. But so strong was the impetus to use markets that prior experiences to do exactly this—experiences that showed the difficulty and meager rewards from such markets—were ignored. For example, in the 1980s, a team of economists had been asked to design a market to control noise pollution at airports. Tom Schelling, who led the effort, reported that the information needed to make the market work—the exact properties of different aircraft types, flight paths, sensitivity to noise, and so on—was essentially identical to that needed for direct regulation that would not use market forces.⁶²

This was the beginning of the end of the sulfur market. After pivoting away from Clear Skies to the Clean Air Interstate Rule, the process of review and sources of penalty defaults also shifted to the courts, which threw out the Bush market initiatives because they strayed too far from the original purpose of the Clean Air Act: to cut pollution, reliably, anywhere and everywhere needed. Bush’s EPA never finished its attempt to update its failed rule. The incoming Obama team, working through administrative action as well, offered its own proposals, but they too failed to deliver what the courts and political leaders saw as essential in protecting the environment. As Clear Skies, the Clean Air Interstate Rule, and the Obama proposals evolved and died, market participants quickly learned that the old sulfur market was no longer credible. The golden era of sulfur trading was over.

Less visible and more important was that the Clean Air Act’s original mission was still in place, and the EPA’s gorilla would be awakened periodically. Industry knew this, and it kept collaborating as it had on sulfur—working on each new frontier in pollution control. For instance, the same lead firm (Southern Company) working with the same industry organization (EPRI) tested multiple methods for cutting mercury.⁶³ The logic remained the same: better to get ahead of the gorilla and collaborate than wait for it to come crashing in the front door.

What are we to make of this story? Some market advocates see the collapse of sulfur trading as an object lesson in government’s fickle nature, sometimes giving (creating a market) and sometimes taking it away. We see this history differently. The market can thrive so long as there are many opportunities for trading—a large trading zone with a single price that homogeneously affects all actors—and the participants are choosing among options with a known performance. Yet the real challenges and goals in controlling pollution are much more complex, as are the technological and operational changes needed to clear the air.

It would be a misreading of sulfur history to conclude that either markets or administration are inherently ineffective. The real lessons are about where and why different policy strategies work best. When uncertainties about control strategies are large and the best strategies forward are unknown, experimentalist regulatory systems work best. All parties have an incentive to decompose the problem into smaller, solvable components and work on solutions; the regulator knows that goals must be treated as provisional and adjustable in light of experience. Once goals can be set more firmly, and the range of technological and behavioral options is better understood, markets can take over. (Under those conditions, however, much of the success of markets is also available to regulators because the best choices are widely known.) That happened in the history of sulfur control, but recalling only that portion of the story distorts the moral, suggesting that regulation was the problem and markets were the solution. As we’ve shown, a fuller account demonstrates the crucial role of experimentalist regulation in making basic innovation practically applicable and revealing the limits of markets in solving local problems. Cap and trade was useful in the interlude when it worked, but as a promise of fundamental reform—a driver of innovation—it failed.

Innovating at the Border of Science and Technology

We turn now to our final case study of ARPA-E. Much of the time, technology-forcing regulation implemented through experimentalist processes can induce or accelerate the innovations needed to address policy goals. But not always. When innovation is especially costly and risky, the payback period for successfully commercialized products is especially long and uncertain, and the full benefits of innovation are hard to appropriate, then the motivations for innovators may be insufficient. In the case of such market failures, it is the role of government to step in and directly encourage as well as monitor investment in those key technologies and capacities without which progress on a broad front will be impossible. In the vocabulary of chapter 3, the form of intervention switches from regulation to industrial policy.

The problem, perhaps especially in the US context, is that alongside the arguments in favor of industrial policy as a response to market failure, there are arguments about the likelihood of government failure that make such industrial action fraught with danger. All the reservations about the shortcomings of administration and all the cautions about regulation, and particularly regulatory capture, count greatly with respect to industrial policy. Public bureaucracy is too sclerotic, the complaint goes, and the political process is too beholden to special interests to allow for dispassionate, expert decision-making, even supposing that public officials can know enough about technology and markets to have the relevant expertise in the first place.

Sure enough, industrial policy is one of the most contentious topics in economic policy. Opponents insist that arrogant ambitions to “pick winners” will produce “white elephants” or “cathedrals in the desert.” Proponents respond that in the face of manifest and severe market failures, the costs of inaction far outweigh those of this or that bad decision. The argument is typically by example—with failed investments such as Fisker (an early electric vehicle start-up) and Solyndra (a company searching for solar cells that would require less silicon) thrown into one pan of the balance, and successes such as Tesla (which repaid a loan nine years early) stacked on the other. Similarly, every study decrying a “technology pork barrel” can be paired with another portraying the (sometimes) proven promise of industrial policy.⁶⁴ Even in development economics—which coined the term “industrial policy” to connote the bundle of measures by which a poor economy could build the industries that were long thought to be the precondition for sustained growth—there is surprisingly little attention given to the governance arrangements that explain why these policies do and don’t work.⁶⁵ Yet it is these governance arrangements that hold the key to understanding when and how industrial policy gets the job done.

One essential element of these arrangements is their impact on technological innovation. Successful industrial policy must be anchored in a strategy for supplying fundamentally new ideas. (The case of climate change is particularly emblematic; a recent study by the International Energy Agency concludes that about three-quarters of all the technologies needed for deep decarbonization aren’t yet technologically or commercially viable.)⁶⁶ ARPA-E, which began operation in 2009, provides an illustration of a highly successful system of clean energy innovation.⁶⁷

ARPA-E’s overarching goal is to eliminate “white spaces” in the map of technical knowledge: missing capabilities just beyond the frontier of current technical possibility that if mastered, would clear the way to advances in an important domain. A program might, for instance, aim to support the investigation of novel battery concepts with the potential to reduce storage costs enough to make an attractive class of electric grid designs economically feasible. The agency fills an important hole in the system of energy innovation supports—between publicly funded start-ups, which emerge from universities or national labs, and privately funded ones, which emerge through venture capital and incubators.⁶⁸ One indication of ARPA-E’s robust success is that despite regular political accusations that it is wasteful and crowds out the private sector, it can count for funding support on a bipartisan group of legislators impressed by the results. For our purposes, ARPA-E is crucial not only because of its role in clean energy but also because its systems for governance have been studied closely.

ARPA-E inherited and then refined its governance structure from its near namesake, the Defense Advanced Projects Agency (DARPA), created in 1958 in response to the Soviet launch of the Sputnik satellite. In discussions of industrial policy, DARPA is often invoked as a reminder that the state played, and continues to play, a fundamental role in organizing the research from which are hewn the building blocks of the information economy. Among its iconic contributions are the computer network protocols underlying the internet, precursors to global positioning systems, and fundamental tools and devices for microprocessor design and fabrication. The accomplishments of DARPA have inspired a number of research agencies along similar lines, of which ARPA-E is both the most successful and most faithful to the procedures of the original model.⁶⁹ (Of course, there are significant differences between DARPA and these descendants. One is that DARPA is oriented around more extensive discussions with customers and users of technology, and about half the DARPA budget is spent on later-stage testing and evaluation along with the early deployment of technologies. ARPA-E, by contrast, focuses almost entirely on early stage innovation, in part because there are other programs at the DOE that work on the later stages.)⁷⁰

At every stage in the organization of research—defining the target or programs of investigation; selecting projects that advance the program purpose; and supervising individual projects in the program portfolio—ARPA-E treats goals as provisional or corrigible, and uses peer review to surface differences triggering further investigation. To begin with, program directors are hired largely on the basis of their promise in giving shape and direction to an emergent area of investigation. A candidate with a background in geology, for example, will be hired to create and concretize a program in advanced geothermal energy. Once the program goals have been framed, the program director does a “deep dive,” supplementing and correcting their own background experience with reviews of the scientific literature, site visits to universities and companies by ARPA-E technical staff, commissioned external studies, and consultation with research managers elsewhere in the DOE. Program directors then test the practicality of the emerging research area in technical workshops involving leading engineering, scientific, and commercial experts. If the research plan, adjusted to reflect the exchanges at the workshop, passes review, a project is formally created as a component of the developing program.⁷¹

Proposals for research within the projects are developed and executed in the same manner, with the goals open to recurrent challenge and revision via peer review with ARPA-E’s professional staff. Applicants first submit a concept paper: a short document explaining why the proposal is superior to alternative approaches, and how it responds to foreseeable technical and commercial risks. Proposals that survive a first round of external review are developed into full applications and reviewed again, with the difference that applicants may rebut criticism by external reviewers.⁷² The winners, designated as “research partners” or “performers,” then negotiate project milestones with the agency staff.

A notable and perhaps surprising feature of this process is that the selection is not based on a consensus view of the project’s prospects. Holding the rating constant, the agency picks the project where the range of reviewer rankings is the greatest—that is, where judgments diverge the most. Since disagreement among experts is a good working definition of uncertainty, this preference for divergent rankings is a further indication that ARPA-E fully recognizes the uncertainty in its environment. In the absence of expert consensus, the managers and selection committee rely on other information, such as rebuttals, observation of the research in workshop, dialogue with peers, and the need for methodological diversity when the best strategies are unknown.⁷³ Managers know that consensus-seeking peer review—common in academia—often has a bias against the novelty and risk that transformative research programs deliberately seek.

Once funding is offered, ARPA-E managers remain intensely involved—checking milestones, adjusting plans, and learning. Missed milestones can touch off an intensification of site visits, conference calls, meetings, and written analyses of problems and possible solutions. When projects struggle, milestones can be reset to permit an alternative to the failed approach. Milestones are added or deleted in fully 45 percent of the projects, not counting substantive modifications, which are said to be frequent. Staff can adjust the length and budget for projects (empirical work finds that most projects see a modest lengthening and rise in budget). If recovery efforts fail, the program director sends an “at-risk” letter warning of the possibility of termination.⁷⁴ In short, the agency rejects the model of hands-off, bet-on-the-person-not-the-project administration preferred by many established and successful research funders, public and private, in favor of the continuous, collaborative review and adjustment central to experimentalist governance, and widespread, as noted in chapter 3, in biotechnology, advanced manufacturing, and venture capital.⁷⁵

ARPA-E is a particularly successful model of experimentalist governance in an organization (the DOE) and domain (energy research) often seen as anathema to nimble experimentalism. Yet ARPA-E is not alone. Within the DOE, there are many other technology demonstration programs, and they too have learned how to elicit ideas from the outside, define progress in terms of commercial potential, and adjust those terms in response to new information. The results from these investments suggest that the pork barrel critique, if it was ever valid, is outdated. A large statistical study of direct DOE support to early technologies through the small business innovation research program has found that the program does increase the prospects for commercial success; comparing proposals on both sides of the line between the DOE’s support and nonsupport, the ideas that get the halo and resources of the DOE backing do better.⁷⁶ (Other government agencies run similar programs, with encouraging results.)

A separate, independent study of US cleantech start-ups has shown that those with government partnership fare better, in part because of quality signals from the partnership; the patenting nearly doubles and private financing rises 155 percent for start-ups that have government alliances compared with those that don’t.⁷⁷ From these numerous, diverse, and encouraging experiences, a whole set of “best practices” has emerged about how to use government research, design, and development resources effectively—a role for intensive peer review, centralized periodic evaluation of portfolios, revision of goals in light of experience, attention to commercialization metrics that are adjustable with experience, and a blend of decentralized effort but centralized assessment.⁷⁸ Those metrics are, in a different word, key elements of experimentalism.

Organizing Innovation

What are we to take away from these three case studies? CARB is dedicated to technology-forcing regulation, the sulfur trading regime to market making, and ARPA-E to a central element of industrial policy: innovation. Together these three strategies constitute the principal choices for climate change intervention, and often these choices are subject to bitter political contest, on the assumption that selection of the “right” strategy for a particular context was necessary and sufficient to ensure success.

Our argument is that these contests are fighting the wrong fight and arguing over the wrong question. There is no knockdown argument to be given for the “right” choice of intervention—at least if our goal is practical success. Settling on one of these strategies is insufficient because it leaves unspecified the all-important governance arrangements that determine whether decisions can be rapidly corrected under uncertainty. In the end, what made CARB, ARPA-E, and sulfur control successful—setting them apart from efforts otherwise similar in their respective strategies—were their structures of experimentalist governance. Experimentalist governance, in short, is a necessary condition of success (though hardly sufficient).

At times these diverse policy strategies may be complementary. We saw in the case of sulfur that regulatory systems that pushed technological innovation were complementary to market incentives. We suspect that this is frequently and perhaps regularly the case; regulations and active investments in innovation are the anchors of industrial policy, and market incentives help optimize the system once directions and technologies are known.

Equally important to the technical processes of regulation under uncertainty are the political effects of these programs. Both CARB and the sulfur program explicitly evolved in ways that kept solving environmental problems at an acceptable cost, which helped to sustain political support. The only big exception to that rule was the brief period when nearly all sulfur reductions occurred through the cap-and-trade program; then the actual effort fell short of what the public probably would have tolerated and drifted away from what was needed for environmental protection in key jurisdictions, such as the states where power plants were located close to vulnerable populations. This political logic to experimentalism is evident as well at ARPA-E, where some of the new technologies emerging from ARPA-E support are offered as evidence that transformations in energy markets can run faster than was originally thought with previous generations of technology.

The cases discussed in this chapter all share a feature in common: they are examples of innovation at the technological frontier. In fact, a great deal of innovation takes place apparently far from the frontier when new products or processes are often adjusted—and sometimes reinvented—to work under local conditions for which they have been only partially designed. Such localized innovation, or contextualization, is particularly important in climate change, where solutions and institutions designed for one place regularly don’t work well in others. We turn to this contextualized innovation in the next chapter.