SB 1047: AI Safety Regulation Gets Real

How I learned to stop worrying and love the world's most controversial AI bill

There are two major categories of AI regulation, which I will call AI Safety and AI Governance. AI Safety is about potentially catastrophic harms from the most powerful forms of AI. AI Governance is about how everyone developing and deploying AI should take appropriate steps to map, measure, and manage its risks. Both are needed. A drugstore deploying off-the-shelf facial recognition to limit shoplifting doesn’t need to worry about its AI being used to create a devastating bioweapon. Yet it should take reasonable steps to mitigate discriminatory bias, or suffer consequences. Those steps, on the other hand, aren’t targeted to the mass harms that could conceivably result from large-scale deployment of increasingly capable frontier models.

For some time, much of the conversation in industry and research centers, as well as some academic quarters, emphasized AI safety. Doomsayers and leaders of AI labs themselves raised the alarm that advanced AI could fail to be “aligned” to human interests and pose devastating threats. Yet most of the action on AI regulation the past five years has been on the AI Governance side. Initiatives such as the European Union’s AI Act and the Biden Administration’s AI Executive Order have limited disclosure requirements for “systemic risk” or “dual-use” foundation models; the bulk of their text details pragmatic governance requirements with far greater scope.

The world’s first major AI Safety legislation, SB 1047, is now moving through the California legislature. It has provoked intense reactions, including scary warnings from a range of influential voices. Opponents of SB 1047 include not only big tech companies and VCs such as Andreessen Horowitz, but leading technologists and academics such as Arvind Narayanan, Andrew Ng, Yann LeCun, and Fei Fei Li. Not to mention House Speaker Emerita Nancy Pelosi and key Democratic members of the California Congressional delegation. The bill has important supporters as well, including distinguished AI researchers Yoshua Bengio, Stuart Russell, and Geoffrey Hinton, as well as legal scholar Larry Lessig, Ethereum creator Vitalik Buterin, and my Wharton colleague Kartik Hosanagar. The opposition, however, has been louder and more widespread, even though the bill sailed through the State Senate and polls with consistently strong public support.

SB 1047 applies to developers of frontier models meeting a compute threshold above any model in deployment today, costing $100 million to train, with the potential to cause $500 million of harms. It requires them to take several steps including implementing standards and best practices for safety, creating a written safety and security protocols, and being subjected to audits. The California Attorney General has the power to act if firms fail to meet the bill’s requirements.

I was originally persuaded by critics that these mandates were unnecessary, chillingly vague, and threatening to AI development in the US. After reading the text of the bill, considering the arguments on both sides, and following the significant amendments by the bill’s sponsor, I’ve come around to seeing SB 1047 as a worthwhile step. It’s not perfect — no regulation is. It might be interpreted over-broadly by the courts or weaponized by a malicious state Attorney General — again, virtually any regulation could be. And the fears of catastrophic AI safety failures might well be overblown — my intuition is they will be. Yet at a time of tremendous uncertainty about AI’s trajectory, and given the magnitude of the potential risks, reasonable preventative action is a pragmatic course.

One of the most compelling objections to SB 1047 comes from Princeton researchers Arvind Narayanan and Sayash Kapoor, who argue that safety is a property of deployed AI systems, not the neural network models that the bill regulates. As a result, model safety efforts are “inherently limited in their effectiveness.” This is certainly true. It doesn’t mean that model developers should make no effort. That sophisticated actors can remove guardrails in deployed systems, especially for open weight models, doesn’t mean those guardrails are useless for any risk scenario. Not to mention that safety research and implementation by the frontier labs advances the state of the art and the understanding of good practice for everyone.

And while there needs to be regulation of AI systems to complement SB 1047’s rules for frontier model developers, trying to oversee the countless AI systems deployed around the world by organizations, individuals, and unidentified parties isn’t an adequate substitute for regulating a small number of well-resourced frontier model developers. I’m frankly more worried about the massively detailed requirements that the European AI Act imposes on anyone developing or deploying a broad range of “high risk” AI systems.

Many of the attacks on SB 1047 make general statements that it “chills innovation,” “kills small AI startups,” “regulates R&D,” or “bans open source AI.” The question is what, specifically, the bill would have these effects? Scott Wiener, the bill’s sponsor, claims SB 1047 just mandates the labs do what they are already doing. All the major frontier AI labs (with the arguable exception of Elon Musk’s X.ai) have made voluntary commitments on AI safety, and are actively pushing development of techniques such as red-teaming, fine-tuning, constitutional AI, watermarking, and mechanistic interpretability, as well as sophisticated cybersecurity, to make their models and systems based on them safer.

Narayanan correctly points out that it is impossible to be certain that a frontier AI model “cannot enable critical harm.” But is that what SB 1047 requires? As recently amended, it just obligates frontier model developers to take “reasonable care,” the bedrock standard of negligence in tort law. That’s a mandate to take preventive steps that a normal similarly-situated actor would be expected to take, not a guarantee of harm prevention. (The bill’s original language of “reasonable assurance” was functionally identical, but could be read as a stronger obligation; the amendment was a helpful check on judicial interpretation.)

Lots of innovative industries are regulated; heck, lots of innovative AI development is already regulated, in a variety of ways. So the question is what, specifically, is so bad about SB 1047?

Many of the original objections to the bill have been addressed through amendments. I’ve already mentioned the “reasonable assurance” language. A threshold for “covered models” that ignores the plunging costs of training with a given level of computation? Addressed with a $100 million training cost requirement, which can be adjusted over time. A “kill switch” requirement impossible for open source model developers to meet? Replaced with one only extending to copies of the models still under their control? Users of those open source models worried about inheriting their developers’ obligations? A limitation to those users who themselves spend $10 million fine-tuning the model, and a “limited duty exemption” when prior safety measures remain sufficient. The new Frontier Model Division of the state government that could have become an unaccountable AI regulator motivated to expand its power? Gone. The threat of criminal perjury charges by the state Attorney General for safety filings? Gone. Cloud computing disclosures that, as Anthropic pointed out, duplicated existing federal requirements? Gone. The potential for sanctions on plans deemed insufficient even when no harm actually occurs? Gone.

This is how public policy is supposed to be made, with considered legislative compromises in response to well-founded objections. Though the changes in the bill haven’t addressed all the concerns of opponents, they are far more than window dressing. Perhaps I’m missing something, but at this point, I believe SB 1047 would be a helpful step to address frontier AI safety risks.

Whether it passes or not, the issue SB 1047 raises about the allocation of obligations between model and system developers, and related questions between developers and deployers of AI systems, will be central for AI regulation and policy in the coming years. There will be a balancing act to create sufficient incentives for appropriate care across the board, while avoiding unreasonable or untenable obligations at any point. Ex ante regulation such as SB 1047 and the European AI Act must also be complemented by post hoc liability rules that influence AI safety investments indirectly. The SB 1047 debate now ongoing in California is just the start.

In a subsequent post, I’ll drill down in more detail into the concerns that SB 1047 would chill AI development. As always, I welcome your comments!