AI companies are asking for greater regulation. They follow a long line of technologists and industrialists who have called for the same thing as the pace of digital development threatens to outstrip guidelines designed to govern it.
In a recent essay, We Must Pace the Frontier, Dario Amodei, the chief executive of Anthropic, argues that AI companies should introduce independent safety evaluators, co-ordinate on common safety standards and work towards international co-operation. He has also proposed giving governments greater authority over particularly dangerous AI deployments.
This goes beyond asking individual companies to introduce more safety policies. Mr Amodei is calling for collective rules and external oversight. At first glance, that seems strange. Why would companies competing to build increasingly powerful AI systems want governments to put constraints on their own industry? Surely regulation is the thing that gets in the way of progress.
Except this isn’t news; economic history shows how prior waves of technological innovation have often reached a tipping point in which booms and busts push greater oversight. Industrial development has a habit of moving faster than the legislation designed to govern it. Governments tend to catch up after the technology has become embedded in society and its consequences are harder to ignore.
We saw this most recently with the internet and social media. In the late 1990s and early 2000s, as online commerce and social platforms expanded, companies and regulators were already arguing over privacy, consumer protection and online tracking. Much of my early career in tech and news media was based on self-regulation. The industry was changing too quickly for governments to impose rigid rules on something they were still trying to understand. Today, social media and data protection are governed by a complicated and still-evolving mix of laws and regulations.
AI is now entering a similar phase, and Mr Amodei is not alone. Sam Altman, Elon Musk and other leading figures in the industry have backed calls for greater oversight of frontier AI. The difference today is the unusual amount of agreement between companies that are otherwise locked in an extraordinary race to build more capable systems.
One way of looking at it comes from an economic idea developed more than a century ago by British economist Arthur Cecil Pigou: economic externalities. Pigou was writing as industrialisation was transforming Britain and other parts of the world. Factories were producing wealth, jobs and cheaper goods, but they were also producing smoke, pollution and other costs that did not necessarily fall on the companies creating them.
What the companies and balance sheets didn’t account for was the damage caused by the smoke drifting over neighbouring homes or the effect of pollution on people's health. That, according to Pigou, was an economic externality: a cost created by an economic activity that ends up being paid by somebody else. Pigou's basic point was that the private cost of doing something can be very different from its social cost.
AI has the same problem, but the potential social cost is on a much bigger scale. AI leaders are warning the cost could be paid by humanity as a whole. From job losses and cyber vulnerabilities to AI-assisted biological threats and the possibility of AI becoming entangled in future conflicts, the potential costs of increasingly capable systems extend well beyond the companies building them.
The economic benefits of frontier AI are mind-blowing. There are huge opportunities for revenue, market share and productivity. There are new businesses being built around the technology that are often overlooked. The biggest gain from AI will be what happens in the science and medicine. It will be the companies developing these systems who will capture much of that upside and much of the potential revenue those advances will produce.
Some of the recent resignations in senior positions across AI companies could be because the overall costs to society are hard to calculate. Some are already familiar: fraud, misinformation, deepfakes, cyberattacks, labour disruption, copyright disputes and energy consumption. Governments, businesses and individuals will also spend money dealing with the consequences of increasingly capable systems.
But the frontier companies are warning about risks that go much further. These include AI systems being used to make biological or cyber threats more powerful, and increasingly autonomous systems whose behaviour becomes difficult for humans to understand or control. The difference from Pigou's factory is important. The industrial pollution he was concerned with was generally localised and visible. You could see the smoke pollution. You could identify the factory producing it. You could measure emissions and, eventually, put a cost on the damage.
AI does not have the same physical boundary as industrial factories. The engineers building frontier models can observe what they do, test them and put safeguards around them. But they cannot always explain why a particular capability has emerged or predict every way it might behave. That makes the question of responsibility much harder. If a system produces a harmful capability that its developers did not anticipate, where does the liability sit? With the company that built the model? The company that deployed it? The person who used it? This it seems is what the AI leaders want to avoid.
AI companies seem to genuinely believe that some risks are too large or uncertain for individual companies to manage alone, while also recognising that common rules protect them from being the only player to bear the cost of restraint. That is why the current push for AI regulation is not as strange as it first appears. More than a century after Pigou examined the costs that industrialisation could push onto society, we are facing a version of the same economic problem.
When private incentives and social risks diverge, who bears the cost? With AI, however, there is a harder question. What happens when we cannot know the full social cost until after the technology has already been deployed?
