A new study argues that weak artificial intelligence safety regulation can backfire, potentially making AI products more dangerous than if no regulation existed at all. The research, published Monday in the Proceedings of the National Academy of Sciences, uses theoretical economics and game theory to show that the structure of AI regulation matters as much as its existence.
Researchers from Cornell University and Carnegie Mellon University built a theoretical model to determine how AI regulation can most effectively improve safety. Their central finding is that regulation must be strict and target the entire AI supply chain, from the general-purpose model developers to the downstream companies that apply the technology in real-world settings.
Key facts from the study
- The study was published on Monday in the peer-reviewed journal PNAS.
- The authors used game theory and economic modeling to analyze AI safety incentives.
- Weak regulation can lead to worse safety outcomes than no regulation.
- For regulation to work, it must cover both upstream AI model makers and downstream users.
- Regulating only downstream companies encourages free-riding by developers.
- A careful mix of strict, well-placed regulation can benefit both safety and business interests.
Why weak regulation can backfire
Over the past few years, governments around the world have struggled to keep pace with the rapid development of artificial intelligence. The United States, in particular, has seen intense debate over how much federal oversight AI companies should face. Some lawmakers and tech executives fear that heavy regulation will slow innovation and undermine America's competitive position in the global AI race, especially against China. Others insist that unregulated AI development is an unacceptable gamble with public safety.
The new study enters this debate with an unexpected message: weak rules may be worse than none. The reason lies in the way incentives are distributed across the AI supply chain. At the top of that chain are general-purpose AI model providers, such as OpenAI, Google, and Anthropic. These companies develop powerful foundation models that can be adapted for many different uses. Below them are downstream companies that integrate these models into products like AI medical diagnostic systems, customer service chatbots, and automated screening tools.
If the government focuses only on downstream companies, the model developers are left with less responsibility. The researchers say this leads to free-riding. Developers assume that the downstream companies will be forced to add the necessary safety layers, so they cut back on their own investments in things like third-party audits and robust internal testing. The result is a less safe end product than if developers had maintained a strong safety culture.
'There's a free-riding behavior that occurs,' said Benjamin Laufer, the study's principal author. 'The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist.'
The prisoner's dilemma at the heart of AI regulation
Game theory helps explain this dynamic. The situation is a classic prisoner's dilemma, a concept in which two rational decision-makers must choose between cooperating and betraying each other. If both cooperate, they achieve the best possible outcome. If both betray, they get a mediocre outcome. If one cooperates while the other betrays, the cooperator gets the worst outcome. Because neither player can be sure what the other will do, both are tempted to betray. In the AI context, the betrayal is choosing to cut safety investment in the hope that someone else will shoulder the burden.
Strict regulation changes the game. When regulators hold every player in the supply chain to meaningful safety standards, no one can benefit from free-riding. This creates the trust needed for cooperation. The researchers argue that strong, well-placed regulation can actually improve outcomes for all players, including the AI companies themselves. The model defines utility as revenue share minus investment cost. Companies may fear that safety investments will eat into their profits, but the study suggests that coordinated safety investments can produce better products and better long-term returns.
Two camps in the AI policy debate
The study arrives at a moment of deep polarization in AI policy. On one side, anti-regulation technologists want light-touch federal guardrails. They often frame their position as pro-innovation and argue that the AI industry should be free from what they view as unnecessary restrictions. According to them, speed is essential for winning the global AI race against China. This group sometimes characterizes supporters of stricter regulation as doomers who overstate the risks, or as players attempting to use regulation to entrench their market position, a phenomenon known as regulatory capture.
On the other side, advocates of stricter AI safety regulation argue that the industry has strong financial incentives to downplay or underestimate the risks of under-regulated development. They point to a wide range of potential harms, from AI-generated misinformation and AI psychosis to the community health effects of large data centers and the prospect of widespread unemployment as AI adoption expands. The new research suggests this debate may be missing a crucial nuance. Safety and revenue do not have to be an either-or choice.
How free-riding plays out in real products
To understand why weak regulation can be dangerous, imagine an AI medical diagnostic system built by a startup that relies on a foundation model from one of the major AI labs. If the government imposes strict safety requirements only on the startup, the startup must spend heavily to test the system, validate its outputs, and mitigate biases. Meanwhile, the foundation model developer faces no equivalent pressure. It can release updates with less rigorous testing, because it expects the startup to catch the problems. In the worst case, the startup misses a flaw that could have been caught upstream, and patients are harmed. The same dynamic can play out in many sectors, from hiring software to autonomous vehicles and financial fraud detection.
The researchers call this supply-chain effect the central blind spot in current regulatory thinking. Laufer said that people often think of AI as a single object, but it is actually a complicated web of stakeholders, each with their own contribution to the technology. He argued that thoughtful regulation must consider the whole supply chain, not just a single provider or entity.
What this means for policymakers
These findings have practical implications for lawmakers crafting AI rules. The first is that regulatory exemptions for general-purpose model developers are not a harmless compromise; they can undermine safety across the entire ecosystem. The second is that audits and safety standards should be applied uniformly to both upstream and downstream actors. A fragmented approach that imposes strict rules on one layer while leaving another free to cut corners may be worse than no regulation.
The study also complicates the assumption that safety investment always hurts competitiveness. The model shows that in a well-regulated environment, all players can benefit. By making safety a shared responsibility, regulation can reduce uncertainty, encourage cooperation, and increase the value of the final product. This is not an argument for maximal regulation, but for the right kind of regulation: strict enough to deter free-riding, and broad enough to cover every stage of the AI supply chain.
As the United States continues to debate the future of AI governance, the researchers' warning is a timely reminder that weak rules can create a false sense of security. Doing something is not always better than doing nothing. If regulation is designed poorly, it can shift responsibility to the least powerful players, weaken accountability, and produce outcomes that are more dangerous than a complete absence of rules. The challenge for policymakers is not simply to regulate AI, but to regulate it in a way that aligns incentives across the entire chain of development, deployment, and use.
Source:Gizmodo News