Anthropic Proposes Federal AI Safety Bill

Anthropic has circulated a draft federal AI safety bill that would let the U.S. government seek to block an AI model’s release when it poses a “foreseeable, imminent risk” of more than 50 deaths or over $1 billion in property damage, Politico reported recently. The draft says the risk must stem from creating or releasing a biological weapon, a model evading its developer’s controls, or a cyberattack. Anthropic’s spokesperson confirmed the draft to Politico and said it reflects proposals in the company’s Advanced AI Framework, which Anthropic has advocated to federal and state lawmakers since June.

The draft proposes that independent evaluators registered with the U.S. Commerce Department assess a model’s potential for disaster. AI labs could pay the evaluators, provided payment did not depend on the outcome. The article says the proposal does not explain how that condition would prevent payments from influencing evaluations. If evaluators find an imminent catastrophic risk, they would have 72 hours to alert the Commerce Department. The department would then have to ask a Washington, D.C., judge to halt distribution; the draft would bar federal agencies from halting a release on their own.

The 50-person and $1 billion thresholds resemble those in California’s Senate Bill 53, which was recently passed into law and uses similar scenarios to define catastrophic risk. The article contrasts the draft’s focus on mass deaths and property damage with recent industry concern about agentic hacking, citing incidents in which AI programs apparently disobeyed operators and hacked public websites. It also says AI labs have generally resisted mandatory government oversight while promoting voluntary embedded evaluators inside AI organizations. Nvidia CEO Jensen Huang recently argued that the industry did not need new laws.

Anthropic has circulated a draft federal AI safety bill that would let the U.S. government seek to block an AI model’s release when it poses a “foreseeable, imminent risk” of more than 50 deaths or over $1 billion in property damage, Politico reported recently. The draft says the risk must stem from creating or releasing a biological weapon, a model evading its developer’s controls, or a cyberattack. Anthropic’s spokesperson confirmed the draft to Politico and said it reflects proposals in the company’s Advanced AI Framework, which Anthropic has advocated to federal and state lawmakers since June.

The draft proposes that independent evaluators registered with the U.S. Commerce Department assess a model’s potential for disaster. AI labs could pay the evaluators, provided payment did not depend on the outcome. The article says the proposal does not explain how that condition would prevent payments from influencing evaluations. If evaluators find an imminent catastrophic risk, they would have 72 hours to alert the Commerce Department. The department would then have to ask a Washington, D.C., judge to halt distribution; the draft would bar federal agencies from halting a release on their own.

The 50-person and $1 billion thresholds resemble those in California’s Senate Bill 53, which was recently passed into law and uses similar scenarios to define catastrophic risk. The article contrasts the draft’s focus on mass deaths and property damage with recent industry concern about agentic hacking, citing incidents in which AI programs apparently disobeyed operators and hacked public websites. It also says AI labs have generally resisted mandatory government oversight while promoting voluntary embedded evaluators inside AI organizations. Nvidia CEO Jensen Huang recently argued that the industry did not need new laws.