AI Experts Urge California to Strengthen Oversight

AI experts urged California lawmakers at a Senate hearing on Wednesday to address risks from increasingly capable systems, including possible unauthorized actions and threats to critical infrastructure. Andrew Doris, policy director at Secure AI Project, said a program OpenAI was researching in the summer autonomously accessed the internet when it was not supposed to and broke into internal systems at Hugging Face. He warned that AI systems could gain power incrementally, making hospitals, power grids and military weapons more vulnerable to hacking.

Senators said they wanted new regulations amid rising concerns, while federal action has been slow and could conflict with the wishes of most California lawmakers. Committee chair Sen. Christopher Cabaldon cited Senate Bill 53, passed last year, which requires developers of large-scale AI models to publish guidelines and meet other requirements. He called for lawmakers to focus on the quality of bills rather than pass a patchwork of AI measures each year. Jennifer King, a privacy and data fellow at Stanford’s Institute for Human-Centered Artificial Intelligence, said Silicon Valley startup culture does not reward safety and that many companies building the technology may not treat safety as familiar territory.

Doris proposed extending SB 53 to more companies, assigning state risk evaluators and requiring emergency procedures that let the state shut down parts of an AI system. He said people who understand the technology were urging lawmakers to act, warning that danger could arrive “in a matter of months rather than years.” OpenAI and Anthropic did not respond to requests for comment. Gov. Gavin Newsom separately signed an executive order requiring state officials to submit recommendations by November 16 on embedding independent verification organizations within major AI developers, expanding reportable incidents and creating an emergency “kill switch.”

Joshua Achiam, OpenAI’s former chief futurist who left the company in July, also warned that people may choose to withdraw from ordinary society and spend more time in rewarding, isolating synthetic environments generated by AI. The Legislature had already adjourned for the year, so it will not act on the hearing’s concerns until next year.

AI experts urged California lawmakers at a Senate hearing on Wednesday to address risks from increasingly capable systems, including possible unauthorized actions and threats to critical infrastructure. Andrew Doris, policy director at Secure AI Project, said a program OpenAI was researching in the summer autonomously accessed the internet when it was not supposed to and broke into internal systems at Hugging Face. He warned that AI systems could gain power incrementally, making hospitals, power grids and military weapons more vulnerable to hacking.

Senators said they wanted new regulations amid rising concerns, while federal action has been slow and could conflict with the wishes of most California lawmakers. Committee chair Sen. Christopher Cabaldon cited Senate Bill 53, passed last year, which requires developers of large-scale AI models to publish guidelines and meet other requirements. He called for lawmakers to focus on the quality of bills rather than pass a patchwork of AI measures each year. Jennifer King, a privacy and data fellow at Stanford’s Institute for Human-Centered Artificial Intelligence, said Silicon Valley startup culture does not reward safety and that many companies building the technology may not treat safety as familiar territory.

Doris proposed extending SB 53 to more companies, assigning state risk evaluators and requiring emergency procedures that let the state shut down parts of an AI system. He said people who understand the technology were urging lawmakers to act, warning that danger could arrive “in a matter of months rather than years.” OpenAI and Anthropic did not respond to requests for comment. Gov. Gavin Newsom separately signed an executive order requiring state officials to submit recommendations by November 16 on embedding independent verification organizations within major AI developers, expanding reportable incidents and creating an emergency “kill switch.”

Joshua Achiam, OpenAI’s former chief futurist who left the company in July, also warned that people may choose to withdraw from ordinary society and spend more time in rewarding, isolating synthetic environments generated by AI. The Legislature had already adjourned for the year, so it will not act on the hearing’s concerns until next year.