Japan’s AI Safety Institute (AISI) plans to complete an evaluation of its in-house benchmark for measuring AI model safety by the end of fiscal 2026. Director Akiko Murakami said AISI is still deciding which models to assess and how much of the benchmark to publish. The institute wants to evaluate a broad range of publicly available models, but she said full disclosure could reveal evaluation methods to malicious users.

Established in February 2024, AISI works with relevant government ministries and overseas counterparts to develop AI safety standards and evaluation methods. At a results-sharing meeting for industry proof-of-concept trials on October 7, Murakami said AISI’s aims include promoting trustworthy AI, strengthening competitiveness in the AI field, and deterring national security threats linked to AI development and adoption. Its four areas of work are developing safety benchmarks, building its ability to assess models and systems, collecting, analyzing and sharing AI-related information, and leading international coordination while strengthening competitiveness. Murakami said it is necessary to discuss risks proactively through activities such as sharing trial results, rather than waiting to respond after safety incidents occur.

AISI has already conducted tests using Inspect, an evaluation tool from the UK AISI, and publicly available benchmarks. It plans to build a system for continuously evaluating many models and to expand safety assessments at the implementation stage, including for AI agents and physical AI. The institute also intends to test components such as harnesses, but has not decided which models to assess or how much benchmark information to disclose.

Asked about a series of cyberattacks, Murakami said on October 7 that there was no evidence AI had been involved, while stating that AI is in fact increasing the speed at which vulnerabilities are found. She said defenders need to use AI to keep pace with attackers and that AI can operate 24 hours a day, 365 days a year, at speeds beyond human capability. Because AISI cannot address attack prevention alone, it is considering working with specialist cybersecurity organizations. Murakami said it is difficult for outside parties to determine whether AI was used in an incident. Reports would use existing cybersecurity incident frameworks, while AISI would also track attack methods and AI capabilities.

Japan’s AI Safety Institute (AISI) plans to complete an evaluation of its in-house benchmark for measuring AI model safety by the end of fiscal 2026. Director Akiko Murakami said AISI is still deciding which models to assess and how much of the benchmark to publish. The institute wants to evaluate a broad range of publicly available models, but she said full disclosure could reveal evaluation methods to malicious users.

Established in February 2024, AISI works with relevant government ministries and overseas counterparts to develop AI safety standards and evaluation methods. At a results-sharing meeting for industry proof-of-concept trials on October 7, Murakami said AISI’s aims include promoting trustworthy AI, strengthening competitiveness in the AI field, and deterring national security threats linked to AI development and adoption. Its four areas of work are developing safety benchmarks, building its ability to assess models and systems, collecting, analyzing and sharing AI-related information, and leading international coordination while strengthening competitiveness. Murakami said it is necessary to discuss risks proactively through activities such as sharing trial results, rather than waiting to respond after safety incidents occur.

AISI has already conducted tests using Inspect, an evaluation tool from the UK AISI, and publicly available benchmarks. It plans to build a system for continuously evaluating many models and to expand safety assessments at the implementation stage, including for AI agents and physical AI. The institute also intends to test components such as harnesses, but has not decided which models to assess or how much benchmark information to disclose.

Asked about a series of cyberattacks, Murakami said on October 7 that there was no evidence AI had been involved, while stating that AI is in fact increasing the speed at which vulnerabilities are found. She said defenders need to use AI to keep pace with attackers and that AI can operate 24 hours a day, 365 days a year, at speeds beyond human capability. Because AISI cannot address attack prevention alone, it is considering working with specialist cybersecurity organizations. Murakami said it is difficult for outside parties to determine whether AI was used in an incident. Reports would use existing cybersecurity incident frameworks, while AISI would also track attack methods and AI capabilities.