A new report from the Youth AI Safety Institute at Common Sense Media labels Google’s integrated search AI features as posing an unacceptable risk to children. The study, conducted between May 16 and July 1, evaluated AI Overviews and AI Mode on accounts configured for minors. Researchers performed over 2,500 searches covering topics ranging from homework to sensitive mental health queries and audited more than 2,000 cited sources. According to the findings, these AI tools failed to detect suicidal ideation, mischaracterized symptoms of eating disorders as normal, and provided instructions for generating sexually explicit deepfakes.

The report highlights that these generative AI features are built directly into the default search experience on both personal and school-issued devices, offering no option for parents or administrators to disable them. This design choice forces children to evaluate information accuracy themselves, a task for which they may lack sufficient media literacy. The institute reported that Google violated seven of its eight core AI principles, including those concerning fairness and human safety. Additionally, the study found that 43 per cent of history-related answers were inconsistent across repeated searches, and 29 per cent of citations originated from social media platforms or forums lacking editorial oversight.

Google defended its products, stating that the report focused on a narrow and unrepresentative set of queries that do not accurately reflect general user interactions. A spokesperson noted that the company employs strong quality and safety guardrails, providing extra protection layers for younger users. While Google acknowledges that the AI features cannot be disabled individually, it stated that parents retain the ability to turn off Search entirely for child accounts. The company also claimed it could not verify many of the report's specific examples, asserting that its own internal testing demonstrated higher reliability and quality in responses.

The findings arrive amid broader global scrutiny regarding the protection of minors online, with various regions implementing social media bans for users under 16. John King Jr., a board adviser for the institute and former US education secretary, expressed significant concern over the systemic integration of these features. He emphasized that Google chose to enable these tools by default, without providing mechanisms for educators or parents to restrict access. The Youth AI Safety Institute, which is partially funded by companies such as OpenAI and Anthropic, maintains that its evaluations remain editorially independent.

A new report from the Youth AI Safety Institute at Common Sense Media labels Google’s integrated search AI features as posing an unacceptable risk to children. The study, conducted between May 16 and July 1, evaluated AI Overviews and AI Mode on accounts configured for minors. Researchers performed over 2,500 searches covering topics ranging from homework to sensitive mental health queries and audited more than 2,000 cited sources. According to the findings, these AI tools failed to detect suicidal ideation, mischaracterized symptoms of eating disorders as normal, and provided instructions for generating sexually explicit deepfakes.

The report highlights that these generative AI features are built directly into the default search experience on both personal and school-issued devices, offering no option for parents or administrators to disable them. This design choice forces children to evaluate information accuracy themselves, a task for which they may lack sufficient media literacy. The institute reported that Google violated seven of its eight core AI principles, including those concerning fairness and human safety. Additionally, the study found that 43 per cent of history-related answers were inconsistent across repeated searches, and 29 per cent of citations originated from social media platforms or forums lacking editorial oversight.

Google defended its products, stating that the report focused on a narrow and unrepresentative set of queries that do not accurately reflect general user interactions. A spokesperson noted that the company employs strong quality and safety guardrails, providing extra protection layers for younger users. While Google acknowledges that the AI features cannot be disabled individually, it stated that parents retain the ability to turn off Search entirely for child accounts. The company also claimed it could not verify many of the report's specific examples, asserting that its own internal testing demonstrated higher reliability and quality in responses.

The findings arrive amid broader global scrutiny regarding the protection of minors online, with various regions implementing social media bans for users under 16. John King Jr., a board adviser for the institute and former US education secretary, expressed significant concern over the systemic integration of these features. He emphasized that Google chose to enable these tools by default, without providing mechanisms for educators or parents to restrict access. The Youth AI Safety Institute, which is partially funded by companies such as OpenAI and Anthropic, maintains that its evaluations remain editorially independent.