Study Compares Chatbots With Audiologists

A blinded comparative study published in the American Journal of Audiology in 2026 compared ChatGPT 3.5 and Gemini 2.0 Flash with qualified audiologists on audiological case interpretation and management recommendations. The researchers used a two-phase, cross-sectional design and an expert-rated quality tool called CLEAR, which scores completeness, lack of false information, evidence, appropriateness and relevance.

In the first phase, 10 audiologists with more than 2 years of experience and two chatbots interpreted 15 anonymized adult cases, producing 60 interpretations. Each interpretation answered questions about diagnosis and management, for 120 responses in total. In the second phase, 30 audiologists independently rated all 120 responses without knowing who produced them. Five raters assessed each response, yielding 600 ratings.

Mixed-effects modeling found no statistically significant difference between chatbot-generated and audiologist-generated responses on CLEAR scores; all p-values were greater than .22. Absolute differences were no more than 1.2 points on the 5-25 scale, and 95% confidence intervals crossed zero. Single-rater agreement was poor (ICC = .17-.18), while aggregated ratings showed moderate reliability (ICC = .50-.53). Internal consistency was excellent (Cronbach's alpha = .94-.95). The researchers said the results do not establish formal equivalence, but provide an empirical benchmark under controlled conditions.

A blinded comparative study published in the American Journal of Audiology in 2026 compared ChatGPT 3.5 and Gemini 2.0 Flash with qualified audiologists on audiological case interpretation and management recommendations. The researchers used a two-phase, cross-sectional design and an expert-rated quality tool called CLEAR, which scores completeness, lack of false information, evidence, appropriateness and relevance.

In the first phase, 10 audiologists with more than 2 years of experience and two chatbots interpreted 15 anonymized adult cases, producing 60 interpretations. Each interpretation answered questions about diagnosis and management, for 120 responses in total. In the second phase, 30 audiologists independently rated all 120 responses without knowing who produced them. Five raters assessed each response, yielding 600 ratings.

Mixed-effects modeling found no statistically significant difference between chatbot-generated and audiologist-generated responses on CLEAR scores; all p-values were greater than .22. Absolute differences were no more than 1.2 points on the 5-25 scale, and 95% confidence intervals crossed zero. Single-rater agreement was poor (ICC = .17-.18), while aggregated ratings showed moderate reliability (ICC = .50-.53). Internal consistency was excellent (Cronbach's alpha = .94-.95). The researchers said the results do not establish formal equivalence, but provide an empirical benchmark under controlled conditions.