A descriptive study published in the Archives of Orthopaedic and Trauma Surgery on July 7, 2026, evaluated the clinical reasoning performance of three contemporary large language models: ChatGPT-5, Gemini 2.5 Flash, and Grok-3. Researchers Suzen Agharia, Shayan Soroush, and Daniel Ameen tested these models against responses from thousands of clinicians using 97 multiple-choice cases across eight orthopaedic subspecialties sourced from OrthoBullets. The study measured how frequently AI responses matched the most popular clinician answer, utilizing the same methodology applied to a 2023 evaluation of earlier models.

Gemini 2.5 Flash demonstrated the highest alignment with clinician consensus at 69.1%, followed by Grok-3 at 66.0% and ChatGPT-5 at 58.8%. While prior-generation models showed a 7.2% refusal rate, the contemporary models evaluated here refused none of the prompts. In subspecialty testing, Gemini 2.5 Flash outperformed others in Hand and Paediatric domains, whereas Grok-3 showed superior results in Reconstruction, Trauma, and controversial cases (defined by less than a 25% margin of consensus). Inter-model agreement was highest between Grok-3 and Gemini 2.5 Flash, with a Cohen’s kappa coefficient of 0.678, suggesting increased consistency compared to previous iterations. The researchers concluded that while these models show potential for orthopaedic education and simulating peer reasoning, they are not currently suitable for independent clinical use due to lingering limitations in reasoning accuracy.

A descriptive study published in the Archives of Orthopaedic and Trauma Surgery on July 7, 2026, evaluated the clinical reasoning performance of three contemporary large language models: ChatGPT-5, Gemini 2.5 Flash, and Grok-3. Researchers Suzen Agharia, Shayan Soroush, and Daniel Ameen tested these models against responses from thousands of clinicians using 97 multiple-choice cases across eight orthopaedic subspecialties sourced from OrthoBullets. The study measured how frequently AI responses matched the most popular clinician answer, utilizing the same methodology applied to a 2023 evaluation of earlier models.

Gemini 2.5 Flash demonstrated the highest alignment with clinician consensus at 69.1%, followed by Grok-3 at 66.0% and ChatGPT-5 at 58.8%. While prior-generation models showed a 7.2% refusal rate, the contemporary models evaluated here refused none of the prompts. In subspecialty testing, Gemini 2.5 Flash outperformed others in Hand and Paediatric domains, whereas Grok-3 showed superior results in Reconstruction, Trauma, and controversial cases (defined by less than a 25% margin of consensus). Inter-model agreement was highest between Grok-3 and Gemini 2.5 Flash, with a Cohen’s kappa coefficient of 0.678, suggesting increased consistency compared to previous iterations. The researchers concluded that while these models show potential for orthopaedic education and simulating peer reasoning, they are not currently suitable for independent clinical use due to lingering limitations in reasoning accuracy.