Study Evaluates Chatbots on Erectile Dysfunction Information

Researchers compared six large language model chatbot interfaces on patient-facing erectile dysfunction (ED) information in a cross-sectional evaluation published in Sexual Health in 2026. They submitted 38 standardized ED questions once to each interface, producing 228 responses, and five independent raters assessed safety, accuracy, empathy, information quality, source transparency, readability and response length. The evaluation used DISCERN, Ensuring Quality Information for Patients, the Journal of the American Medical Association benchmark and Global Quality Score.

Twenty-six responses (11.4%) were flagged for safety. Flag rates differed across interfaces (Q = 11.765, P = 0.038), but the effect size was small (0.062; 95% CI, 0.026–0.148), and no pairwise difference remained significant after correction for multiple comparisons. Median accuracy was 4.00/5 for every interface, though scores differed slightly overall (P = 0.010, Kendall’s W = 0.080).

Scores differed significantly for empathy (P < 0.001, W = 0.160), DISCERN (P = 0.003, W = 0.508), Ensuring Quality Information for Patients (P < 0.001, W = 0.576) and the JAMA benchmark (P < 0.001, W = 0.888); Global Quality Score did not differ significantly (P = 0.146). Readability and response length also varied, with P < 0.001 for both. The researchers concluded that chatbots generally provided accurate ED information, but performance varied across clinically relevant measures; they may support information seeking, but should not replace individualized clinical assessment.

Researchers compared six large language model chatbot interfaces on patient-facing erectile dysfunction (ED) information in a cross-sectional evaluation published in Sexual Health in 2026. They submitted 38 standardized ED questions once to each interface, producing 228 responses, and five independent raters assessed safety, accuracy, empathy, information quality, source transparency, readability and response length. The evaluation used DISCERN, Ensuring Quality Information for Patients, the Journal of the American Medical Association benchmark and Global Quality Score.

Twenty-six responses (11.4%) were flagged for safety. Flag rates differed across interfaces (Q = 11.765, P = 0.038), but the effect size was small (0.062; 95% CI, 0.026–0.148), and no pairwise difference remained significant after correction for multiple comparisons. Median accuracy was 4.00/5 for every interface, though scores differed slightly overall (P = 0.010, Kendall’s W = 0.080).

Scores differed significantly for empathy (P < 0.001, W = 0.160), DISCERN (P = 0.003, W = 0.508), Ensuring Quality Information for Patients (P < 0.001, W = 0.576) and the JAMA benchmark (P < 0.001, W = 0.888); Global Quality Score did not differ significantly (P = 0.146). Readability and response length also varied, with P < 0.001 for both. The researchers concluded that chatbots generally provided accurate ED information, but performance varied across clinically relevant measures; they may support information seeking, but should not replace individualized clinical assessment.