A study published in the American Journal of Sports Medicine on July 12, 2026, evaluated the ability of a large language model (LLM) to assist in the peer review process. Researchers analyzed 54 manuscripts submitted between September 2024 and October 2024 using a locally deployed Ministral 3 14B model. The AI generated a categorical recommendation and a numerical score (0-100) for each submission to compare against human editorial decisions.

Results showed that pooled human reviewers reached 42.4% agreement with final decisions (κ = 0.181). The AI model exhibited slight, nonsignificant agreement at 37.0% (κ = 0.126), displaying a "revision bias" where it recommended revision for 61.1% of manuscripts, despite 72.7% of those ultimately being rejected or cascaded. However, the AI demonstrated potential as a triage tool; when the model recommended rejection or cascade, 90.5% of manuscripts received such a final decision, yielding a positive predictive value (PPV) of 90.5% and 81.8% specificity. Furthermore, manuscripts receiving an AI score under 70 were rejected or cascaded 88.0% of the time. The authors concluded that while AI cannot replace human nuanced judgment, its high PPV for rejections suggests utility for exploratory screening.

A study published in the American Journal of Sports Medicine on July 12, 2026, evaluated the ability of a large language model (LLM) to assist in the peer review process. Researchers analyzed 54 manuscripts submitted between September 2024 and October 2024 using a locally deployed Ministral 3 14B model. The AI generated a categorical recommendation and a numerical score (0-100) for each submission to compare against human editorial decisions.

Results showed that pooled human reviewers reached 42.4% agreement with final decisions (κ = 0.181). The AI model exhibited slight, nonsignificant agreement at 37.0% (κ = 0.126), displaying a "revision bias" where it recommended revision for 61.1% of manuscripts, despite 72.7% of those ultimately being rejected or cascaded. However, the AI demonstrated potential as a triage tool; when the model recommended rejection or cascade, 90.5% of manuscripts received such a final decision, yielding a positive predictive value (PPV) of 90.5% and 81.8% specificity. Furthermore, manuscripts receiving an AI score under 70 were rejected or cascaded 88.0% of the time. The authors concluded that while AI cannot replace human nuanced judgment, its high PPV for rejections suggests utility for exploratory screening.