Benchmark Finds Gender Blind Spots in AI

The Collective Intelligence Project and Our Secure Future built the WPS AI Leaderboard, a public benchmark testing whether large language models (LLMs) apply Women, Peace and Security (WPS) principles in peacekeeping analysis without being prompted. The researchers report a WPS Competence Gap: models can state the principles when asked but often omit women and gender from routine analysis. They say leaders of peacekeeping missions and military intelligence officers already use LLMs, including to draft situation reports, community engagement plans and conflict analyses.

The WPS agenda dates to UN Security Council Resolution 1325, adopted in October 2000, after women were largely excluded from decisions on conflict and its effects on women and girls went unexamined. The resolution sets commitments on women’s participation, protection from conflict-related violence, prevention of conflict and gendered harm, and relief and recovery. Twenty-five years later, gender advisors and related training and legislation have put these mandates into practice, but advisors may be under-resourced, understaffed or assigned multiple roles.

The benchmark identifies passive omission and active capitulation as two failure modes. In a sparse scenario about 300–400 fighters, ~8,000 displaced people and food prices rising 40% in two weeks, neither model mentioned women, girls or gender, although both produced professional-sounding military assessments. Gemini proposed using displaced people to gather human intelligence and keeping settlements clear of military routes. The authors say displaced populations in the corridor are typically mostly women and children, and that rising food costs can expose female-headed households to risks including survival sex, early marriage and unsafe firewood collection. They also identify women’s community networks as an early-warning resource.

In an adversarial scenario asking how to cut 15% from a mission budget, some models recommended eliminating the Gender Advisor post, shifting its duties to commanders and moving funds to force protection and intelligence collection. One drafted a rationale for the cuts. Other responses challenged the premise, warned that cuts could violate Resolution 1325 and donor agreements, described advisors’ operational contributions, and suggested reviewing duplicated reporting and trainings without measured behavior change.

OSF first assembled 215 National Action Plans, 141 research documents and core WPS doctrine, then tried prompting, retrieval and a prototype WPS agent. The researchers say those approaches depended on users knowing to ask gender-related questions or institutions adding a specialized tool to their workflows. They instead designed a benchmark to test whether systems apply WPS principles without cues and reason about scenarios rather than repeat memorized cases. Its five tiers contain 30 prompts: 8 context-rich, 4 cue-stripped pairs, 7 gender-neutral, 8 sparse or ambiguous, and 3 adversarial prompts.

The scoring framework adds Policy Alignment and Gender/WPS Integration to criteria including operational relevance, analytical depth, actionability, trust building, due diligence, factual integrity and acknowledgment of limitations. WPS Guardrails and Substantive Pushback assess responses to adversarial requests; the overall score averages the five tier means equally. Gemini 2.5 Flash, GPT-4.1 mini and Claude Haiku 4.5 judge responses using Weval’s LLM-as-judge method. In June and July, practitioners and researchers provided more than 140 detailed comments over two discussion rounds, prompting changes to the adversarial tier, cue-stripped pairs, scoring guidance and the definition of gender. The article says all tested frontier models failed to surface gender dimensions in sparse scenarios without explicit cues; it cuts off before giving further results.

The Collective Intelligence Project and Our Secure Future built the WPS AI Leaderboard, a public benchmark testing whether large language models (LLMs) apply Women, Peace and Security (WPS) principles in peacekeeping analysis without being prompted. The researchers report a WPS Competence Gap: models can state the principles when asked but often omit women and gender from routine analysis. They say leaders of peacekeeping missions and military intelligence officers already use LLMs, including to draft situation reports, community engagement plans and conflict analyses.

The WPS agenda dates to UN Security Council Resolution 1325, adopted in October 2000, after women were largely excluded from decisions on conflict and its effects on women and girls went unexamined. The resolution sets commitments on women’s participation, protection from conflict-related violence, prevention of conflict and gendered harm, and relief and recovery. Twenty-five years later, gender advisors and related training and legislation have put these mandates into practice, but advisors may be under-resourced, understaffed or assigned multiple roles.

The benchmark identifies passive omission and active capitulation as two failure modes. In a sparse scenario about 300–400 fighters, ~8,000 displaced people and food prices rising 40% in two weeks, neither model mentioned women, girls or gender, although both produced professional-sounding military assessments. Gemini proposed using displaced people to gather human intelligence and keeping settlements clear of military routes. The authors say displaced populations in the corridor are typically mostly women and children, and that rising food costs can expose female-headed households to risks including survival sex, early marriage and unsafe firewood collection. They also identify women’s community networks as an early-warning resource.

In an adversarial scenario asking how to cut 15% from a mission budget, some models recommended eliminating the Gender Advisor post, shifting its duties to commanders and moving funds to force protection and intelligence collection. One drafted a rationale for the cuts. Other responses challenged the premise, warned that cuts could violate Resolution 1325 and donor agreements, described advisors’ operational contributions, and suggested reviewing duplicated reporting and trainings without measured behavior change.

OSF first assembled 215 National Action Plans, 141 research documents and core WPS doctrine, then tried prompting, retrieval and a prototype WPS agent. The researchers say those approaches depended on users knowing to ask gender-related questions or institutions adding a specialized tool to their workflows. They instead designed a benchmark to test whether systems apply WPS principles without cues and reason about scenarios rather than repeat memorized cases. Its five tiers contain 30 prompts: 8 context-rich, 4 cue-stripped pairs, 7 gender-neutral, 8 sparse or ambiguous, and 3 adversarial prompts.

The scoring framework adds Policy Alignment and Gender/WPS Integration to criteria including operational relevance, analytical depth, actionability, trust building, due diligence, factual integrity and acknowledgment of limitations. WPS Guardrails and Substantive Pushback assess responses to adversarial requests; the overall score averages the five tier means equally. Gemini 2.5 Flash, GPT-4.1 mini and Claude Haiku 4.5 judge responses using Weval’s LLM-as-judge method. In June and July, practitioners and researchers provided more than 140 detailed comments over two discussion rounds, prompting changes to the adversarial tier, cue-stripped pairs, scoring guidance and the definition of gender. The article says all tested frontier models failed to surface gender dimensions in sparse scenarios without explicit cues; it cuts off before giving further results.