Benchmarks/AIB DecisionBench-Agent · agent-0.1

Korean & Japanese judgment ranking Real-time game skill ranking

Can decision AI serve as the router and guard of an agent?

AIB DecisionBench-Agent measures the working roles decision AI takes on inside AI agents. It measures how well a model serves as the router, which decides whether to call a tool, and the guard, which stops harmful requests and answers, and uses hard knowledge questions to show the areas where decision models reach their ceiling.

23
Models: 3
Tests: 2,508
Items: 80,691
Decisions

Design intent

Decision AI does not write text; it picks one of a set of options, with a probability, quickly and cheaply. That is why agents often hand it calls such as where to route a request, whether a tool may be called and whether content is harmful. We check whether it can actually do these jobs, using public tests that researchers already rely on.

How it is measured

Items are drawn from three public datasets with a fixed seed, and every AI model gets the same items in the same format. Items with options are asked a second time in reversed order, to see whether the option position sways the answer. We don't republish the items, but the datasets, versions and seed are listed, so anyone can re-run the measurement under the same conditions.

Metrics

Overall score — the router score and the guard score, averaged with equal weight, out of 100
Router (When2Call) — accuracy at choosing whether to call a tool, ask for more information, or say it can't help
Guard (WildGuardTest) — average F1 score at telling whether a request is harmful, whether a response is harmful, and whether the response refuses
Hard knowledge (HLE multiple choice) — accuracy on very hard knowledge questions, shown separately and kept out of the score
Response time and cost — kept out of the score, shown in their own columns

In the October 17 – 24, 2026 run, the top overall score among 23 decision AIs went to D1 Ultra (92.8).

Best router: D1 Ultra — router score 89.8
Best guard: Drex Max — guard score 96.7
Decider 1 most often answered right away when it should have used a tool or asked for information (always wrong, 24.6%)

Overall ranking

🥇Liquid AID1 Ultra92.8±1.6
Router role
89.8
Guard role
95.9
Hard knowledge
16.1
Time · 1,000
10m 10s
Cost · 1,000
$0.024
🥈PerplexityPerplexity Decider v1.1 27B92.2±1.6
Router role
89.2
Guard role
95.2
Hard knowledge
15.4
Time · 1,000
10m 15s
Cost · 1,000
$0.013
🥉PerplexityDecider V1.292.2±1.6
Router role
89.2
Guard role
95.1
Hard knowledge
15.4
Time · 1,000
10m 28s
Cost · 1,000
$0.013
#4Liquid AID190.2±1.8
Router role
86.2
Guard role
94.2
Hard knowledge
12.8
Time · 1,000
14m 52s
Cost · 1,000
$0.011
#5TypeSafeJev 1.1490.0±1.8
Router role
85.4
Guard role
94.6
Hard knowledge
19.5
Time · 1,000
7m 12s
Cost · 1,000
$0.039
#6Nace AIDrex Max89.8±1.8
Router role
83.0
Guard role
96.7
Hard knowledge
19.3
Time · 1,000
9m 23s
Cost · 1,000
$0.039
#7TypeSafeJev 1.1389.2±1.8
Router role
83.6
Guard role
94.7
Hard knowledge
20.9
Time · 1,000
7m 2s
Cost · 1,000
$0.039
#8PerplexityDecider Flash V1.288.6±1.9
Router role
84.0
Guard role
93.2
Hard knowledge
9.6
Time · 1,000
7m 46s
Cost · 1,000
$0.0065
#9MicrosoftMicrosoft-Decision-188.0±2.0
Router role
84.4
Guard role
91.6
Hard knowledge
12.2
Time · 1,000
11m 13s
Cost · 1,000
$0.023
#10Nace AIDrex v2.087.5±1.9
Router role
84.4
Guard role
90.5
Hard knowledge
14.0
Time · 1,000
10m 48s
Cost · 1,000
$0.035
#11Nace AIDrex v1.587.0±1.9
Router role
86.4
Guard role
87.6
Hard knowledge
11.6
Time · 1,000
12m 23s
Cost · 1,000
$0.023
#12RespanSpan-0286.3±2.2
Router role
87.8
Guard role
84.9
Hard knowledge
11.4
Time · 1,000
16m 20s
Cost · 1,000
$0.0058
#13InceptionMercury Decide86.0±2.0
Router role
79.0
Guard role
92.9
Hard knowledge
11.4
Time · 1,000
10m 12s
Cost · 1,000
Free for now
#14CloudflareClef85.4±2.1
Router role
76.0
Guard role
94.9
Hard knowledge
12.8
Time · 1,000
8m 22s
Cost · 1,000
$0.166
#15OpenAIGPT-6 Luna Decisions83.5±2.2
Router role
76.6
Guard role
90.4
Hard knowledge
12.8
Time · 1,000
4m 24s
Cost · 1,000
$0.065
#16UpstageSolar Decide80.6±2.3
Router role
74.2
Guard role
87.0
Hard knowledge
14.6
Time · 1,000
1h 20m
Cost · 1,000
$0.044
#17CloudflareClef Omni80.4±2.3
Router role
73.0
Guard role
87.7
Hard knowledge
9.8
Time · 1,000
5m 49s
Cost · 1,000
$0.103
#18Jared PalmerKev 4B71.5±2.6
Router role
68.2
Guard role
74.8
Hard knowledge
9.8
Time · 1,000
13m 54s
Cost · 1,000
$0.024
#19UpstageSolar Decide Flash71.4±2.5
Router role
61.2
Guard role
81.6
Hard knowledge
14.0
Time · 1,000
8m 37s
Cost · 1,000
$0.044
#20CloudflareClef Flash71.0±2.8
Router role
75.4
Guard role
66.5
Hard knowledge
11.0
Time · 1,000
7m 5s
Cost · 1,000
$0.061
#21ScaleDownClassify-167.0±2.6
Router role
46.6
Guard role
87.3
Hard knowledge
12.4
Time · 1,000
13m 51s
Cost · 1,000
$0.037
#22meraGPTDecider 147.7±2.5
Router role
30.2
Guard role
65.2
Hard knowledge
—
Time · 1,000
11m 8s
Cost · 1,000
$0.029
#23AlibabaDecision Model Preview29.6±2.8
Router role
40.4
Guard role
18.8
Hard knowledge
11.6
Time · 1,000
7m 9s
Cost · 1,000
Free for now

Rank	Model	Overall score	Router role	Guard role	Hard knowledge	Time · 1,000	Cost · 1,000
🥇	Liquid AID1 Ultra	92.8±1.6	89.8	95.9	16.1	10m 10s	$0.024
🥈	PerplexityPerplexity Decider v1.1 27B	92.2±1.6	89.2	95.2	15.4	10m 15s	$0.013
🥉	PerplexityDecider V1.2	92.2±1.6	89.2	95.1	15.4	10m 28s	$0.013
#4	Liquid AID1	90.2±1.8	86.2	94.2	12.8	14m 52s	$0.011
#5	TypeSafeJev 1.14	90.0±1.8	85.4	94.6	19.5	7m 12s	$0.039
#6	Nace AIDrex Max	89.8±1.8	83.0	96.7	19.3	9m 23s	$0.039
#7	TypeSafeJev 1.13	89.2±1.8	83.6	94.7	20.9	7m 2s	$0.039
#8	PerplexityDecider Flash V1.2	88.6±1.9	84.0	93.2	9.6	7m 46s	$0.0065
#9	MicrosoftMicrosoft-Decision-1	88.0±2.0	84.4	91.6	12.2	11m 13s	$0.023
#10	Nace AIDrex v2.0	87.5±1.9	84.4	90.5	14.0	10m 48s	$0.035
#11	Nace AIDrex v1.5	87.0±1.9	86.4	87.6	11.6	12m 23s	$0.023
#12	RespanSpan-02	86.3±2.2	87.8	84.9	11.4	16m 20s	$0.0058
#13	InceptionMercury Decide	86.0±2.0	79.0	92.9	11.4	10m 12s	Free for now
#14	CloudflareClef	85.4±2.1	76.0	94.9	12.8	8m 22s	$0.166
#15	OpenAIGPT-6 Luna Decisions	83.5±2.2	76.6	90.4	12.8	4m 24s	$0.065
#16	UpstageSolar Decide	80.6±2.3	74.2	87.0	14.6	1h 20m	$0.044
#17	CloudflareClef Omni	80.4±2.3	73.0	87.7	9.8	5m 49s	$0.103
#18	Jared PalmerKev 4B	71.5±2.6	68.2	74.8	9.8	13m 54s	$0.024
#19	UpstageSolar Decide Flash	71.4±2.5	61.2	81.6	14.0	8m 37s	$0.044
#20	CloudflareClef Flash	71.0±2.8	75.4	66.5	11.0	7m 5s	$0.061
#21	ScaleDownClassify-1	67.0±2.6	46.6	87.3	12.4	13m 51s	$0.037
#22	meraGPTDecider 1	47.7±2.5	30.2	65.2	—	11m 8s	$0.029
#23	AlibabaDecision Model Preview	29.6±2.8	40.4	18.8	11.6	7m 9s	Free for now

The overall score averages the router (When2Call accuracy) and the guard (WildGuardTest F1) with equal weight. Hard knowledge (HLE multiple-choice accuracy), time and cost are shown for reference and do not count toward the score. ± is a 95% interval from resampling the items, and items the provider refused to judge count as wrong.

Overall score × cost

Overall score × response time

Overall score × cost

Overall score × response time

Reliability

D1 Ultraoff by 5.5 pts on average · overconfident

Perplexity Decider v1.1 27Boff by 8.6 pts on average · overconfident

Decider V1.2off by 8.4 pts on average · overconfident

D1off by 8.7 pts on average · overconfident

Jev 1.14off by 6.5 pts on average · overconfident

Drex Maxoff by 10.2 pts on average · overconfident

Jev 1.13off by 8.0 pts on average · overconfident

Decider Flash V1.2off by 12.0 pts on average · overconfident

Microsoft-Decision-1off by 8.7 pts on average · overconfident

Drex v2.0off by 6.5 pts on average · overconfident

Drex v1.5off by 10.3 pts on average · overconfident

Span-02off by 13.7 pts on average · overconfident

Mercury Decideoff by 14.9 pts on average · overconfident

Clefoff by 9.0 pts on average · overconfident

GPT-6 Luna Decisionsoff by 12.4 pts on average · overconfident

Solar Decideoff by 15.8 pts on average · overconfident

Clef Omnioff by 8.7 pts on average · overconfident

Kev 4Boff by 6.7 pts on average · overconfident

Solar Decide Flashoff by 12.3 pts on average · overconfident

Clef Flashoff by 11.9 pts on average · overconfident

Classify-1off by 5.8 pts on average · overconfident

Decider 1off by 12.3 pts on average · overconfident

Decision Model Previewoff by 39.9 pts on average · overconfident

Each dot = a confidence range · size = number of answers in it

Datasets

Router role — When2Call evaluation details

Given a user message and the list of available tools, the model picks the right one of four replies: call a tool, ask for more information, say it can't help, or answer straight away. The correct reply is always one of the first three; answering straight away makes something up without the tool, so it is always wrong.

When2Call

English

	D1 Ultra	Perplexity Decider v1.1 27B	Decider V1.2	D1	Jev 1.14	Drex Max	Jev 1.13	Decider Flash V1.2	Microsoft-Decision-1	Drex v2.0	Drex v1.5	Span-02	Mercury Decide	Clef	GPT-6 Luna Decisions	Solar Decide	Clef Omni	Kev 4B	Solar Decide Flash	Clef Flash	Classify-1	Decider 1	Decision Model Preview
Overall · Accuracy	89.8(500)	89.2(500)	89.2(500)	86.2(500)	85.4(500)	83.0(500)	83.6(500)	84.0(500)	84.4(500)	84.4(500)	86.4(500)	87.8(500)	79.0(500)	76.0(500)	76.6(500)	74.2(500)	73.0(500)	68.2(500)	61.2(500)	75.4(500)	46.6(500)	30.2(500)	40.4(500)
Direct-answer rate	0.6%	0.8%	1.0%	2.4%	2.4%	0.4%	1.8%	1.8%	2.8%	0.4%	1.0%	1.4%	6.4%	6.4%	5.0%	4.6%	7.4%	3.2%	6.2%	8.6%	3.2%	24.6%	8.8%
Option-order flips	3.8% (500)	3.0% (500)	2.8% (500)	5.6% (500)	2.0% (500)	8.0% (500)	2.6% (500)	5.6% (500)	1.6% (500)	7.6% (500)	10.8% (500)	6.8% (500)	10.4% (500)	0.0% (500)	18.4% (500)	17.0% (500)	0.0% (500)	19.0% (500)	27.4% (500)	0.0% (500)	22.4% (500)	42.6% (500)	20.4% (500)
Confidence vs. accuracy gap	2.8 pp	5.0 pp	4.7 pp	7.8 pp	6.0 pp	9.0 pp	6.2 pp	10.7 pp	8.2 pp	6.7 pp	5.1 pp	5.0 pp	13.2 pp	9.0 pp	9.2 pp	20.2 pp	5.6 pp	4.9 pp	18.6 pp	3.4 pp	23.4 pp	28.7 pp	37.4 pp
Median latency	599ms	637ms	553ms	1.2s	388ms	484ms	407ms	460ms	642ms	532ms	589ms	651ms	375ms	425ms	238ms	567ms	333ms	810ms	514ms	461ms	626ms	626ms	422ms
Correct reply
Use the tool	89.2(166)	93.4(166)	93.4(166)	91.6(166)	91.6(166)	89.8(166)	91.6(166)	91.6(166)	92.8(166)	88.0(166)	81.9(166)	89.2(166)	89.8(166)	72.9(166)	90.4(166)	89.2(166)	69.3(166)	86.1(166)	83.7(166)	80.1(166)	94.6(166)	52.4(166)	95.8(166)
Ask for information	86.8(167)	80.8(167)	81.4(167)	82.0(167)	82.0(167)	79.0(167)	80.2(167)	83.2(167)	79.0(167)	89.8(167)	91.6(167)	88.0(167)	78.4(167)	84.4(167)	71.9(167)	70.7(167)	85.0(167)	78.4(167)	63.5(167)	82.6(167)	26.9(167)	31.1(167)	18.6(167)
Say it can't help	93.4(167)	93.4(167)	92.8(167)	85.0(167)	82.6(167)	80.2(167)	79.0(167)	77.2(167)	81.4(167)	75.4(167)	85.6(167)	86.2(167)	68.9(167)	70.7(167)	67.7(167)	62.9(167)	64.7(167)	40.1(167)	36.5(167)	63.5(167)	18.6(167)	7.2(167)	7.2(167)

Guard role — WildGuardTest evaluation details

The model reads a user request and an AI response and answers three yes/no questions: is the request harmful, is the response harmful, and does the response refuse the request.

Only a small share of rows have a harmful response, so answering "no" every time still scores high on accuracy. So instead of accuracy, the guard is scored with an F1 score that counts both misses and false flags, as in the original paper.

WildGuardTest — harmful request

English

	D1 Ultra	Perplexity Decider v1.1 27B	Decider V1.2	D1	Jev 1.14	Drex Max	Jev 1.13	Decider Flash V1.2	Microsoft-Decision-1	Drex v2.0	Drex v1.5	Span-02	Mercury Decide	Clef	GPT-6 Luna Decisions	Solar Decide	Clef Omni	Kev 4B	Solar Decide Flash	Clef Flash	Classify-1	Decider 1	Decision Model Preview
Overall · Accuracy	97.0(500)	96.2(500)	96.0(500)	94.6(500)	93.4(500)	97.4(500)	93.2(500)	95.6(500)	94.6(500)	87.0(500)	88.8(500)	87.0(500)	94.8(500)	94.0(500)	85.0(500)	88.4(500)	89.2(500)	92.2(500)	85.4(500)	78.4(500)	92.0(500)	83.4(500)	35.2(500)
Guard score (F1)	96.9	96.1	95.9	94.4	93.1	97.4	92.8	95.5	94.6	85.3	88.5	85.0	94.6	93.8	82.4	87.1	88.4	92.0	83.1	74.8	91.4	83.2	28.6
Confidence vs. accuracy gap	2.7 pp	1.3 pp	2.0 pp	2.0 pp	3.9 pp	0.8 pp	3.4 pp	1.5 pp	2.7 pp	4.7 pp	3.4 pp	9.0 pp	2.9 pp	1.2 pp	11.4 pp	7.3 pp	3.2 pp	18.6 pp	4.7 pp	6.6 pp	15.3 pp	5.7 pp	38.6 pp
Median latency	576ms	399ms	525ms	545ms	384ms	462ms	393ms	449ms	612ms	646ms	484ms	643ms	308ms	379ms	224ms	512ms	309ms	709ms	506ms	225ms	584ms	469ms	385ms
Request style
Disguised request	94.0(249)	92.8(249)	92.4(249)	90.8(249)	89.6(249)	95.2(249)	90.0(249)	92.0(249)	91.2(249)	83.5(249)	86.3(249)	80.7(249)	90.0(249)	90.4(249)	78.3(249)	84.7(249)	88.0(249)	90.0(249)	82.3(249)	82.7(249)	86.3(249)	77.9(249)	40.6(249)
Direct request	100.0(251)	99.6(251)	99.6(251)	98.4(251)	97.2(251)	99.6(251)	96.4(251)	99.2(251)	98.0(251)	90.4(251)	91.2(251)	93.2(251)	99.6(251)	97.6(251)	91.6(251)	92.0(251)	90.4(251)	94.4(251)	88.4(251)	74.1(251)	97.6(251)	88.8(251)	29.9(251)
Request type
Harmful request	94.8(249)	93.2(249)	93.2(249)	92.0(249)	89.2(249)	96.4(249)	88.0(249)	94.4(249)	94.8(249)	75.9(249)	86.7(249)	73.9(249)	91.6(249)	91.6(249)	70.7(249)	78.3(249)	82.3(249)	89.6(249)	72.3(249)	64.3(249)	85.5(249)	82.7(249)	26.1(249)
Benign request	99.2(251)	99.2(251)	98.8(251)	97.2(251)	97.6(251)	98.4(251)	98.4(251)	96.8(251)	94.4(251)	98.0(251)	90.8(251)	100.0(251)	98.0(251)	96.4(251)	99.2(251)	98.4(251)	96.0(251)	94.8(251)	98.4(251)	92.4(251)	98.4(251)	84.1(251)	44.2(251)

WildGuardTest — harmful response

English

	D1 Ultra	Perplexity Decider v1.1 27B	Decider V1.2	D1	Jev 1.14	Drex Max	Jev 1.13	Decider Flash V1.2	Microsoft-Decision-1	Drex v2.0	Drex v1.5	Span-02	Mercury Decide	Clef	GPT-6 Luna Decisions	Solar Decide	Clef Omni	Kev 4B	Solar Decide Flash	Clef Flash	Classify-1	Decider 1	Decision Model Preview
Overall · Accuracy	97.8(500)	97.4(500)	97.4(500)	96.4(500)	97.6(500)	98.4(500)	97.8(500)	97.2(500)	92.6(500)	96.0(500)	94.8(500)	92.4(500)	95.8(500)	97.6(500)	96.6(500)	94.2(500)	95.4(500)	92.0(500)	91.6(500)	86.2(500)	91.0(500)	65.2(500)	68.4(500)
Guard score (F1)	94.3	93.0	93.0	90.3	93.8	95.9	94.2	92.5	83.4	88.8	85.2	75.0	89.3	93.9	91.2	85.4	87.6	81.0	80.0	68.8	75.7	42.0	16.8
Confidence vs. accuracy gap	2.3 pp	1.0 pp	0.6 pp	1.2 pp	5.3 pp	0.8 pp	4.5 pp	1.9 pp	2.8 pp	4.4 pp	1.6 pp	5.6 pp	2.6 pp	1.6 pp	1.9 pp	2.3 pp	1.2 pp	12.6 pp	1.9 pp	2.1 pp	18.2 pp	16.2 pp	10.8 pp
Median latency	510ms	614ms	535ms	568ms	376ms	470ms	392ms	454ms	626ms	542ms	494ms	626ms	342ms	488ms	237ms	508ms	319ms	775ms	503ms	294ms	618ms	430ms	400ms
Request style
Disguised request	95.6(249)	96.0(249)	96.0(249)	94.0(249)	96.0(249)	96.8(249)	96.8(249)	95.6(249)	88.0(249)	94.4(249)	92.8(249)	88.4(249)	93.6(249)	95.6(249)	94.0(249)	92.0(249)	94.0(249)	86.7(249)	90.4(249)	90.0(249)	86.7(249)	62.7(249)	66.7(249)
Direct request	100.0(251)	98.8(251)	98.8(251)	98.8(251)	99.2(251)	100.0(251)	98.8(251)	98.8(251)	97.2(251)	97.6(251)	96.8(251)	96.4(251)	98.0(251)	99.6(251)	99.2(251)	96.4(251)	96.8(251)	97.2(251)	92.8(251)	82.5(251)	95.2(251)	67.7(251)	70.1(251)
Request type
Harmful request	95.6(249)	94.8(249)	94.8(249)	93.6(249)	95.2(249)	96.8(249)	95.6(249)	94.4(249)	95.2(249)	92.0(249)	90.4(249)	85.1(249)	94.8(249)	95.6(249)	94.8(249)	89.2(249)	92.0(249)	85.1(249)	83.9(249)	88.8(249)	83.9(249)	36.9(249)	61.0(249)
Benign request	100.0(251)	100.0(251)	100.0(251)	99.2(251)	100.0(251)	100.0(251)	100.0(251)	100.0(251)	90.0(251)	100.0(251)	99.2(251)	99.6(251)	96.8(251)	99.6(251)	98.4(251)	99.2(251)	98.8(251)	98.8(251)	99.2(251)	83.7(251)	98.0(251)	93.2(251)	75.7(251)

WildGuardTest — refusal

English

	D1 Ultra	Perplexity Decider v1.1 27B	Decider V1.2	D1	Jev 1.14	Drex Max	Jev 1.13	Decider Flash V1.2	Microsoft-Decision-1	Drex v2.0	Drex v1.5	Span-02	Mercury Decide	Clef	GPT-6 Luna Decisions	Solar Decide	Clef Omni	Kev 4B	Solar Decide Flash	Clef Flash	Classify-1	Decider 1	Decision Model Preview
Overall · Accuracy	97.6(500)	97.6(500)	97.6(500)	98.6(500)	98.0(500)	97.8(500)	98.2(500)	94.2(500)	97.8(500)	98.4(500)	92.2(500)	96.4(500)	96.6(500)	98.0(500)	98.4(500)	91.8(500)	90.8(500)	53.4(500)	86.6(500)	64.8(500)	96.6(500)	79.0(500)	14.6(500)
Guard score (F1)	96.4	96.4	96.4	97.8	97.0	96.7	97.2	91.6	96.7	97.5	89.1	94.6	94.9	96.9	97.5	88.5	87.1	51.4	81.8	56.0	94.9	70.3	10.9
Confidence vs. accuracy gap	0.8 pp	0.8 pp	1.0 pp	1.3 pp	3.9 pp	0.9 pp	2.4 pp	3.3 pp	3.4 pp	4.3 pp	6.0 pp	1.0 pp	2.1 pp	1.7 pp	1.0 pp	3.2 pp	3.1 pp	9.7 pp	3.3 pp	12.7 pp	15.9 pp	9.4 pp	68.9 pp
Median latency	515ms	391ms	544ms	563ms	381ms	484ms	408ms	458ms	629ms	534ms	529ms	600ms	368ms	389ms	226ms	514ms	325ms	730ms	507ms	291ms	566ms	529ms	401ms
Request style
Disguised request	97.2(249)	97.2(249)	97.2(249)	98.4(249)	98.0(249)	97.6(249)	98.4(249)	94.8(249)	97.2(249)	98.4(249)	93.2(249)	96.0(249)	96.0(249)	99.2(249)	98.4(249)	94.4(249)	95.2(249)	57.8(249)	88.8(249)	74.3(249)	94.8(249)	83.1(249)	10.0(249)
Direct request	98.0(251)	98.0(251)	98.0(251)	98.8(251)	98.0(251)	98.0(251)	98.0(251)	93.6(251)	98.4(251)	98.4(251)	91.2(251)	96.8(251)	97.2(251)	96.8(251)	98.4(251)	89.2(251)	86.5(251)	49.0(251)	84.5(251)	55.4(251)	98.4(251)	74.9(251)	19.1(251)
Request type
Harmful request	99.2(249)	99.2(249)	99.2(249)	99.2(249)	98.0(249)	100.0(249)	98.4(249)	94.4(249)	98.8(249)	98.0(249)	93.6(249)	97.2(249)	99.2(249)	99.6(249)	98.8(249)	92.8(249)	88.4(249)	62.7(249)	86.3(249)	79.1(249)	96.8(249)	68.7(249)	21.7(249)
Benign request	96.0(251)	96.0(251)	96.0(251)	98.0(251)	98.0(251)	95.6(251)	98.0(251)	94.0(251)	96.8(251)	98.8(251)	90.8(251)	95.6(251)	94.0(251)	96.4(251)	98.0(251)	90.8(251)	93.2(251)	44.2(251)	86.9(251)	50.6(251)	96.4(251)	89.2(251)	7.6(251)

Hard knowledge — HLE evaluation details

The router and guard are jobs decision AI actually does inside agents, so they make up the overall score. The hard knowledge test is the opposite: a reference for whether judgments that need expert knowledge can be left to decision AI or should go to a general LLM. It uses the text-only multiple-choice questions from Humanity's Last Exam, hard knowledge questions written by experts; with 5–26 options each, random guessing gets 16.5% on average. Because it measures a different ability from the roles, it is shown separately from the overall score.

Choosing at random would be right 16.5% of the time. Of the 23 models that finished, 3 scored higher than that.

Not measurable: the model's limit on options or input length kept it from taking more than 2% of this test's items. Counting only the items it could take would leave the ones with fewer options or shorter text and inflate the score, so this column is left empty. At 2% or less, only the items it couldn't take are left out.

Humanity's Last Exam — multiple choice (text only)

English

	D1 Ultra	Perplexity Decider v1.1 27B	Decider V1.2	D1	Jev 1.14	Drex Max	Jev 1.13	Decider Flash V1.2	Microsoft-Decision-1	Drex v2.0	Drex v1.5	Span-02	Mercury Decide	Clef	GPT-6 Luna Decisions	Solar Decide	Clef Omni	Kev 4B	Solar Decide Flash	Clef Flash	Classify-1	Decider 1	Decision Model Preview
Overall · Accuracy	16.1(508)	15.4(508)	15.4(508)	12.8(508)	19.5(508)	19.3(508)	20.9(508)	9.6(508)	12.2(507)	14.0(508)	11.6(508)	11.4(508)	11.4(508)	12.8(508)	12.8(508)	14.6(508)	9.8(508)	9.8(508)	14.0(508)	11.0(507)	12.4(508)	—	11.6(508)
vs. random	−0.4	−1.1	−1.1	−3.7	+3.0	+2.8	+4.4	−6.9	−4.3	−2.5	−4.9	−5.1	−5.1	−3.7	−3.7	−1.9	−6.7	−6.7	−2.5	−5.5	−4.1	—	−4.9
Option-order flips	34.1% (508)	27.8% (508)	28.0% (508)	43.9% (508)	17.1% (508)	31.9% (508)	15.6% (508)	31.5% (508)	7.1% (507)	46.9% (508)	46.1% (508)	45.1% (508)	48.2% (508)	0.0% (508)	40.4% (503)	65.9% (508)	0.0% (508)	49.2% (508)	61.8% (508)	0.0% (507)	60.4% (508)	—	53.5% (508)
Confidence vs. accuracy gap	32.9 pp	36.1 pp	35.4 pp	38.7 pp	37.6 pp	42.3 pp	41.7 pp	45.3 pp	43.5 pp	31.6 pp	48.1 pp	48.5 pp	53.6 pp	39.1 pp	42.6 pp	46.9 pp	38.6 pp	25.6 pp	38.4 pp	36.7 pp	27.4 pp	—	43.8 pp
Median latency	513ms	373ms	537ms	524ms	403ms	473ms	407ms	452ms	630ms	555ms	919ms	913ms	328ms	375ms	225ms	11.4s	347ms	809ms	534ms	301ms	609ms	—	416ms
Field
Math	20.2(89)	15.7(89)	13.5(89)	16.9(89)	33.7(89)	22.5(89)	33.7(89)	13.5(89)	18.2(88)	11.2(89)	7.9(89)	10.1(89)	5.6(89)	13.5(89)	14.6(89)	14.6(89)	9.0(89)	9.0(89)	13.5(89)	12.5(88)	15.7(89)	—	12.4(89)
Biology/Medicine	13.7(146)	15.1(146)	13.7(146)	10.3(146)	11.0(146)	16.4(146)	15.1(146)	11.0(146)	13.0(146)	15.8(146)	15.8(146)	9.6(146)	13.7(146)	11.0(146)	12.3(146)	11.0(146)	8.9(146)	11.6(146)	13.0(146)	12.3(146)	12.3(146)	—	13.7(146)
Computer Science/AI	19.0(63)	17.5(63)	20.6(63)	17.5(63)	19.0(63)	30.2(63)	22.2(63)	3.2(63)	12.7(63)	17.5(63)	9.5(63)	12.7(63)	20.6(63)	9.5(63)	12.7(63)	12.7(63)	12.7(63)	4.8(63)	15.9(63)	7.9(63)	9.5(63)	—	6.3(63)
Physics	13.5(37)	18.9(37)	13.5(37)	8.1(37)	21.6(37)	16.2(37)	18.9(37)	10.8(37)	2.7(37)	8.1(37)	10.8(37)	10.8(37)	16.2(37)	18.9(37)	8.1(37)	18.9(37)	10.8(37)	8.1(37)	16.2(37)	8.1(37)	13.5(37)	—	8.1(37)
Chemistry	15.4(26)	19.2(26)	19.2(26)	7.7(26)	23.1(26)	30.8(26)	34.6(26)	7.7(26)	15.4(26)	3.8(26)	15.4(26)	7.7(26)	7.7(26)	11.5(26)	7.7(26)	7.7(26)	11.5(26)	7.7(26)	11.5(26)	15.4(26)	11.5(26)	—	19.2(26)
Engineering	8.3(24)	4.2(24)	4.2(24)	12.5(24)	8.3(24)	4.2(24)	8.3(24)	4.2(24)	12.5(24)	16.7(24)	16.7(24)	12.5(24)	8.3(24)	8.3(24)	12.5(24)	16.7(24)	4.2(24)	4.2(24)	12.5(24)	4.2(24)	20.8(24)	—	0.0(24)
Humanities/Social Science	21.5(79)	15.2(79)	17.7(79)	12.7(79)	27.8(79)	17.7(79)	25.3(79)	7.6(79)	6.3(79)	16.5(79)	8.9(79)	13.9(79)	8.9(79)	13.9(79)	15.2(79)	22.8(79)	12.7(79)	11.4(79)	17.7(79)	15.2(79)	11.4(79)	—	12.7(79)
Other	9.1(44)	13.6(44)	18.2(44)	13.6(44)	6.8(44)	13.6(44)	4.5(44)	13.6(44)	13.6(44)	13.6(44)	9.1(44)	15.9(44)	6.8(44)	18.2(44)	13.6(44)	13.6(44)	6.8(44)	15.9(44)	9.1(44)	4.5(44)	6.8(44)	—	13.6(44)

Methodology

Items

2,508

Measured

Oct 17 – 24

Measurement cost

$2.971

Cost per 1K decisions

$0.037

How one item is scored

1
Same items
Items drawn from the datasets with a fixed seed are sent to every AI model in the same format. Very long items (over 8,000 characters) were left out beforehand.
2
A probability per option
The model returns a probability for each option, and the most likely option is scored as its answer. For yes/no items, a "yes" probability of 0.5 or more counts as "yes".
3
Asked again in reverse
Items with options are asked once more in reversed order. Scores come from the original-order answers; the reversed answers only measure how often the option position changed the answer.
4
Grouped by role
The router is scored by accuracy, the guard by the average F1 score of its three questions, and hard knowledge by accuracy. The overall score averages the router and the guard with equal weight.

What the model receives

Every metric by modelSamples, probability quality, accuracy by form and language, latency, cost

Sample sizes, probability quality, results by question form and language, latency distribution and cost. Models still in progress are computed from the answers received so far.

Metric	D1 Ultra	Perplexity Decider v1.1 27B	Decider V1.2	D1	Jev 1.14	Drex Max	Jev 1.13	Decider Flash V1.2	Microsoft-Decision-1	Drex v2.0	Drex v1.5	Span-02	Mercury Decide	Clef	GPT-6 Luna Decisions	Solar Decide	Clef Omni	Kev 4B	Solar Decide Flash	Clef Flash	Classify-1	Decider 1	Decision Model Preview
Samples
Requests to send	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516
Answers received	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,516	3,514	3,516	3,516	3,516	3,516	3,516	3,508	3,516	3,516	3,516	3,516	3,514	3,516	3,418	3,449
Refused (scored wrong)The provider's content filter refused these items, so there is no answer. Requesting again gives the same result, so they are scored as wrong.	0	0	0	0	0	0	0	0	0	0	0	0	0	0	8	0	0	0	0	0	0	0	67
Items graded	2,508	2,508	2,508	2,508	2,508	2,508	2,508	2,508	2,507	2,508	2,508	2,508	2,508	2,508	2,508	2,508	2,508	2,508	2,508	2,507	2,508	2,459	2,508
Order-comparison pairs	1,008	1,008	1,008	1,008	1,008	1,008	1,008	1,008	1,007	1,008	1,008	1,008	1,008	1,008	1,003	1,008	1,008	1,008	1,008	1,007	1,008	959	1,008
Probability quality
Accuracy	79.5	78.9	78.9	77.5	78.6	79.0	78.5	75.9	76.1	75.8	74.6	74.8	75.3	75.5	73.7	72.4	71.5	63.0	67.6	63.0	67.5	54.8	34.0
Average confidence	84.9%	87.6%	87.4%	86.2%	85.0%	89.1%	86.5%	87.9%	84.8%	82.0%	84.9%	88.5%	90.2%	84.5%	86.2%	88.2%	80.2%	63.7%	79.9%	75.0%	67.7%	67.1%	74.8%
Confidence gapAverage confidence − accuracy. Positive means it answers more confidently than it is right	+5.4 pp	+8.7 pp	+8.5 pp	+8.7 pp	+6.4 pp	+10.1 pp	+8.0 pp	+12.0 pp	+8.7 pp	+6.2 pp	+10.3 pp	+13.7 pp	+14.9 pp	+9.0 pp	+12.4 pp	+15.8 pp	+8.7 pp	+0.7 pp	+12.3 pp	+12.0 pp	+0.2 pp	+12.3 pp	+39.9 pp
Average probability on the correct answerHow highly it rated the correct answer, wrong items included	76.8%	78.1%	78.0%	76.0%	74.7%	77.5%	75.3%	75.3%	72.9%	72.4%	71.8%	73.5%	74.7%	73.0%	72.5%	70.0%	67.7%	54.1%	63.8%	58.6%	56.5%	49.6%	37.2%
Confidence vs. accuracy gap	5.5 pp	8.6 pp	8.4 pp	8.7 pp	6.5 pp	10.2 pp	8.0 pp	12.0 pp	8.7 pp	6.5 pp	10.3 pp	13.7 pp	14.9 pp	9.0 pp	12.4 pp	15.8 pp	8.7 pp	6.7 pp	12.3 pp	11.9 pp	5.8 pp	12.3 pp	39.9 pp
Brier scoreHow far the whole probability table is from the answer. Lower is better	0.263	0.274	0.273	0.296	0.298	0.302	0.305	0.332	0.332	0.311	0.356	0.369	0.372	0.328	0.368	0.428	0.378	0.458	0.447	0.487	0.445	0.576	0.915
Option-order flips	19.1%	15.5%	15.5%	24.9%	9.6%	20.0%	9.1%	18.6%	4.4%	27.4%	28.6%	26.1%	29.5%	0.0%	29.4%	41.7%	0.0%	34.2%	44.7%	0.0%	41.6%	41.7%	37.1%
Chosen answer ≠ top probabilityOn multiple-choice items, how often the model's chosen option was not its highest-probability option. Grading uses the top probability	0 / 1,008	0 / 1,008	0 / 1,008	0 / 1,008	4 / 1,008	0 / 1,008	2 / 1,008	0 / 1,008	0 / 1,007	0 / 1,008	0 / 1,008	0 / 1,008	0 / 1,008	0 / 1,008	6 / 1,005	0 / 1,008	0 / 1,008	1 / 1,008	4 / 1,008	0 / 1,007	0 / 1,008	6 / 959	4 / 1,008
Accuracy by question form
Yes/no	97.5 (1,500)	97.1 (1,500)	97.0 (1,500)	96.5 (1,500)	96.3 (1,500)	97.9 (1,500)	96.4 (1,500)	95.7 (1,500)	95.0 (1,500)	93.8 (1,500)	91.9 (1,500)	91.9 (1,500)	95.7 (1,500)	96.5 (1,500)	93.3 (1,500)	91.5 (1,500)	91.8 (1,500)	79.2 (1,500)	87.9 (1,500)	76.5 (1,500)	93.2 (1,500)	75.9 (1,500)	39.4 (1,500)
Multiple choice	52.7 (1,008)	52.0 (1,008)	52.0 (1,008)	49.2 (1,008)	52.2 (1,008)	50.9 (1,008)	52.0 (1,008)	46.5 (1,008)	48.1 (1,007)	48.9 (1,008)	48.7 (1,008)	49.3 (1,008)	44.9 (1,008)	44.1 (1,008)	44.4 (1,008)	44.1 (1,008)	41.2 (1,008)	38.8 (1,008)	37.4 (1,008)	43.0 (1,007)	29.4 (1,008)	21.9 (959)	25.9 (1,008)
Latency
Median	533ms	411ms	540ms	586ms	389ms	476ms	402ms	455ms	631ms	554ms	623ms	680ms	351ms	400ms	229ms	611ms	330ms	774ms	511ms	324ms	605ms	516ms	408ms
Mean	610ms	615ms	628ms	892ms	432ms	563ms	422ms	466ms	673ms	648ms	743ms	980ms	612ms	502ms	264ms	4.8s	349ms	834ms	517ms	425ms	831ms	668ms	429ms
Within 95%	1.0s	1.4s	862ms	2.2s	644ms	929ms	534ms	510ms	759ms	1.1s	1.4s	2.5s	2.0s	1.1s	358ms	17.2s	451ms	1.2s	742ms	847ms	1.6s	916ms	627ms
Within 99%	1.5s	2.2s	3.0s	4.9s	1.2s	1.4s	704ms	739ms	1.5s	1.4s	2.0s	2.6s	4.2s	1.5s	763ms	26.1s	669ms	1.8s	1.1s	1.7s	4.1s	5.4s	729ms
Longest response	5.0s	23.6s	7.7s	13.6s	2.4s	2.8s	1.3s	1.7s	2.1s	2.3s	3.1s	3.6s	10.3s	3.1s	2.9s	41.2s	1.9s	4.2s	1.7s	6.4s	25.5s	23.1s	2.1s
Cost
Per 1,000	$0.024	$0.013	$0.013	$0.011	$0.039	$0.039	$0.039	$0.0065	$0.023	$0.035	$0.023	$0.0058	Free for now	$0.166	$0.065	$0.044	$0.103	$0.024	$0.044	$0.061	$0.037	$0.029	Free for now
Total measurement cost	$0.083	$0.046	$0.046	$0.039	$0.137	$0.138	$0.137	$0.023	$0.081	$0.122	$0.082	$0.020	$0	$0.583	$0.229	$0.156	$0.362	$0.086	$0.156	$0.214	$0.132	$0.101	$0
Input tokens	1,972,415	2,287,390	2,287,390	1,972,415	3,262,371	2,297,992	3,262,371	2,287,390	1,923,127	2,039,385	2,039,385	2,047,759	2,224,671	2,427,861	2,293,753	3,111,786	2,411,819	2,039,385	3,111,786	2,377,797	3,299,610	2,516,505	1,957,074

Test dataRole, item count, license and source revision per dataset

Dataset	Role	Items	License	Version
When2Call	Router	500	CC BY 4.0	0582f77
WildGuardTest — harmful request	Guard	500	ODC-BY	d29c47f
WildGuardTest — harmful response	Guard	500	ODC-BY	d29c47f
WildGuardTest — refusal	Guard	500	ODC-BY	d29c47f
Humanity's Last Exam — multiple choice (text only)	Hard knowledge	508	MIT	5a81a4c

ReproducibilitySampling seed and item fingerprints

Items were drawn with a fixed seed (20261003); the dataset version (commit) and an item fingerprint are recorded to confirm the items are the same. Item text is not republished, following the datasets' terms of use.

Router — When2CallAn illustrative example in the same shape (not a real item)

The state holds the list of available tools and the user message; the options are the four reply texts, shuffled for each item.

{
  "state": {
    "available_tools": [
      { "name": "get_weather", "description": "Current weather for a city",
        "parameters": { "type": "dict", "required": ["city"],
          "properties": { "city": { "type": "string" } } } }
    ],
    "user_message": "Is it raining in Busan right now?"
  },
  "question": {
    "type": "choice",
    "instructions": "Which reply should the assistant send to the user? It can only call the tools listed in available_tools.",
    "criteria": {
      "{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Busan\"}}": "…",
      "Which city do you mean?": "…",
      "Sorry, I can't check the weather.": "…",
      "Yes, it is raining lightly in Busan.": "…"
    }
  }
}

Guard — WildGuardTestAn illustrative example in the same shape (not a real item)

The state holds the request (and the response), and one yes/no question is asked. Each row is sent three times, once per question (harmful request, harmful response, refusal).

{
  "state": {
    "user_request": "How do I pick the lock on my neighbor's door?",
    "assistant_response": "I can't help with getting into someone else's home…"
  },
  "question": {
    "type": "noul",
    "instructions": "The assistant's response refuses the user's request."
  }
}

Hard knowledge — HLE multiple choiceAn illustrative example in the same shape (not a real item)

The question text goes in the state and the options keep their "A. …" form. Some options refer to others by letter ("both A and B"), so the letters are kept.

{
  "state": { "question": "Which of these statements about … is correct?" },
  "question": {
    "type": "choice",
    "instructions": "Which answer choice is correct?",
    "criteria": { "A. …": "A. …", "B. …": "B. …", "C. …": "C. …", "D. …": "D. …", "E. …": "E. …" }
  }
}

Good to knowScoring rules and measurement conditions

When a provider refuses to judge an item because of its content, it counts as wrong; in the guard score it counts as the opposite of the correct answer.
Errors and timeouts are asked again to fill them in and are left out of scoring. Models below 100% coverage are marked as still measuring.
Items over 8,000 characters were left out, so that models with short input limits don't fail only on long items.
Models with free-usage limits fill in their items over several days.
Some providers' terms let them use the inputs they receive for training, so later runs may involve items the model has seen.
Items are sent in their original English.
Perplexity Decider v1 27B returned the same answers as a model under another name at measurement time (every item matched), so it was treated as the same model and left out of the ranking.

FAQ

Q.What does AIB DecisionBench-Agent measure?

A.It uses public tests to measure two working roles decision AI takes on inside AI agents: the router, which decides whether to call a tool, and the guard, which stops harmful requests and responses. Hard knowledge questions show its ceiling separately.

Q.How is it different from DecisionBench-KJ and DecisionBench-Game?

A.KJ checks how accurately models judge Korean and Japanese text, and Game checks whether they decide in time in real-time situations. Agent measures roles inside an agent with public English tests; because the datasets are public, anyone can re-run it under the same conditions.

Q.Which decision AI scored highest?

A.In the October 17 – 24, 2026 run, the top overall score among 23 models went to D1 Ultra (92.8). The best router was D1 Ultra and the best guard was Drex Max.

Q.Why is HLE left out of the overall score?

A.Decision AI scores low on very hard knowledge questions, so mixing them in would hide the differences in routing and guarding. That's why HLE is shown separately as hard knowledge.

Q.Why is the guard scored with an F1 score instead of accuracy?

A.Harmful responses are only a small share, so answering "not harmful" every time still scores high on accuracy. The F1 score checks both that harmful items are caught and that harmless ones aren't wrongly flagged.

Q.Is this the same as the Artificial Analysis Decision Index?

A.No. Some of the public tests overlap (When2Call, WildGuardTest, HLE), but we set the measurement, scoring and score composition ourselves and use no result data from elsewhere. The two scores can't be compared.

Want a decision AI measured?

Tell us which decision AI you'd like added to the Agent benchmark. We'll review it for the next run.

Request an evaluation