Benchmarks/AIB DecisionBench-KJ 0.2 · CC BY 4.0

Which decision model can you trust most in Korean and Japanese?

AIB DecisionBench-KJ is a decision-model benchmark for comparing AI performance in Korean and Japanese. It measures and analyzes how accurately decision models (Decision AI) judge Korean and Japanese problems, and how far you can trust the probabilities they give for their judgments.

35
Models measured
8
Datasets
2,000
Items
KO・JA
Languages

Since TypeSafe released Jev on September 14, 2026, the number of decision models AIB has counted has grown to 42 in under 10 weeks. Instead of generating long text like a typical LLM, a decision model is built to return structured results fast: a yes/no judgment, a multiple-choice pick, or the probability of each option. Its speed and low cost are drawing attention in industries that need the same decision made over and over.

Typical uses include sorting customer inquiries, detecting harmful content, classifying documents and data, and routing branches in automated workflows. The more a job repeats the same judgment across large volumes of input, the more speed, cost and the reliability of the judgment probability matter alongside accuracy.

AIB DecisionBench-KJ compares leading decision models under identical conditions on 2,000 items from 8 public datasets. Centered on Korean and Japanese decision-model performance, it analyzes accuracy and stability per dataset along with practical fit, so you can see which model suits which job.

The top decision model, Decider V1.3, scores 82.4, 1.5 points below GPT-6.1 Sol (83.9), but handles 1,000 decisions 4× faster (8m 3s vs 34m 25s) at 1/192 of the cost. Gemini Decision 1 has the most accurate probabilities (confidence off from accuracy by 1.4 pts on average), and Decider Flash V1.3 is the cheapest ($0.0021 per 1,000).

Overall ranking

  • Ref.OpenAIGPT-6.1 Sol83.9±1.5
  • Ref.AnthropicClaude Sonnet 5.580.9±1.6
  1. 🥇PerplexityDecider V1.382.4±1.6
  2. 🥈PerplexityDecider V1.282.3±1.5
  3. 🥉Nace AIDrex Max82.2±1.6
  4. #4PerplexityPerplexity Decider v1.1 27B81.7±1.6
  5. #5Liquid AId1.2-ultra81.6±1.5
  6. #6Liquid AID1 Ultra81.5±1.5
  7. #7Liquid AId1.280.6±1.6
  8. #8Liquid AId1.179.9±1.6
  9. #9GoogleGemini Decision 179.8±1.7
  10. #10BasetenMammoth 178.9±1.7
  11. #11TypeSafeJev 1.1478.7±1.7
  12. #12TypeSafeJev 1.1378.6±1.8
  13. #13InceptionMercury Decide77.7±1.7
  14. #14MicrosoftMicrosoft-Decision-177.5±1.8
  15. #15PerplexityPerplexity Decider v1 27B77.2±1.7
  16. #16OpenAIGPT-6 Luna Decisions77.2±1.8
  17. #17RespanSpan-0276.9±1.8
  18. #18UpstageSolar Decide75.3±1.8
  19. #19Liquid AID175.1±1.8
  20. #20BasetenElephant 174.8±1.8
  21. #21PerplexityDecider Flash V1.274.2±1.9
  22. #22CloudflareClef73.9±1.8
  23. #23KakaoKanana 2 V Decide73.9±1.8
  24. #24PerplexityDecider Flash V1.373.8±1.8
  25. #25Nace AIDrex v2.072.7±1.9
  26. #26CloudflareClef Omni71.5±1.9
  27. #27ScaleDownClassify-170.5±1.9
  28. #28Nace AIDrex v1.569.6±2.0
  29. #29CloudflareClef Flash69.4±1.9
  30. #30Jared PalmerKev 4B68.0±2.0
  31. #31UpstageSolar Decide Flash67.0±2.0
  32. #32AlibabaDecision Model Preview62.9±2.0
  33. #33BasetenJoey 159.7±1.9
  34. #34meraGPTDecider 1.159.2±2.1
  35. #35meraGPTDecider 154.7±2.1
RankModelAccuracy
Ref.OpenAIGPT-6.1 Sol83.9±1.5
Ref.AnthropicClaude Sonnet 5.580.9±1.6
🥇PerplexityDecider V1.382.4±1.6
🥈PerplexityDecider V1.282.3±1.5
🥉Nace AIDrex Max82.2±1.6
#4PerplexityPerplexity Decider v1.1 27B81.7±1.6
#5Liquid AId1.2-ultra81.6±1.5
#6Liquid AID1 Ultra81.5±1.5
#7Liquid AId1.280.6±1.6
#8Liquid AId1.179.9±1.6
#9GoogleGemini Decision 179.8±1.7
#10BasetenMammoth 178.9±1.7
#11TypeSafeJev 1.1478.7±1.7
#12TypeSafeJev 1.1378.6±1.8
#13InceptionMercury Decide77.7±1.7
#14MicrosoftMicrosoft-Decision-177.5±1.8
#15PerplexityPerplexity Decider v1 27B77.2±1.7
#16OpenAIGPT-6 Luna Decisions77.2±1.8
#17RespanSpan-0276.9±1.8
#18UpstageSolar Decide75.3±1.8
#19Liquid AID175.1±1.8
#20BasetenElephant 174.8±1.8
#21PerplexityDecider Flash V1.274.2±1.9
#22CloudflareClef73.9±1.8
#23KakaoKanana 2 V Decide73.9±1.8
#24PerplexityDecider Flash V1.373.8±1.8
#25Nace AIDrex v2.072.7±1.9
#26CloudflareClef Omni71.5±1.9
#27ScaleDownClassify-170.5±1.9
#28Nace AIDrex v1.569.6±2.0
#29CloudflareClef Flash69.4±1.9
#30Jared PalmerKev 4B68.0±2.0
#31UpstageSolar Decide Flash67.0±2.0
#32AlibabaDecision Model Preview62.9±2.0
#33BasetenJoey 159.7±1.9
#34meraGPTDecider 1.159.2±2.1
#35meraGPTDecider 154.7±2.1
Accuracy × cost
53616977$0.0021$0.012$0.069Cost · 1,000Accuracy
Decider V1.3
Decider V1.2
Drex Max
d1.2-ultra
D1 Ultra
d1.2
d1.1
Gemini Decision 1
Mammoth 1
Jev 1.14
Jev 1.13
GPT-6 Luna Decisions
Span-02
Solar Decide
D1
Elephant 1
Decider Flash V1.2
Clef
Decider Flash V1.3
Drex v2.0
Clef Omni
Classify-1
Drex v1.5
Clef Flash
Kev 4B
Solar Decide Flash
Joey 1
Decider 1.1
Decider 1
Accuracy × latency
53616977211ms451ms691msLatency · medianAccuracy
Decider V1.3
Decider V1.2
Drex Max
Perplexity Decider v1.1 27B
d1.2-ultra
d1.2
d1.1
Gemini Decision 1
Jev 1.14
Jev 1.13
Microsoft-Decision-1
Span-02
Solar Decide
D1
Elephant 1
Clef
Kanana 2 V Decide
Drex v2.0
Clef Omni
Classify-1
Drex v1.5
Clef Flash
Kev 4B
Solar Decide Flash
Decision Model Preview
Joey 1
Decider 1.1
Decider 1
Accuracy × cost
53616977$0.0021$0.012$0.069Cost · 1,000Accuracy
Decider V1.3
Drex Max
d1.2-ultra
D1 Ultra
d1.2
d1.1
Mammoth 1
Jev 1.14
Jev 1.13
Span-02
Solar Decide
D1
Elephant 1
Clef
Decider Flash V1.3
Drex v2.0
Clef Omni
Drex v1.5
Clef Flash
Kev 4B
Solar Decide Flash
Joey 1
Decider 1.1
Decider 1
Accuracy × latency
53616977211ms451ms691msLatency · medianAccuracy
Decider V1.3
Decider V1.2
Drex Max
d1.2-ultra
d1.2
d1.1
Gemini Decision 1
Jev 1.14
Microsoft-Decision-1
Span-02
D1
Clef
Drex v2.0
Clef Omni
Classify-1
Drex v1.5
Clef Flash
Kev 4B
Solar Decide Flash
Decision Model Preview
Joey 1
Decider 1.1
Decider 1

Use cases

Checking AI answersPick the better of two AI answers
Frontier-level+0.8%p
PerplexityDecider V1.289.2%
  • Drex Max89.2%
  • Mammoth 189.2%
Datasets
  • M-RewardBench (한국어)
  • M-RewardBench (日本語)
Checking claims against a sourceDecide whether a claim follows from or contradicts a source
Frontier-level+3.2%p
PerplexityDecider V1.394.4%
  • Decider V1.294.0%
  • Drex Max94.0%
Datasets
  • KLUE NLI
  • JGLUE JNLI
Blocking harmful requestsDecide whether a request is dangerous or illegal
Frontier-level+7.2%p
BasetenElephant 191.2%
  • d1.290.8%
  • Mammoth 190.4%
Datasets
  • PolyGuardPrompts (한국어)
  • PolyGuardPrompts (日本語)
Catching hate speechDetect demeaning content about origin, gender or religion
Frontier-level−2.0%p
Liquid AId1.2-ultra94.4%
  • d1.194.0%
  • Kanana 2 V Decide93.2%
Datasets
  • K-MHaS
Grading toxicityRate toxicity on a 5-level scale
Frontier-level−2.0%p
OpenAIGPT-6 Luna Decisions56.8%
  • Jev 1.1456.4%
  • Microsoft-Decision-156.0%
Datasets
  • RTP-LX (한국어)
  • RTP-LX (日本語)
Checking expert knowledgeKnowledge questions at Korean licensing-exam and Japanese school level
Gap−8.1%p
GoogleGemini Decision 184.7%
  • D1 Ultra83.2%
  • d1.281.2%
Datasets
  • KMMLU
  • JMMLU
Finding similar sentencesRate meaning similarity on a 6-level scale
Frontier-level+8.6%p
Nace AIDrex Max60.6%
  • Decider V1.258.3%
  • Perplexity Decider v1.1 27B56.6%
Datasets
  • KLUE STS
  • JGLUE JSTS
Commonsense and cause-effectPick the plausible cause, effect or commonsense answer
Frontier-level−0.6%p
PerplexityDecider V1.398.8%
  • Drex Max98.8%
  • d1.2-ultra98.8%
Datasets
  • KoBEST COPA
  • JGLUE JCommonsenseQA
Yes/no answers from a documentAnswer yes/no from a passage alone
Frontier-level+1.0%p
PerplexityDecider V1.3100.0%
  • Drex Max100.0%
  • d1.2100.0%
Datasets
  • KoBEST BoolQ
Classifying sentimentTell positive from negative, even with negation
Frontier-level±0.0%p
PerplexityDecider V1.398.3%
  • Decider V1.298.3%
  • Drex Max98.3%
Datasets
  • KoBEST SentiNeg

Reliability

Decider V1.3off by 2.1 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%30–40% · n=30 · 33%40–50% · n=144 · 38%50–60% · n=179 · 52%60–70% · n=154 · 63%70–80% · n=114 · 72%80–90% · n=200 · 84%90–100% · n=1179 · 97%ConfidenceActual accuracy
Decider V1.2off by 2.3 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=3 · 67%30–40% · n=72 · 32%40–50% · n=148 · 51%50–60% · n=165 · 55%60–70% · n=117 · 62%70–80% · n=132 · 71%80–90% · n=181 · 85%90–100% · n=1182 · 96%ConfidenceActual accuracy
Drex Maxoff by 2.4 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=2 · 0%30–40% · n=71 · 41%40–50% · n=148 · 43%50–60% · n=163 · 57%60–70% · n=111 · 61%70–80% · n=134 · 69%80–90% · n=174 · 82%90–100% · n=1197 · 96%ConfidenceActual accuracy
Perplexity Decider v1.1 27Boff by 3.4 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=2 · 50%30–40% · n=46 · 41%40–50% · n=133 · 47%50–60% · n=172 · 49%60–70% · n=151 · 55%70–80% · n=106 · 72%80–90% · n=171 · 83%90–100% · n=1219 · 96%ConfidenceActual accuracy
d1.2-ultraoff by 3.2 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=7 · 86%30–40% · n=70 · 31%40–50% · n=185 · 34%50–60% · n=168 · 54%60–70% · n=147 · 69%70–80% · n=160 · 78%80–90% · n=241 · 93%90–100% · n=1022 · 98%ConfidenceActual accuracy
D1 Ultraoff by 2.1 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=5 · 40%30–40% · n=72 · 38%40–50% · n=209 · 41%50–60% · n=157 · 46%60–70% · n=98 · 66%70–80% · n=141 · 78%80–90% · n=230 · 89%90–100% · n=1088 · 98%ConfidenceActual accuracy
d1.2off by 3.0 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=6 · 50%30–40% · n=117 · 38%40–50% · n=159 · 43%50–60% · n=191 · 51%60–70% · n=162 · 69%70–80% · n=177 · 87%80–90% · n=303 · 88%90–100% · n=885 · 98%ConfidenceActual accuracy
d1.1off by 1.9 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=7 · 43%30–40% · n=92 · 34%40–50% · n=148 · 34%50–60% · n=169 · 54%60–70% · n=161 · 61%70–80% · n=172 · 79%80–90% · n=261 · 87%90–100% · n=990 · 97%ConfidenceActual accuracy
Gemini Decision 1off by 1.4 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=9 · 22%30–40% · n=114 · 36%40–50% · n=177 · 49%50–60% · n=144 · 58%60–70% · n=126 · 63%70–80% · n=126 · 78%80–90% · n=219 · 87%90–100% · n=1042 · 97%ConfidenceActual accuracy
Mammoth 1off by 2.3 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=15 · 33%30–40% · n=92 · 34%40–50% · n=150 · 43%50–60% · n=176 · 55%60–70% · n=136 · 54%70–80% · n=139 · 63%80–90% · n=200 · 82%90–100% · n=1092 · 97%ConfidenceActual accuracy
Jev 1.14off by 3.4 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=4 · 0%30–40% · n=40 · 43%40–50% · n=75 · 39%50–60% · n=251 · 47%60–70% · n=178 · 65%70–80% · n=190 · 65%80–90% · n=264 · 85%90–100% · n=998 · 95%ConfidenceActual accuracy
Jev 1.13off by 3.7 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=3 · 0%30–40% · n=36 · 36%40–50% · n=89 · 46%50–60% · n=217 · 52%60–70% · n=183 · 58%70–80% · n=190 · 64%80–90% · n=256 · 80%90–100% · n=1026 · 95%ConfidenceActual accuracy
Mercury Decideoff by 10.7 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=1 · 0%30–40% · n=15 · 27%40–50% · n=62 · 39%50–60% · n=139 · 44%60–70% · n=121 · 45%70–80% · n=150 · 44%80–90% · n=165 · 67%90–100% · n=1347 · 92%ConfidenceActual accuracy
Microsoft-Decision-1off by 8.6 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=1 · 0%30–40% · n=19 · 5%40–50% · n=68 · 34%50–60% · n=190 · 60%60–70% · n=173 · 46%70–80% · n=198 · 64%80–90% · n=208 · 69%90–100% · n=1143 · 93%ConfidenceActual accuracy
Perplexity Decider v1 27Boff by 2.3 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=22 · 50%30–40% · n=110 · 39%40–50% · n=151 · 40%50–60% · n=213 · 45%60–70% · n=181 · 65%70–80% · n=186 · 75%80–90% · n=253 · 87%90–100% · n=884 · 97%ConfidenceActual accuracy
GPT-6 Luna Decisionsoff by 8.7 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=3 · 0%30–40% · n=19 · 37%40–50% · n=67 · 42%50–60% · n=169 · 47%60–70% · n=145 · 59%70–80% · n=171 · 57%80–90% · n=227 · 67%90–100% · n=1197 · 92%ConfidenceActual accuracy
Span-02off by 9.3 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=2 · 0%30–40% · n=25 · 24%40–50% · n=80 · 41%50–60% · n=141 · 46%60–70% · n=157 · 41%70–80% · n=162 · 58%80–90% · n=234 · 69%90–100% · n=1199 · 93%ConfidenceActual accuracy
Solar Decideoff by 18.1 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=2 · 0%30–40% · n=5 · 20%40–50% · n=16 · 38%50–60% · n=32 · 38%60–70% · n=29 · 41%70–80% · n=34 · 53%80–90% · n=303 · 61%90–100% · n=1579 · 81%ConfidenceActual accuracy
D1off by 5.9 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=21 · 33%30–40% · n=117 · 30%40–50% · n=136 · 33%50–60% · n=201 · 52%60–70% · n=124 · 56%70–80% · n=137 · 66%80–90% · n=208 · 75%90–100% · n=1056 · 94%ConfidenceActual accuracy
Elephant 1off by 5.3 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=41 · 41%30–40% · n=104 · 37%40–50% · n=126 · 37%50–60% · n=190 · 46%60–70% · n=141 · 45%70–80% · n=170 · 65%80–90% · n=229 · 80%90–100% · n=999 · 95%ConfidenceActual accuracy
Decider Flash V1.2off by 4.7 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=31 · 29%30–40% · n=199 · 44%40–50% · n=124 · 44%50–60% · n=170 · 51%60–70% · n=162 · 59%70–80% · n=180 · 64%80–90% · n=235 · 80%90–100% · n=899 · 94%ConfidenceActual accuracy
Clefoff by 8.0 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=5 · 40%30–40% · n=58 · 28%40–50% · n=158 · 32%50–60% · n=178 · 34%60–70% · n=161 · 54%70–80% · n=140 · 58%80–90% · n=227 · 78%90–100% · n=1073 · 93%ConfidenceActual accuracy
Kanana 2 V Decideoff by 3.8 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=37 · 16%30–40% · n=127 · 29%40–50% · n=168 · 39%50–60% · n=203 · 44%60–70% · n=154 · 65%70–80% · n=164 · 68%80–90% · n=273 · 86%90–100% · n=874 · 95%ConfidenceActual accuracy
Decider Flash V1.3off by 8.4 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=2 · 50%30–40% · n=59 · 32%40–50% · n=152 · 36%50–60% · n=215 · 46%60–70% · n=157 · 48%70–80% · n=119 · 54%80–90% · n=197 · 72%90–100% · n=1099 · 93%ConfidenceActual accuracy
Drex v2.0off by 4.5 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%10–20% · n=1 · 0%20–30% · n=76 · 29%30–40% · n=147 · 44%40–50% · n=131 · 40%50–60% · n=173 · 54%60–70% · n=199 · 66%70–80% · n=169 · 67%80–90% · n=296 · 78%90–100% · n=808 · 92%ConfidenceActual accuracy
Clef Omnioff by 9.0 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=12 · 17%30–40% · n=66 · 35%40–50% · n=165 · 25%50–60% · n=189 · 36%60–70% · n=140 · 56%70–80% · n=141 · 63%80–90% · n=267 · 77%90–100% · n=1020 · 90%ConfidenceActual accuracy
Classify-1off by 3.7 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=45 · 29%30–40% · n=177 · 30%40–50% · n=150 · 47%50–60% · n=353 · 53%60–70% · n=312 · 70%70–80% · n=258 · 81%80–90% · n=336 · 91%90–100% · n=369 · 96%ConfidenceActual accuracy
Drex v1.5off by 13.1 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=12 · 42%30–40% · n=100 · 32%40–50% · n=138 · 41%50–60% · n=150 · 45%60–70% · n=135 · 44%70–80% · n=135 · 53%80–90% · n=196 · 59%90–100% · n=1134 · 87%ConfidenceActual accuracy
Clef Flashoff by 9.9 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=3 · 33%30–40% · n=66 · 35%40–50% · n=125 · 39%50–60% · n=234 · 44%60–70% · n=191 · 49%70–80% · n=210 · 64%80–90% · n=297 · 70%90–100% · n=874 · 89%ConfidenceActual accuracy
Kev 4Boff by 3.7 pts on average · well calibrated
OverconfidentCautious0%0%50%50%100%100%20–30% · n=40 · 43%30–40% · n=207 · 30%40–50% · n=163 · 45%50–60% · n=307 · 51%60–70% · n=257 · 64%70–80% · n=218 · 69%80–90% · n=193 · 82%90–100% · n=615 · 94%ConfidenceActual accuracy
Solar Decide Flashoff by 12.9 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=4 · 25%30–40% · n=59 · 24%40–50% · n=111 · 33%50–60% · n=212 · 34%60–70% · n=218 · 57%70–80% · n=225 · 55%80–90% · n=274 · 62%90–100% · n=897 · 89%ConfidenceActual accuracy
Decision Model Previewoff by 11.0 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=32 · 19%30–40% · n=152 · 30%40–50% · n=180 · 38%50–60% · n=241 · 46%60–70% · n=201 · 52%70–80% · n=194 · 59%80–90% · n=248 · 67%90–100% · n=740 · 87%ConfidenceActual accuracy
Joey 1off by 10.2 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=122 · 36%30–40% · n=216 · 29%40–50% · n=135 · 33%50–60% · n=232 · 46%60–70% · n=229 · 57%70–80% · n=231 · 59%80–90% · n=366 · 69%90–100% · n=469 · 89%ConfidenceActual accuracy
Decider 1.1off by 5.7 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=50 · 30%30–40% · n=260 · 27%40–50% · n=196 · 36%50–60% · n=337 · 55%60–70% · n=315 · 64%70–80% · n=292 · 68%80–90% · n=299 · 72%90–100% · n=251 · 90%ConfidenceActual accuracy
Decider 1off by 9.7 pts on average · overconfident
OverconfidentCautious0%0%50%50%100%100%20–30% · n=140 · 25%30–40% · n=187 · 39%40–50% · n=193 · 41%50–60% · n=319 · 45%60–70% · n=316 · 47%70–80% · n=353 · 60%80–90% · n=278 · 75%90–100% · n=214 · 90%ConfidenceActual accuracy

Each dot = a confidence range · size = number of answers in it

Datasets

KoBEST dataset evaluation details

KoBEST BoolQ

Korean
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy100.0(100)99.0(100)100.0(100)99.0(100)99.0(100)99.0(100)100.0(100)100.0(100)99.0(100)100.0(100)99.0(100)100.0(100)100.0(100)100.0(100)99.0(100)98.0(100)98.0(100)94.0(100)96.0(100)100.0(100)98.0(100)99.0(100)93.0(100)99.0(100)97.0(100)96.0(100)87.0(100)93.0(100)94.0(100)94.0(100)86.0(100)91.0(100)89.0(100)75.0(100)89.0(100)
Confidence vs. accuracy gap2.4 pp2.7 pp1.2 pp2.2 pp3.8 pp3.1 pp4.3 pp3.3 pp3.8 pp2.5 pp8.7 pp8.5 pp2.0 pp4.0 pp4.3 pp2.0 pp2.8 pp3.1 pp3.2 pp4.4 pp5.2 pp3.3 pp5.6 pp1.3 pp9.1 pp2.7 pp12.2 pp5.5 pp4.4 pp3.1 pp7.6 pp7.9 pp5.8 pp10.9 pp4.4 pp
Median latency385ms546ms464ms347ms475ms507ms469ms475ms257ms461ms375ms222ms296ms622ms300ms268ms675ms318ms359ms444ms450ms427ms446ms279ms478ms355ms544ms479ms244ms700ms552ms403ms430ms588ms604ms
Gold answer
Yes100.0(50)100.0(50)100.0(50)100.0(50)100.0(50)98.0(50)100.0(50)100.0(50)98.0(50)100.0(50)100.0(50)100.0(50)100.0(50)100.0(50)100.0(50)96.0(50)100.0(50)100.0(50)100.0(50)100.0(50)100.0(50)100.0(50)94.0(50)100.0(50)100.0(50)100.0(50)96.0(50)100.0(50)100.0(50)100.0(50)100.0(50)96.0(50)96.0(50)90.0(50)94.0(50)
No100.0(50)98.0(50)100.0(50)98.0(50)98.0(50)100.0(50)100.0(50)100.0(50)100.0(50)100.0(50)98.0(50)100.0(50)100.0(50)100.0(50)98.0(50)100.0(50)96.0(50)88.0(50)92.0(50)100.0(50)96.0(50)98.0(50)92.0(50)98.0(50)94.0(50)92.0(50)78.0(50)86.0(50)88.0(50)88.0(50)72.0(50)86.0(50)82.0(50)60.0(50)84.0(50)

KoBEST COPA

Korean
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy98.9(90)97.8(90)98.9(90)97.8(90)98.9(90)97.8(90)95.6(90)97.8(90)98.9(90)97.8(90)97.8(90)97.8(90)96.7(90)96.7(90)97.8(90)96.7(90)97.8(90)95.6(90)96.7(90)96.7(90)95.6(90)96.7(90)94.4(90)96.7(90)96.7(90)95.6(90)92.2(90)96.7(90)96.7(90)92.2(90)95.6(90)88.9(90)71.1(90)64.4(90)72.2(90)
Option-order flips0.0% (90)1.1% (90)0.0% (90)1.1% (90)0.0% (90)1.1% (90)3.3% (90)1.1% (90)0.0% (90)0.0% (90)0.0% (90)0.0% (90)1.1% (90)0.0% (90)1.1% (90)2.2% (90)0.0% (90)5.6% (90)2.2% (90)0.0% (90)2.2% (90)0.0% (90)7.8% (90)2.2% (90)1.1% (90)0.0% (90)8.9% (90)2.2% (90)0.0% (90)4.4% (90)5.6% (90)11.1% (90)7.8% (90)13.3% (90)16.7% (90)
Confidence vs. accuracy gap1.5 pp1.2 pp1.5 pp1.7 pp2.6 pp1.0 pp3.1 pp1.0 pp2.3 pp2.5 pp1.8 pp2.7 pp2.5 pp2.7 pp5.7 pp3.0 pp2.2 pp1.7 pp2.5 pp2.3 pp3.2 pp2.2 pp4.3 pp2.1 pp2.3 pp5.3 pp6.8 pp1.8 pp4.1 pp5.9 pp2.9 pp5.7 pp13.3 pp8.9 pp5.6 pp
Median latency383ms519ms474ms340ms469ms502ms468ms462ms394ms441ms378ms218ms273ms618ms302ms226ms531ms318ms364ms432ms449ms507ms395ms281ms575ms354ms558ms574ms255ms683ms554ms394ms425ms613ms605ms
Asks for
Cause100.0(44)97.7(44)100.0(44)100.0(44)100.0(44)100.0(44)97.7(44)100.0(44)100.0(44)100.0(44)100.0(44)100.0(44)100.0(44)97.7(44)100.0(44)97.7(44)100.0(44)95.5(44)97.7(44)100.0(44)97.7(44)97.7(44)95.5(44)97.7(44)100.0(44)97.7(44)88.6(44)97.7(44)100.0(44)97.7(44)95.5(44)93.2(44)77.3(44)68.2(44)77.3(44)
Effect97.8(46)97.8(46)97.8(46)95.7(46)97.8(46)95.7(46)93.5(46)95.7(46)97.8(46)95.7(46)95.7(46)95.7(46)93.5(46)95.7(46)95.7(46)95.7(46)95.7(46)95.7(46)95.7(46)93.5(46)93.5(46)95.7(46)93.5(46)95.7(46)93.5(46)93.5(46)95.7(46)95.7(46)93.5(46)87.0(46)95.7(46)84.8(46)65.2(46)60.9(46)67.4(46)
Position of the correct option
First option97.8(45)97.8(45)97.8(45)97.8(45)97.8(45)97.8(45)93.3(45)97.8(45)97.8(45)97.8(45)97.8(45)97.8(45)97.8(45)95.6(45)97.8(45)95.6(45)97.8(45)97.8(45)95.6(45)95.6(45)95.6(45)93.3(45)91.1(45)95.6(45)95.6(45)93.3(45)93.3(45)95.6(45)95.6(45)91.1(45)95.6(45)86.7(45)68.9(45)75.6(45)73.3(45)
Second option100.0(45)97.8(45)100.0(45)97.8(45)100.0(45)97.8(45)97.8(45)97.8(45)100.0(45)97.8(45)97.8(45)97.8(45)95.6(45)97.8(45)97.8(45)97.8(45)97.8(45)93.3(45)97.8(45)97.8(45)95.6(45)100.0(45)97.8(45)97.8(45)97.8(45)97.8(45)91.1(45)97.8(45)97.8(45)93.3(45)95.6(45)91.1(45)73.3(45)53.3(45)71.1(45)

KoBEST SentiNeg

Korean
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy98.3(60)98.3(60)98.3(60)98.3(60)98.3(60)98.3(60)96.7(60)98.3(60)98.3(60)98.3(60)96.7(60)96.7(60)96.7(60)98.3(60)98.3(60)98.3(60)98.3(60)98.3(60)98.3(60)96.7(60)95.0(60)95.0(60)95.0(60)93.3(60)91.7(60)98.3(60)96.7(60)96.7(60)98.3(60)95.0(60)98.3(60)90.0(60)90.0(60)90.0(60)91.7(60)
Confidence vs. accuracy gap2.5 pp2.3 pp2.0 pp1.4 pp2.3 pp0.9 pp1.7 pp1.4 pp1.8 pp2.9 pp6.1 pp6.4 pp2.2 pp2.6 pp3.4 pp5.0 pp1.1 pp1.0 pp1.2 pp1.7 pp4.4 pp0.8 pp10.4 pp5.1 pp10.6 pp0.5 pp9.9 pp2.0 pp2.3 pp6.9 pp1.7 pp5.9 pp5.7 pp7.4 pp10.6 pp
Median latency378ms578ms458ms344ms472ms510ms467ms454ms366ms445ms370ms217ms273ms618ms303ms220ms514ms367ms349ms425ms450ms611ms339ms282ms572ms351ms562ms473ms245ms689ms565ms392ms420ms614ms586ms
Gold answer
Positive96.7(30)96.7(30)96.7(30)96.7(30)96.7(30)96.7(30)96.7(30)96.7(30)96.7(30)96.7(30)96.7(30)96.7(30)93.3(30)96.7(30)96.7(30)96.7(30)96.7(30)96.7(30)96.7(30)93.3(30)96.7(30)90.0(30)90.0(30)93.3(30)83.3(30)96.7(30)96.7(30)93.3(30)96.7(30)93.3(30)96.7(30)90.0(30)86.7(30)83.3(30)83.3(30)
Negative100.0(30)100.0(30)100.0(30)100.0(30)100.0(30)100.0(30)96.7(30)100.0(30)100.0(30)100.0(30)96.7(30)96.7(30)100.0(30)100.0(30)100.0(30)100.0(30)100.0(30)100.0(30)100.0(30)100.0(30)93.3(30)100.0(30)100.0(30)93.3(30)100.0(30)100.0(30)96.7(30)100.0(30)100.0(30)96.7(30)100.0(30)90.0(30)93.3(30)96.7(30)100.0(30)
KLUE dataset evaluation details

KLUE NLI

Korean
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy98.7(150)98.7(150)99.3(150)98.7(150)98.7(150)97.3(150)96.0(150)95.3(150)96.0(150)98.0(150)92.0(150)92.7(150)96.7(150)89.3(150)96.0(150)82.0(150)88.0(150)80.0(150)92.0(150)91.3(150)94.0(150)81.3(150)93.3(150)93.3(150)92.0(150)78.7(150)88.7(150)94.0(150)79.3(150)86.7(150)66.0(150)87.3(150)82.7(150)64.0(150)57.3(150)
Option-order flips1.3% (150)0.0% (150)1.3% (150)0.0% (150)0.0% (150)3.3% (150)2.0% (150)4.0% (150)0.0% (150)0.0% (150)1.3% (150)0.7% (150)1.3% (150)0.0% (150)0.7% (150)8.7% (150)4.7% (150)9.3% (150)2.7% (150)1.3% (150)0.7% (150)0.0% (150)0.0% (150)0.7% (150)4.7% (150)0.0% (150)7.3% (150)1.3% (150)0.0% (150)6.0% (150)12.0% (150)1.4% (147)2.0% (150)4.7% (150)19.3% (150)
Confidence vs. accuracy gap3.7 pp2.3 pp3.2 pp3.9 pp10.3 pp10.9 pp11.5 pp11.9 pp5.4 pp3.6 pp3.6 pp3.2 pp3.6 pp4.2 pp5.4 pp2.4 pp5.4 pp15.1 pp2.8 pp2.4 pp4.9 pp7.9 pp7.4 pp3.6 pp7.9 pp9.3 pp8.2 pp5.6 pp4.6 pp9.4 pp14.2 pp8.6 pp7.2 pp3.7 pp8.5 pp
Median latency382ms530ms466ms347ms470ms505ms464ms461ms273ms449ms376ms206ms280ms602ms302ms269ms961ms401ms349ms431ms453ms472ms340ms285ms473ms355ms544ms465ms270ms700ms539ms393ms422ms596ms585ms
Gold relation
Entailment100.0(50)100.0(50)100.0(50)100.0(50)100.0(50)96.0(50)98.0(50)98.0(50)98.0(50)100.0(50)94.0(50)94.0(50)100.0(50)90.0(50)98.0(50)80.0(50)86.0(50)76.0(50)92.0(50)96.0(50)98.0(50)84.0(50)98.0(50)96.0(50)90.0(50)94.0(50)88.0(50)98.0(50)84.0(50)90.0(50)58.0(50)92.0(50)90.0(50)70.0(50)54.0(50)
Neutral98.0(50)96.0(50)98.0(50)98.0(50)96.0(50)96.0(50)96.0(50)94.0(50)92.0(50)94.0(50)90.0(50)94.0(50)90.0(50)92.0(50)96.0(50)96.0(50)94.0(50)90.0(50)94.0(50)86.0(50)90.0(50)86.0(50)88.0(50)92.0(50)94.0(50)58.0(50)94.0(50)92.0(50)78.0(50)88.0(50)66.0(50)80.0(50)80.0(50)54.0(50)84.0(50)
Contradiction98.0(50)100.0(50)100.0(50)98.0(50)100.0(50)100.0(50)94.0(50)94.0(50)98.0(50)100.0(50)92.0(50)90.0(50)100.0(50)86.0(50)94.0(50)70.0(50)84.0(50)74.0(50)90.0(50)92.0(50)94.0(50)74.0(50)94.0(50)92.0(50)92.0(50)84.0(50)84.0(50)92.0(50)76.0(50)82.0(50)74.0(50)90.0(50)78.0(50)68.0(50)34.0(50)

KLUE STS

Korean
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy52.0(100)58.0(100)62.0(100)53.0(100)56.0(100)48.0(100)51.0(100)44.0(100)46.0(100)37.0(100)38.0(100)35.0(100)52.0(100)46.0(100)44.0(100)41.0(100)34.0(100)48.0(100)39.0(100)36.0(100)49.0(100)33.0(100)51.0(100)49.0(100)41.0(100)32.0(100)41.0(100)30.0(100)33.0(100)37.0(100)30.0(100)29.0(100)34.0(100)26.0(100)31.0(100)
Option-order flips1.0% (100)8.0% (100)15.0% (100)6.0% (100)9.0% (100)17.0% (100)16.0% (100)25.0% (100)29.0% (100)13.0% (100)13.0% (100)17.0% (100)34.0% (100)22.0% (100)27.0% (100)30.0% (100)39.0% (100)25.0% (100)33.0% (100)32.0% (100)7.0% (100)23.0% (100)3.0% (100)14.0% (100)24.0% (100)16.0% (100)33.0% (100)48.0% (100)20.0% (100)44.0% (100)28.0% (100)19.0% (100)16.0% (100)7.0% (100)90.0% (100)
Confidence vs. accuracy gap11.4 pp10.1 pp7.1 pp13.0 pp11.6 pp9.5 pp8.3 pp9.6 pp3.1 pp13.1 pp22.1 pp26.3 pp19.3 pp24.8 pp14.0 pp29.8 pp29.6 pp49.1 pp14.1 pp17.5 pp8.5 pp16.2 pp15.4 pp8.0 pp6.4 pp23.7 pp6.2 pp27.3 pp19.3 pp18.8 pp29.1 pp21.7 pp11.4 pp21.2 pp9.2 pp
Level error0.550.520.480.590.540.660.570.710.730.880.820.880.570.680.610.760.970.630.730.890.660.850.560.660.720.910.730.880.900.751.131.030.931.080.99
Median latency383ms528ms476ms346ms471ms511ms468ms464ms343ms840ms387ms208ms279ms602ms303ms237ms911ms6.5s365ms433ms452ms525ms342ms285ms497ms351ms556ms466ms257ms697ms571ms396ms423ms608ms611ms
Human-rated level
Level 070.6(17)76.5(17)70.6(17)70.6(17)70.6(17)47.1(17)70.6(17)64.7(17)5.9(17)52.9(17)11.8(17)5.9(17)88.2(17)47.1(17)94.1(17)11.8(17)35.3(17)52.9(17)64.7(17)76.5(17)47.1(17)23.5(17)70.6(17)47.1(17)76.5(17)52.9(17)64.7(17)35.3(17)47.1(17)58.8(17)29.4(17)35.3(17)64.7(17)17.6(17)29.4(17)
Level 141.2(17)29.4(17)41.2(17)29.4(17)70.6(17)70.6(17)23.5(17)29.4(17)76.5(17)29.4(17)41.2(17)41.2(17)29.4(17)64.7(17)5.9(17)58.8(17)29.4(17)52.9(17)0.0(17)0.0(17)52.9(17)35.3(17)41.2(17)58.8(17)11.8(17)11.8(17)17.6(17)23.5(17)5.9(17)5.9(17)29.4(17)0.0(17)17.6(17)0.0(17)0.0(17)
Level 237.5(16)50.0(16)62.5(16)37.5(16)37.5(16)43.8(16)50.0(16)18.8(16)25.0(16)6.3(16)18.8(16)18.8(16)18.8(16)25.0(16)12.5(16)62.5(16)0.0(16)12.5(16)31.3(16)0.0(16)6.3(16)0.0(16)25.0(16)25.0(16)18.8(16)0.0(16)18.8(16)12.5(16)0.0(16)25.0(16)0.0(16)0.0(16)6.3(16)25.0(16)25.0(16)
Level 331.3(16)43.8(16)50.0(16)37.5(16)56.3(16)31.3(16)68.8(16)37.5(16)37.5(16)18.8(16)25.0(16)18.8(16)50.0(16)37.5(16)43.8(16)68.8(16)18.8(16)56.3(16)25.0(16)50.0(16)50.0(16)37.5(16)68.8(16)37.5(16)18.8(16)25.0(16)37.5(16)18.8(16)43.8(16)43.8(16)18.8(16)31.3(16)6.3(16)0.0(16)18.8(16)
Level 452.9(17)76.5(17)88.2(17)70.6(17)88.2(17)76.5(17)76.5(17)100.0(17)64.7(17)64.7(17)76.5(17)58.8(17)64.7(17)17.6(17)70.6(17)47.1(17)88.2(17)88.2(17)41.2(17)47.1(17)88.2(17)76.5(17)64.7(17)76.5(17)41.2(17)76.5(17)100.0(17)82.4(17)82.4(17)88.2(17)100.0(17)64.7(17)64.7(17)52.9(17)47.1(17)
Level 576.5(17)70.6(17)58.8(17)70.6(17)11.8(17)17.6(17)17.6(17)11.8(17)64.7(17)47.1(17)52.9(17)64.7(17)58.8(17)82.4(17)35.3(17)0.0(17)29.4(17)23.5(17)70.6(17)41.2(17)47.1(17)23.5(17)35.3(17)47.1(17)76.5(17)23.5(17)5.9(17)5.9(17)17.6(17)0.0(17)0.0(17)41.2(17)41.2(17)58.8(17)64.7(17)
JGLUE dataset evaluation details

JGLUE JCommonsenseQA

Japanese
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy98.7(75)97.3(75)98.7(75)98.7(75)98.7(75)98.7(75)98.7(75)97.3(75)97.3(75)97.3(75)97.3(75)97.3(75)97.3(75)96.0(75)97.3(75)96.0(75)97.3(75)97.3(75)97.3(75)98.7(75)92.0(75)98.7(75)89.3(75)94.7(75)98.7(75)92.0(75)90.7(75)96.0(75)98.7(75)96.0(75)89.3(75)86.7(75)82.7(75)65.3(75)73.3(75)
Option-order flips1.3% (75)5.3% (75)1.3% (75)0.0% (75)2.7% (75)0.0% (75)2.7% (75)1.3% (75)0.0% (75)1.3% (75)0.0% (75)0.0% (75)0.0% (75)0.0% (75)4.0% (75)5.3% (75)2.7% (75)5.3% (75)1.3% (75)0.0% (75)5.3% (75)0.0% (75)6.7% (75)2.7% (75)2.7% (75)0.0% (75)16.0% (75)2.7% (75)0.0% (75)14.7% (75)18.7% (75)13.3% (75)5.3% (75)5.3% (75)14.7% (75)
Confidence vs. accuracy gap2.9 pp4.3 pp3.2 pp2.5 pp4.5 pp3.9 pp5.1 pp3.3 pp5.6 pp2.3 pp1.6 pp1.9 pp2.6 pp3.7 pp6.3 pp5.3 pp3.9 pp3.9 pp2.5 pp3.7 pp4.0 pp7.7 pp7.1 pp3.8 pp6.7 pp10.5 pp15.9 pp3.4 pp8.0 pp12.6 pp9.4 pp6.6 pp12.0 pp12.0 pp14.0 pp
Median latency377ms534ms497ms328ms468ms517ms466ms466ms380ms445ms375ms217ms293ms647ms292ms286ms542ms420ms371ms434ms449ms491ms435ms283ms648ms346ms550ms560ms252ms685ms551ms392ms427ms611ms611ms
Position of the correct option
Option 1100.0(15)93.3(15)93.3(15)93.3(15)93.3(15)93.3(15)93.3(15)93.3(15)93.3(15)93.3(15)93.3(15)93.3(15)86.7(15)93.3(15)93.3(15)93.3(15)93.3(15)100.0(15)93.3(15)100.0(15)93.3(15)93.3(15)93.3(15)93.3(15)100.0(15)93.3(15)93.3(15)100.0(15)100.0(15)93.3(15)86.7(15)86.7(15)80.0(15)73.3(15)86.7(15)
Option 2100.0(15)93.3(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)93.3(15)100.0(15)93.3(15)100.0(15)100.0(15)100.0(15)86.7(15)100.0(15)73.3(15)93.3(15)100.0(15)93.3(15)80.0(15)86.7(15)100.0(15)100.0(15)86.7(15)73.3(15)80.0(15)73.3(15)60.0(15)
Option 3100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)93.3(15)100.0(15)100.0(15)93.3(15)100.0(15)100.0(15)100.0(15)93.3(15)93.3(15)100.0(15)93.3(15)66.7(15)73.3(15)
Option 4100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)93.3(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)86.7(15)100.0(15)86.7(15)93.3(15)100.0(15)93.3(15)100.0(15)100.0(15)100.0(15)100.0(15)86.7(15)93.3(15)93.3(15)66.7(15)80.0(15)
Option 593.3(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)100.0(15)93.3(15)93.3(15)93.3(15)93.3(15)93.3(15)100.0(15)93.3(15)100.0(15)86.7(15)100.0(15)86.7(15)93.3(15)93.3(15)93.3(15)100.0(15)100.0(15)93.3(15)93.3(15)86.7(15)80.0(15)93.3(15)93.3(15)93.3(15)93.3(15)80.0(15)66.7(15)46.7(15)66.7(15)

JGLUE JNLI

Japanese
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy88.0(100)87.0(100)86.0(100)84.0(100)82.0(100)83.0(100)79.0(100)80.0(100)82.0(100)86.0(100)76.0(100)78.0(100)85.0(100)77.0(100)84.0(100)64.0(100)75.0(100)60.0(100)83.0(100)84.0(100)85.0(100)81.0(100)81.0(100)81.0(100)79.0(100)74.0(100)73.0(100)79.0(100)72.0(100)75.0(100)58.0(100)77.0(100)71.0(100)59.0(100)61.0(100)
Option-order flips2.0% (100)0.0% (100)0.0% (100)1.0% (100)4.0% (100)3.0% (100)5.0% (100)9.0% (100)0.0% (100)2.0% (100)2.0% (100)0.0% (100)4.0% (100)2.0% (100)2.0% (100)2.0% (100)5.0% (100)9.0% (100)4.0% (100)0.0% (100)1.0% (100)0.0% (100)0.0% (100)2.0% (100)3.0% (100)0.0% (100)6.0% (100)7.0% (100)0.0% (100)4.0% (100)13.0% (100)3.0% (100)0.0% (100)0.0% (100)19.0% (100)
Confidence vs. accuracy gap5.9 pp5.8 pp7.0 pp7.7 pp8.7 pp6.6 pp6.9 pp5.7 pp9.8 pp2.6 pp12.7 pp13.6 pp10.4 pp16.5 pp7.0 pp27.9 pp13.1 pp33.5 pp7.6 pp9.9 pp4.7 pp3.0 pp5.7 pp10.2 pp9.6 pp8.6 pp12.4 pp13.5 pp6.4 pp17.7 pp22.7 pp18.3 pp13.6 pp13.6 pp7.3 pp
Median latency384ms528ms466ms331ms467ms511ms466ms463ms277ms448ms376ms211ms341ms644ms310ms299ms520ms384ms355ms432ms457ms537ms338ms288ms475ms346ms546ms468ms311ms693ms555ms395ms425ms609ms617ms
Gold relation
Entailment87.9(33)90.9(33)93.9(33)90.9(33)78.8(33)84.8(33)78.8(33)81.8(33)93.9(33)90.9(33)78.8(33)81.8(33)81.8(33)90.9(33)84.8(33)75.8(33)81.8(33)72.7(33)81.8(33)90.9(33)90.9(33)84.8(33)81.8(33)81.8(33)84.8(33)93.9(33)90.9(33)84.8(33)81.8(33)81.8(33)72.7(33)81.8(33)75.8(33)57.6(33)81.8(33)
Neutral82.4(34)85.3(34)82.4(34)82.4(34)88.2(34)91.2(34)88.2(34)91.2(34)94.1(34)82.4(34)97.1(34)97.1(34)88.2(34)94.1(34)97.1(34)97.1(34)100.0(34)85.3(34)94.1(34)94.1(34)85.3(34)94.1(34)91.2(34)91.2(34)97.1(34)76.5(34)82.4(34)94.1(34)88.2(34)97.1(34)76.5(34)85.3(34)79.4(34)79.4(34)50.0(34)
Contradiction93.9(33)84.8(33)81.8(33)78.8(33)78.8(33)72.7(33)69.7(33)66.7(33)57.6(33)84.8(33)51.5(33)54.5(33)84.8(33)45.5(33)69.7(33)18.2(33)42.4(33)21.2(33)72.7(33)66.7(33)78.8(33)63.6(33)69.7(33)69.7(33)54.5(33)51.5(33)45.5(33)57.6(33)45.5(33)45.5(33)24.2(33)63.6(33)57.6(33)39.4(33)51.5(33)

JGLUE JSTS

Japanese
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy53.3(75)58.7(75)58.7(75)61.3(75)49.3(75)54.7(75)53.3(75)56.0(75)53.3(75)52.0(75)49.3(75)56.0(75)54.7(75)56.0(75)49.3(75)42.7(75)49.3(75)50.7(75)58.7(75)49.3(75)57.3(75)49.3(75)48.0(75)53.3(75)49.3(75)53.3(75)50.7(75)50.7(75)44.0(75)37.3(75)41.3(75)50.7(75)42.7(75)50.7(75)53.3(75)
Option-order flips4.0% (75)8.0% (75)14.7% (75)4.0% (75)6.7% (75)4.0% (75)12.0% (75)18.7% (75)12.0% (75)13.3% (75)12.0% (75)16.0% (75)32.0% (75)9.3% (75)28.0% (75)25.3% (75)38.7% (75)25.3% (75)18.7% (75)16.0% (75)9.3% (75)24.0% (75)1.3% (75)9.3% (75)21.3% (75)14.7% (75)21.3% (75)22.7% (75)9.3% (75)25.3% (75)24.0% (75)12.0% (75)28.0% (75)13.3% (75)84.0% (75)
Confidence vs. accuracy gap16.3 pp14.4 pp12.3 pp12.4 pp11.5 pp8.1 pp6.1 pp13.8 pp18.9 pp13.2 pp22.3 pp21.7 pp20.6 pp21.3 pp14.6 pp35.3 pp22.1 pp45.3 pp11.5 pp13.2 pp11.1 pp15.2 pp19.1 pp13.8 pp19.3 pp11.5 pp10.0 pp19.5 pp18.2 pp18.0 pp21.3 pp22.2 pp12.9 pp9.2 pp9.7 pp
Level error0.500.510.480.480.560.540.590.560.570.590.560.570.580.470.670.650.670.560.510.620.560.640.600.530.570.650.580.650.700.710.720.690.690.660.69
Median latency386ms519ms467ms328ms468ms506ms466ms462ms265ms448ms374ms219ms324ms606ms293ms357ms788ms6.8s383ms431ms456ms485ms340ms288ms480ms350ms545ms460ms294ms692ms577ms397ms426ms610ms635ms
Human-rated level
Level 083.3(12)91.7(12)91.7(12)91.7(12)83.3(12)83.3(12)83.3(12)83.3(12)83.3(12)91.7(12)75.0(12)58.3(12)100.0(12)83.3(12)100.0(12)75.0(12)83.3(12)100.0(12)100.0(12)91.7(12)83.3(12)75.0(12)91.7(12)83.3(12)100.0(12)91.7(12)100.0(12)91.7(12)83.3(12)75.0(12)83.3(12)83.3(12)100.0(12)83.3(12)83.3(12)
Level 115.4(13)7.7(13)15.4(13)0.0(13)23.1(13)23.1(13)7.7(13)7.7(13)30.8(13)0.0(13)15.4(13)23.1(13)30.8(13)15.4(13)15.4(13)23.1(13)23.1(13)23.1(13)0.0(13)0.0(13)7.7(13)30.8(13)0.0(13)7.7(13)0.0(13)0.0(13)0.0(13)0.0(13)0.0(13)0.0(13)30.8(13)0.0(13)7.7(13)46.2(13)15.4(13)
Level 241.7(12)50.0(12)58.3(12)66.7(12)41.7(12)58.3(12)83.3(12)66.7(12)50.0(12)33.3(12)33.3(12)50.0(12)41.7(12)41.7(12)25.0(12)66.7(12)0.0(12)41.7(12)41.7(12)25.0(12)58.3(12)16.7(12)50.0(12)41.7(12)16.7(12)25.0(12)41.7(12)33.3(12)8.3(12)25.0(12)8.3(12)0.0(12)0.0(12)25.0(12)41.7(12)
Level 361.5(13)76.9(13)53.8(13)76.9(13)61.5(13)61.5(13)61.5(13)69.2(13)46.2(13)76.9(13)69.2(13)61.5(13)46.2(13)61.5(13)61.5(13)38.5(13)76.9(13)53.8(13)84.6(13)100.0(13)76.9(13)84.6(13)61.5(13)84.6(13)69.2(13)69.2(13)61.5(13)76.9(13)76.9(13)38.5(13)38.5(13)76.9(13)23.1(13)15.4(13)61.5(13)
Level 441.7(12)50.0(12)50.0(12)58.3(12)41.7(12)58.3(12)41.7(12)66.7(12)58.3(12)58.3(12)41.7(12)66.7(12)33.3(12)50.0(12)41.7(12)25.0(12)75.0(12)50.0(12)58.3(12)33.3(12)66.7(12)58.3(12)33.3(12)58.3(12)41.7(12)66.7(12)58.3(12)58.3(12)58.3(12)58.3(12)66.7(12)83.3(12)66.7(12)83.3(12)41.7(12)
Level 576.9(13)76.9(13)84.6(13)76.9(13)46.2(13)46.2(13)46.2(13)46.2(13)53.8(13)53.8(13)61.5(13)76.9(13)76.9(13)84.6(13)53.8(13)30.8(13)38.5(13)38.5(13)69.2(13)46.2(13)53.8(13)30.8(13)53.8(13)46.2(13)69.2(13)69.2(13)46.2(13)46.2(13)38.5(13)30.8(13)23.1(13)61.5(13)61.5(13)53.8(13)76.9(13)
KMMLU · JMMLU dataset evaluation details

KMMLU

Korean
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy70.4(125)68.8(125)68.0(125)68.0(125)72.8(125)75.2(125)73.6(125)70.4(125)78.4(125)70.4(125)68.8(125)69.6(125)64.0(125)57.6(125)61.6(125)67.2(125)63.2(125)68.8(125)63.2(125)64.8(125)48.0(125)67.2(125)36.8(125)54.4(125)64.8(125)59.2(125)44.0(125)48.0(125)66.4(125)44.8(125)50.4(125)36.8(125)25.6(125)24.8(125)31.2(125)
Option-order flips8.8% (125)15.2% (125)13.6% (125)16.0% (125)12.0% (125)12.8% (125)23.2% (125)20.0% (125)0.0% (125)11.2% (125)16.0% (125)19.2% (125)25.6% (125)1.6% (125)32.0% (125)30.6% (124)26.4% (125)32.0% (125)26.4% (125)20.8% (125)22.4% (125)0.0% (125)26.4% (125)20.8% (125)35.2% (125)0.0% (125)52.8% (125)35.2% (125)0.0% (125)48.8% (125)40.8% (125)49.6% (125)33.6% (125)31.2% (125)47.2% (125)
Confidence vs. accuracy gap11.4 pp11.7 pp12.7 pp11.5 pp7.9 pp4.7 pp6.8 pp6.2 pp6.1 pp12.5 pp9.9 pp6.0 pp15.6 pp19.1 pp4.9 pp7.1 pp14.0 pp19.6 pp9.4 pp8.6 pp22.5 pp9.0 pp14.1 pp17.5 pp6.8 pp11.4 pp5.8 pp25.8 pp9.1 pp5.8 pp17.1 pp14.6 pp17.4 pp28.3 pp26.2 pp
Median latency379ms528ms459ms343ms469ms507ms465ms478ms274ms451ms379ms206ms269ms605ms293ms251ms840ms8.2s355ms437ms446ms482ms376ms287ms482ms346ms552ms506ms247ms702ms561ms395ms431ms607ms606ms
Field
Humanities & social sciences77.4(31)77.4(31)71.0(31)77.4(31)80.6(31)90.3(31)80.6(31)71.0(31)80.6(31)71.0(31)71.0(31)67.7(31)77.4(31)61.3(31)71.0(31)80.6(31)80.6(31)87.1(31)80.6(31)71.0(31)58.1(31)80.6(31)38.7(31)61.3(31)77.4(31)71.0(31)48.4(31)45.2(31)67.7(31)45.2(31)74.2(31)35.5(31)16.1(31)19.4(31)32.3(31)
STEM56.3(32)62.5(32)59.4(32)56.3(32)65.6(32)75.0(32)68.8(32)75.0(32)90.6(32)71.9(32)68.8(32)65.6(32)62.5(32)59.4(32)62.5(32)65.6(32)65.6(32)68.8(32)59.4(32)68.8(32)34.4(32)56.3(32)43.8(32)50.0(32)68.8(32)59.4(32)40.6(32)50.0(32)65.6(32)50.0(32)46.9(32)50.0(32)31.3(32)28.1(32)34.4(32)
Applied science71.9(32)65.6(32)75.0(32)62.5(32)71.9(32)59.4(32)65.6(32)59.4(32)68.8(32)75.0(32)71.9(32)65.6(32)62.5(32)50.0(32)56.3(32)50.0(32)50.0(32)65.6(32)53.1(32)56.3(32)46.9(32)59.4(32)28.1(32)46.9(32)53.1(32)56.3(32)34.4(32)40.6(32)62.5(32)40.6(32)28.1(32)40.6(32)25.0(32)31.3(32)31.3(32)
Other licenses76.7(30)70.0(30)66.7(30)76.7(30)73.3(30)76.7(30)80.0(30)76.7(30)73.3(30)63.3(30)63.3(30)80.0(30)53.3(30)60.0(30)56.7(30)73.3(30)56.7(30)53.3(30)60.0(30)63.3(30)53.3(30)73.3(30)36.7(30)60.0(30)60.0(30)50.0(30)53.3(30)56.7(30)70.0(30)43.3(30)53.3(30)20.0(30)30.0(30)20.0(30)26.7(30)
Test-taker accuracy
Under 40% (hard)66.7(63)63.5(63)65.1(63)63.5(63)63.5(63)66.7(63)61.9(63)61.9(63)76.2(63)69.8(63)66.7(63)65.1(63)54.0(63)42.9(63)55.6(63)60.3(63)52.4(63)55.6(63)54.0(63)58.7(63)34.9(63)61.9(63)31.7(63)39.7(63)50.8(63)52.4(63)39.7(63)36.5(63)55.6(63)42.9(63)38.1(63)34.9(63)28.6(63)25.4(63)25.4(63)
40–70%76.4(55)76.4(55)72.7(55)74.5(55)83.6(55)83.6(55)85.5(55)78.2(55)85.5(55)69.1(55)72.7(55)76.4(55)76.4(55)70.9(55)69.1(55)78.2(55)74.5(55)81.8(55)70.9(55)69.1(55)61.8(55)70.9(55)36.4(55)70.9(55)81.8(55)67.3(55)49.1(55)60.0(55)78.2(55)43.6(55)61.8(55)36.4(55)25.5(55)21.8(55)36.4(55)
70% or more (easy)66.7(3)66.7(3)66.7(3)66.7(3)66.7(3)66.7(3)100.0(3)100.0(3)33.3(3)100.0(3)66.7(3)66.7(3)66.7(3)100.0(3)66.7(3)33.3(3)100.0(3)100.0(3)100.0(3)66.7(3)66.7(3)100.0(3)66.7(3)66.7(3)66.7(3)66.7(3)66.7(3)66.7(3)66.7(3)100.0(3)66.7(3)33.3(3)0.0(3)0.0(3)66.7(3)
Position of the correct option
A70.0(30)63.3(30)60.0(30)63.3(30)70.0(30)76.7(30)70.0(30)63.3(30)73.3(30)60.0(30)53.3(30)53.3(30)56.7(30)53.3(30)50.0(30)63.3(30)56.7(30)66.7(30)53.3(30)53.3(30)56.7(30)60.0(30)33.3(30)60.0(30)66.7(30)50.0(30)53.3(30)50.0(30)70.0(30)26.7(30)43.3(30)33.3(30)20.0(30)23.3(30)33.3(30)
B65.6(32)65.6(32)68.8(32)71.9(32)81.3(32)71.9(32)81.3(32)68.8(32)87.5(32)78.1(32)75.0(32)84.4(32)56.3(32)65.6(32)62.5(32)65.6(32)71.9(32)65.6(32)62.5(32)62.5(32)46.9(32)59.4(32)31.3(32)50.0(32)59.4(32)59.4(32)40.6(32)40.6(32)62.5(32)37.5(32)50.0(32)28.1(32)21.9(32)34.4(32)34.4(32)
C68.8(32)68.8(32)62.5(32)56.3(32)65.6(32)62.5(32)62.5(32)68.8(32)71.9(32)68.8(32)75.0(32)65.6(32)65.6(32)65.6(32)65.6(32)68.8(32)62.5(32)78.1(32)78.1(32)68.8(32)40.6(32)68.8(32)46.9(32)53.1(32)68.8(32)56.3(32)53.1(32)56.3(32)59.4(32)62.5(32)71.9(32)46.9(32)37.5(32)18.8(32)40.6(32)
D77.4(31)77.4(31)80.6(31)80.6(31)74.2(31)90.3(31)80.6(31)80.6(31)80.6(31)74.2(31)71.0(31)74.2(31)77.4(31)45.2(31)67.7(31)71.0(31)61.3(31)64.5(31)58.1(31)74.2(31)48.4(31)80.6(31)35.5(31)54.8(31)64.5(31)71.0(31)29.0(31)45.2(31)74.2(31)51.6(31)35.5(31)38.7(31)22.6(31)22.6(31)16.1(31)

JMMLU

Japanese
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy87.2(125)84.8(125)87.2(125)83.2(125)84.0(125)91.2(125)88.8(125)85.6(125)91.1(123)88.8(125)88.8(125)87.2(125)77.6(125)84.8(125)81.6(125)85.6(125)84.8(125)82.4(125)83.2(125)80.0(125)78.4(125)83.2(125)63.2(125)76.0(125)84.8(125)84.0(125)69.6(125)77.6(125)88.8(125)67.2(125)68.8(125)66.4(125)51.2(125)45.6(125)52.0(125)
Option-order flips5.6% (125)4.8% (125)4.8% (125)8.0% (125)3.2% (125)3.2% (125)4.8% (125)8.8% (125)0.0% (123)5.6% (125)2.4% (125)1.6% (125)11.2% (125)1.6% (125)15.2% (125)5.7% (124)12.0% (125)12.8% (125)15.2% (125)12.0% (125)6.4% (125)0.0% (125)23.2% (125)12.0% (125)9.6% (125)0.0% (125)28.8% (125)13.6% (125)0.0% (125)24.8% (125)27.2% (125)30.4% (125)22.4% (125)32.0% (125)34.4% (125)
Confidence vs. accuracy gap8.6 pp7.1 pp7.1 pp8.7 pp6.2 pp3.7 pp4.6 pp5.9 pp6.3 pp3.4 pp3.0 pp4.3 pp13.7 pp11.0 pp4.9 pp5.2 pp6.5 pp11.4 pp9.4 pp5.7 pp9.7 pp6.4 pp9.4 pp10.1 pp4.6 pp7.9 pp9.2 pp9.5 pp4.8 pp6.4 pp11.5 pp12.8 pp7.9 pp15.8 pp12.0 pp
Median latency379ms542ms477ms335ms468ms503ms467ms463ms245ms457ms377ms209ms304ms623ms291ms294ms651ms338ms369ms431ms466ms503ms384ms284ms490ms352ms584ms490ms277ms694ms568ms398ms426ms593ms601ms
Question origin
Japan-specific88.9(63)88.9(63)92.1(63)92.1(63)88.9(63)96.8(63)93.7(63)88.9(63)96.8(63)92.1(63)92.1(63)90.5(63)84.1(63)81.0(63)88.9(63)93.7(63)88.9(63)87.3(63)92.1(63)82.5(63)84.1(63)85.7(63)74.6(63)79.4(63)92.1(63)87.3(63)76.2(63)82.5(63)87.3(63)73.0(63)82.5(63)77.8(63)52.4(63)54.0(63)60.3(63)
Translated from MMLU85.5(62)80.6(62)82.3(62)74.2(62)79.0(62)85.5(62)83.9(62)82.3(62)85.0(60)85.5(62)85.5(62)83.9(62)71.0(62)88.7(62)74.2(62)77.4(62)80.6(62)77.4(62)74.2(62)77.4(62)72.6(62)80.6(62)51.6(62)72.6(62)77.4(62)80.6(62)62.9(62)72.6(62)90.3(62)61.3(62)54.8(62)54.8(62)50.0(62)37.1(62)43.5(62)
Japan-specific subjects
Japanese history61.5(13)76.9(13)76.9(13)76.9(13)69.2(13)92.3(13)84.6(13)76.9(13)92.3(13)76.9(13)84.6(13)84.6(13)61.5(13)46.2(13)76.9(13)84.6(13)69.2(13)61.5(13)84.6(13)53.8(13)46.2(13)69.2(13)53.8(13)46.2(13)84.6(13)69.2(13)53.8(13)61.5(13)69.2(13)46.2(13)76.9(13)61.5(13)30.8(13)38.5(13)23.1(13)
World history100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)100.0(13)92.3(13)100.0(13)100.0(13)100.0(13)100.0(13)84.6(13)100.0(13)100.0(13)100.0(13)100.0(13)92.3(13)100.0(13)84.6(13)92.3(13)92.3(13)38.5(13)53.8(13)84.6(13)
Civics92.3(13)92.3(13)100.0(13)92.3(13)100.0(13)100.0(13)100.0(13)84.6(13)100.0(13)100.0(13)92.3(13)92.3(13)92.3(13)84.6(13)84.6(13)100.0(13)92.3(13)92.3(13)92.3(13)84.6(13)100.0(13)92.3(13)84.6(13)92.3(13)92.3(13)92.3(13)76.9(13)84.6(13)92.3(13)84.6(13)69.2(13)84.6(13)76.9(13)61.5(13)61.5(13)
Japanese geography91.7(12)91.7(12)91.7(12)91.7(12)83.3(12)91.7(12)91.7(12)91.7(12)91.7(12)91.7(12)91.7(12)83.3(12)83.3(12)83.3(12)83.3(12)91.7(12)91.7(12)91.7(12)91.7(12)83.3(12)83.3(12)75.0(12)58.3(12)75.0(12)91.7(12)83.3(12)66.7(12)83.3(12)83.3(12)58.3(12)75.0(12)66.7(12)50.0(12)66.7(12)58.3(12)
Idioms100.0(12)83.3(12)91.7(12)100.0(12)91.7(12)100.0(12)91.7(12)91.7(12)100.0(12)91.7(12)91.7(12)91.7(12)83.3(12)91.7(12)100.0(12)91.7(12)91.7(12)100.0(12)91.7(12)91.7(12)91.7(12)91.7(12)91.7(12)83.3(12)91.7(12)91.7(12)83.3(12)91.7(12)91.7(12)91.7(12)100.0(12)83.3(12)66.7(12)50.0(12)75.0(12)
Position of the correct option
A90.0(30)90.0(30)86.7(30)83.3(30)83.3(30)90.0(30)90.0(30)90.0(30)89.7(29)93.3(30)93.3(30)90.0(30)76.7(30)86.7(30)80.0(30)90.0(30)76.7(30)86.7(30)73.3(30)86.7(30)76.7(30)86.7(30)63.3(30)76.7(30)83.3(30)83.3(30)83.3(30)76.7(30)90.0(30)53.3(30)70.0(30)66.7(30)50.0(30)40.0(30)56.7(30)
B81.3(32)75.0(32)84.4(32)78.1(32)75.0(32)87.5(32)87.5(32)75.0(32)87.1(31)81.3(32)90.6(32)90.6(32)68.8(32)81.3(32)75.0(32)78.1(32)78.1(32)75.0(32)84.4(32)71.9(32)75.0(32)71.9(32)62.5(32)62.5(32)84.4(32)84.4(32)62.5(32)75.0(32)84.4(32)65.6(32)71.9(32)53.1(32)40.6(32)59.4(32)43.8(32)
C93.5(31)87.1(31)96.8(31)87.1(31)90.3(31)100.0(31)93.5(31)93.5(31)93.5(31)96.8(31)90.3(31)87.1(31)90.3(31)87.1(31)87.1(31)87.1(31)96.8(31)90.3(31)93.5(31)77.4(31)83.9(31)90.3(31)54.8(31)90.3(31)87.1(31)87.1(31)58.1(31)77.4(31)93.5(31)71.0(31)87.1(31)74.2(31)71.0(31)41.9(31)54.8(31)
D84.4(32)87.5(32)81.3(32)84.4(32)87.5(32)87.5(32)84.4(32)84.4(32)93.8(32)84.4(32)81.3(32)81.3(32)75.0(32)84.4(32)84.4(32)87.5(32)87.5(32)78.1(32)81.3(32)84.4(32)78.1(32)84.4(32)71.9(32)75.0(32)84.4(32)81.3(32)75.0(32)81.3(32)87.5(32)78.1(32)46.9(32)71.9(32)43.8(32)40.6(32)53.1(32)
M-RewardBench dataset evaluation details

M-RewardBench (한국어)

Korean
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy91.2(125)90.4(125)92.0(125)90.4(125)88.8(125)88.0(125)84.0(125)84.8(125)82.4(125)90.4(125)86.4(125)87.2(125)88.0(125)83.2(125)88.0(125)87.2(125)86.4(125)78.4(125)85.6(125)86.4(125)80.0(125)80.0(125)70.4(125)83.2(125)80.8(125)76.0(125)69.6(125)78.4(125)72.8(125)83.2(125)68.0(125)72.8(125)73.6(125)60.0(125)65.6(125)
Option-order flips1.6% (125)0.8% (125)3.2% (125)1.6% (125)0.8% (125)3.2% (125)4.0% (125)7.2% (125)0.0% (117)0.0% (125)1.6% (125)2.4% (125)6.4% (125)0.8% (125)3.2% (125)4.0% (125)3.2% (125)22.4% (125)3.2% (125)0.8% (125)2.4% (125)0.0% (125)1.6% (125)0.8% (125)7.2% (125)0.0% (125)64.8% (125)3.2% (125)0.0% (125)1.6% (125)29.6% (125)8.1% (123)0.8% (125)4.0% (125)3.2% (125)
Confidence vs. accuracy gap4.4 pp4.6 pp3.6 pp6.1 pp6.8 pp4.3 pp4.9 pp5.5 pp6.7 pp5.5 pp6.1 pp3.2 pp9.7 pp11.6 pp3.1 pp5.3 pp10.0 pp18.5 pp1.7 pp5.0 pp6.8 pp15.2 pp7.8 pp10.3 pp7.5 pp18.1 pp3.6 pp14.6 pp18.7 pp3.4 pp18.1 pp13.7 pp10.3 pp12.4 pp11.3 pp
Median latency383ms537ms471ms340ms488ms585ms483ms474ms354ms499ms373ms217ms275ms632ms297ms224ms581ms520ms360ms440ms454ms576ms358ms285ms482ms356ms556ms485ms244ms694ms614ms414ms425ms637ms596ms
Category
Chat96.9(32)100.0(32)100.0(32)100.0(32)93.8(32)90.6(32)96.9(32)93.8(32)93.8(32)87.5(32)96.9(32)96.9(32)90.6(32)93.8(32)93.8(32)90.6(32)96.9(32)87.5(32)93.8(32)84.4(32)96.9(32)96.9(32)93.8(32)93.8(32)87.5(32)87.5(32)78.1(32)96.9(32)93.8(32)96.9(32)84.4(32)78.1(32)87.5(32)90.6(32)87.5(32)
Chat Hard80.6(31)80.6(31)74.2(31)74.2(31)77.4(31)74.2(31)74.2(31)77.4(31)77.4(31)83.9(31)74.2(31)74.2(31)71.0(31)67.7(31)74.2(31)77.4(31)74.2(31)54.8(31)67.7(31)77.4(31)54.8(31)51.6(31)38.7(31)67.7(31)54.8(31)45.2(31)54.8(31)54.8(31)48.4(31)67.7(31)41.9(31)45.2(31)58.1(31)32.3(31)35.5(31)
Safety90.3(31)80.6(31)96.8(31)90.3(31)90.3(31)90.3(31)71.0(31)74.2(31)64.5(31)96.8(31)83.9(31)87.1(31)96.8(31)80.6(31)93.5(31)90.3(31)80.6(31)83.9(31)87.1(31)87.1(31)71.0(31)87.1(31)71.0(31)80.6(31)87.1(31)80.6(31)83.9(31)77.4(31)71.0(31)77.4(31)83.9(31)83.9(31)64.5(31)64.5(31)71.0(31)
Reasoning96.8(31)100.0(31)96.8(31)96.8(31)93.5(31)96.8(31)93.5(31)93.5(31)93.5(31)93.5(31)90.3(31)90.3(31)93.5(31)90.3(31)90.3(31)90.3(31)93.5(31)87.1(31)93.5(31)96.8(31)96.8(31)83.9(31)77.4(31)90.3(31)93.5(31)90.3(31)61.3(31)83.9(31)77.4(31)90.3(31)61.3(31)83.9(31)83.9(31)51.6(31)67.7(31)
Position of the correct option
Better answer is A85.5(62)88.7(62)88.7(62)85.5(62)83.9(62)85.5(62)83.9(62)83.9(62)83.9(62)85.5(62)83.9(62)83.9(62)88.7(62)79.0(62)83.9(62)88.7(62)80.6(62)64.5(62)83.9(62)80.6(62)79.0(62)75.8(62)71.0(62)80.6(62)74.2(62)69.4(62)56.5(62)69.4(62)71.0(62)79.0(62)45.2(62)66.1(62)69.4(62)56.5(62)58.1(62)
Better answer is B96.8(63)92.1(63)95.2(63)95.2(63)93.7(63)90.5(63)84.1(63)85.7(63)81.0(63)95.2(63)88.9(63)90.5(63)87.3(63)87.3(63)92.1(63)85.7(63)92.1(63)92.1(63)87.3(63)92.1(63)81.0(63)84.1(63)69.8(63)85.7(63)87.3(63)82.5(63)82.5(63)87.3(63)74.6(63)87.3(63)90.5(63)79.4(63)77.8(63)63.5(63)73.0(63)

M-RewardBench (日本語)

Japanese
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy85.6(125)88.0(125)86.4(125)87.2(125)85.6(125)84.8(125)83.2(125)82.4(125)81.6(125)88.0(125)84.0(125)85.6(125)84.8(125)80.0(125)83.2(125)81.6(125)78.4(125)76.8(125)80.0(125)84.0(125)77.6(125)76.0(125)73.6(125)83.2(125)81.6(125)72.8(125)69.6(125)75.2(125)75.2(125)73.6(125)67.2(125)76.0(125)75.2(125)66.4(125)68.8(125)
Option-order flips4.0% (125)0.0% (125)3.2% (125)2.4% (125)4.8% (125)3.2% (125)6.4% (125)8.0% (125)0.0% (121)0.8% (125)0.0% (125)0.0% (125)3.2% (125)0.8% (125)13.6% (125)12.0% (125)7.2% (125)20.8% (125)4.0% (125)3.2% (125)3.2% (125)0.0% (125)1.6% (125)0.8% (125)8.0% (125)0.0% (125)68.8% (125)1.6% (125)0.0% (125)4.8% (125)25.6% (125)10.5% (124)1.6% (125)3.2% (125)0.0% (125)
Confidence vs. accuracy gap5.1 pp5.1 pp5.4 pp7.9 pp4.9 pp4.4 pp5.2 pp8.3 pp4.8 pp5.8 pp10.8 pp11.7 pp8.9 pp9.7 pp3.1 pp7.1 pp15.9 pp17.4 pp7.6 pp6.4 pp7.5 pp17.9 pp9.2 pp7.0 pp9.4 pp19.3 pp4.9 pp15.0 pp14.1 pp10.8 pp16.4 pp11.1 pp5.9 pp14.4 pp11.2 pp
Median latency391ms541ms469ms339ms484ms526ms478ms477ms357ms481ms383ms212ms277ms624ms295ms228ms613ms421ms368ms432ms462ms575ms360ms284ms492ms363ms550ms495ms248ms689ms639ms407ms424ms628ms618ms
Category
Chat96.8(31)96.8(31)96.8(31)93.5(31)96.8(31)90.3(31)90.3(31)93.5(31)90.3(31)93.5(31)93.5(31)96.8(31)93.5(31)93.5(31)90.3(31)83.9(31)93.5(31)90.3(31)93.5(31)93.5(31)96.8(31)90.3(31)90.3(31)96.8(31)96.8(31)90.3(31)77.4(31)90.3(31)96.8(31)93.5(31)90.3(31)96.8(31)90.3(31)96.8(31)90.3(31)
Chat Hard62.5(32)75.0(32)62.5(32)75.0(32)62.5(32)62.5(32)62.5(32)56.3(32)62.5(32)68.8(32)59.4(32)59.4(32)65.6(32)56.3(32)65.6(32)56.3(32)53.1(32)56.3(32)50.0(32)62.5(32)50.0(32)50.0(32)50.0(32)62.5(32)53.1(32)46.9(32)34.4(32)40.6(32)53.1(32)46.9(32)50.0(32)46.9(32)46.9(32)34.4(32)50.0(32)
Safety87.1(31)80.6(31)87.1(31)87.1(31)83.9(31)90.3(31)87.1(31)87.1(31)74.2(31)90.3(31)87.1(31)90.3(31)93.5(31)80.6(31)87.1(31)93.5(31)80.6(31)74.2(31)83.9(31)87.1(31)77.4(31)80.6(31)74.2(31)80.6(31)87.1(31)71.0(31)90.3(31)83.9(31)74.2(31)74.2(31)67.7(31)80.6(31)83.9(31)67.7(31)77.4(31)
Reasoning96.8(31)100.0(31)100.0(31)93.5(31)100.0(31)96.8(31)93.5(31)93.5(31)100.0(31)100.0(31)96.8(31)96.8(31)87.1(31)90.3(31)90.3(31)93.5(31)87.1(31)87.1(31)93.5(31)93.5(31)87.1(31)83.9(31)80.6(31)93.5(31)90.3(31)83.9(31)77.4(31)87.1(31)77.4(31)80.6(31)61.3(31)80.6(31)80.6(31)67.7(31)58.1(31)
Position of the correct option
Better answer is A87.3(63)87.3(63)88.9(63)87.3(63)85.7(63)85.7(63)85.7(63)85.7(63)84.1(63)90.5(63)81.0(63)84.1(63)85.7(63)77.8(63)82.5(63)87.3(63)69.8(63)61.9(63)81.0(63)82.5(63)76.2(63)71.4(63)73.0(63)85.7(63)79.4(63)65.1(63)58.7(63)66.7(63)69.8(63)71.4(63)46.0(63)69.8(63)77.8(63)68.3(63)60.3(63)
Better answer is B83.9(62)88.7(62)83.9(62)87.1(62)85.5(62)83.9(62)80.6(62)79.0(62)79.0(62)85.5(62)87.1(62)87.1(62)83.9(62)82.3(62)83.9(62)75.8(62)87.1(62)91.9(62)79.0(62)85.5(62)79.0(62)80.6(62)74.2(62)80.6(62)83.9(62)80.6(62)80.6(62)83.9(62)80.6(62)75.8(62)88.7(62)82.3(62)72.6(62)64.5(62)77.4(62)
PolyGuard dataset evaluation details

PolyGuardPrompts (한국어)

Korean
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy88.8(125)88.8(125)91.2(125)88.8(125)90.4(125)88.8(125)92.8(125)90.4(125)84.0(125)90.4(125)88.8(125)89.6(125)90.4(125)88.0(125)87.2(125)82.4(125)82.4(125)84.8(125)87.2(125)91.2(125)84.0(125)88.0(125)88.8(125)85.6(125)84.0(125)84.0(125)76.0(125)80.0(125)64.8(125)70.4(125)84.0(125)39.2(125)84.0(125)73.6(125)70.4(125)
Confidence vs. accuracy gap7.6 pp8.3 pp7.0 pp9.3 pp6.3 pp6.6 pp6.7 pp7.1 pp6.0 pp2.7 pp4.6 pp3.7 pp7.8 pp4.9 pp6.9 pp14.0 pp11.7 pp9.3 pp8.4 pp2.0 pp9.6 pp6.9 pp3.1 pp9.7 pp5.3 pp8.3 pp3.3 pp12.6 pp20.5 pp5.3 pp9.6 pp32.6 pp6.0 pp10.8 pp5.7 pp
Distance from human ratings0.1600.1650.1470.1590.1350.1540.1480.1550.2070.1730.1710.1720.1400.1820.1820.2070.2190.2020.1670.1680.1940.1580.1900.1890.2160.2000.2710.2290.3640.3250.2010.4830.2220.2910.297
Median latency403ms530ms470ms338ms482ms804ms478ms466ms267ms445ms372ms210ms269ms610ms295ms247ms967ms318ms360ms431ms451ms573ms339ms281ms505ms355ms537ms470ms241ms686ms578ms396ms423ms616ms581ms
Gold label
Harmful request80.6(62)80.6(62)85.5(62)80.6(62)83.9(62)82.3(62)88.7(62)87.1(62)69.4(62)83.9(62)85.5(62)87.1(62)88.7(62)82.3(62)79.0(62)67.7(62)67.7(62)74.2(62)82.3(62)85.5(62)82.3(62)83.9(62)85.5(62)82.3(62)72.6(62)74.2(62)61.3(62)64.5(62)56.5(62)64.5(62)71.0(62)43.5(62)71.0(62)56.5(62)53.2(62)
Benign request96.8(63)96.8(63)96.8(63)96.8(63)96.8(63)95.2(63)96.8(63)93.7(63)98.4(63)96.8(63)92.1(63)92.1(63)92.1(63)93.7(63)95.2(63)96.8(63)96.8(63)95.2(63)92.1(63)96.8(63)85.7(63)92.1(63)92.1(63)88.9(63)95.2(63)93.7(63)90.5(63)95.2(63)73.0(63)76.2(63)96.8(63)34.9(63)96.8(63)90.5(63)87.3(63)
Jailbreak attempt
Adversarial88.9(63)87.3(63)92.1(63)90.5(63)90.5(63)87.3(63)92.1(63)92.1(63)77.8(63)90.5(63)85.7(63)87.3(63)90.5(63)85.7(63)84.1(63)81.0(63)79.4(63)81.0(63)85.7(63)92.1(63)79.4(63)85.7(63)85.7(63)85.7(63)84.1(63)79.4(63)69.8(63)74.6(63)73.0(63)74.6(63)82.5(63)39.7(63)84.1(63)66.7(63)65.1(63)
Plain88.7(62)90.3(62)90.3(62)87.1(62)90.3(62)90.3(62)93.5(62)88.7(62)90.3(62)90.3(62)91.9(62)91.9(62)90.3(62)90.3(62)90.3(62)83.9(62)85.5(62)88.7(62)88.7(62)90.3(62)88.7(62)90.3(62)91.9(62)85.5(62)83.9(62)88.7(62)82.3(62)85.5(62)56.5(62)66.1(62)85.5(62)38.7(62)83.9(62)80.6(62)75.8(62)
Rater agreement
3 of 3 agree93.4(91)92.3(91)95.6(91)93.4(91)95.6(91)93.4(91)97.8(91)95.6(91)86.8(91)95.6(91)93.4(91)94.5(91)95.6(91)90.1(91)90.1(91)85.7(91)84.6(91)87.9(91)91.2(91)96.7(91)91.2(91)92.3(91)93.4(91)91.2(91)87.9(91)86.8(91)81.3(91)85.7(91)67.0(91)70.3(91)89.0(91)37.4(91)87.9(91)74.7(91)73.6(91)
2 of 3 agree76.5(34)79.4(34)79.4(34)76.5(34)76.5(34)76.5(34)79.4(34)76.5(34)76.5(34)76.5(34)76.5(34)76.5(34)76.5(34)82.4(34)79.4(34)73.5(34)76.5(34)76.5(34)76.5(34)76.5(34)64.7(34)76.5(34)76.5(34)70.6(34)73.5(34)76.5(34)61.8(34)64.7(34)58.8(34)70.6(34)70.6(34)44.1(34)73.5(34)70.6(34)61.8(34)

PolyGuardPrompts (日本語)

Japanese
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy88.8(125)88.0(125)88.8(125)87.2(125)89.6(125)90.4(125)88.8(125)87.2(125)77.6(125)90.4(125)85.6(125)86.4(125)85.6(125)82.4(125)83.2(125)76.8(125)81.6(125)81.6(125)84.0(125)91.2(125)88.8(125)84.8(125)87.2(125)87.2(125)75.2(125)87.2(125)76.8(125)80.0(125)70.4(125)80.0(125)76.8(125)43.2(125)84.0(125)76.0(125)68.0(125)
Confidence vs. accuracy gap7.7 pp10.7 pp6.5 pp11.1 pp6.9 pp3.9 pp3.9 pp3.1 pp11.0 pp3.9 pp2.5 pp6.5 pp9.0 pp9.6 pp4.7 pp19.7 pp12.5 pp12.2 pp6.5 pp2.9 pp6.6 pp9.0 pp4.8 pp6.8 pp13.2 pp5.1 pp4.6 pp9.6 pp10.6 pp12.7 pp10.5 pp31.3 pp5.9 pp7.6 pp7.8 pp
Distance from human ratings0.1560.1660.1530.1750.1500.1600.1550.1630.2200.1650.1770.1750.1740.1850.1980.2480.2100.2340.1670.1710.1560.1820.1850.1590.2270.1800.2540.2100.3110.2710.2440.4460.2060.2680.289
Median latency399ms534ms474ms346ms484ms996ms479ms469ms261ms450ms371ms220ms248ms619ms292ms260ms802ms326ms377ms428ms447ms568ms344ms283ms541ms358ms561ms513ms248ms691ms613ms395ms426ms606ms569ms
Gold label
Harmful request81.0(63)79.4(63)82.5(63)76.2(63)82.5(63)85.7(63)82.5(63)81.0(63)57.1(63)84.1(63)77.8(63)81.0(63)81.0(63)69.8(63)73.0(63)57.1(63)63.5(63)65.1(63)77.8(63)82.5(63)82.5(63)79.4(63)79.4(63)77.8(63)54.0(63)79.4(63)60.3(63)61.9(63)52.4(63)68.3(63)55.6(63)36.5(63)69.8(63)54.0(63)61.9(63)
Benign request96.8(62)96.8(62)95.2(62)98.4(62)96.8(62)95.2(62)95.2(62)93.5(62)98.4(62)96.8(62)93.5(62)91.9(62)90.3(62)95.2(62)93.5(62)96.8(62)100.0(62)98.4(62)90.3(62)100.0(62)95.2(62)90.3(62)95.2(62)96.8(62)96.8(62)95.2(62)93.5(62)98.4(62)88.7(62)91.9(62)98.4(62)50.0(62)98.4(62)98.4(62)74.2(62)
Jailbreak attempt
Adversarial87.1(62)85.5(62)87.1(62)87.1(62)87.1(62)88.7(62)85.5(62)83.9(62)72.6(62)88.7(62)83.9(62)87.1(62)83.9(62)83.9(62)82.3(62)75.8(62)79.0(62)82.3(62)82.3(62)90.3(62)87.1(62)82.3(62)88.7(62)88.7(62)74.2(62)87.1(62)71.0(62)77.4(62)74.2(62)82.3(62)74.2(62)50.0(62)80.6(62)72.6(62)64.5(62)
Plain90.5(63)90.5(63)90.5(63)87.3(63)92.1(63)92.1(63)92.1(63)90.5(63)82.5(63)92.1(63)87.3(63)85.7(63)87.3(63)81.0(63)84.1(63)77.8(63)84.1(63)81.0(63)85.7(63)92.1(63)90.5(63)87.3(63)85.7(63)85.7(63)76.2(63)87.3(63)82.5(63)82.5(63)66.7(63)77.8(63)79.4(63)36.5(63)87.3(63)79.4(63)71.4(63)
Rater agreement
3 of 3 agree96.2(79)96.2(79)96.2(79)94.9(79)96.2(79)94.9(79)98.7(79)93.7(79)86.1(79)97.5(79)93.7(79)93.7(79)92.4(79)92.4(79)89.9(79)83.5(79)89.9(79)86.1(79)93.7(79)96.2(79)97.5(79)92.4(79)93.7(79)94.9(79)86.1(79)92.4(79)88.6(79)89.9(79)70.9(79)84.8(79)84.8(79)46.8(79)89.9(79)79.7(79)82.3(79)
2 of 3 agree76.1(46)73.9(46)76.1(46)73.9(46)78.3(46)82.6(46)71.7(46)76.1(46)63.0(46)78.3(46)71.7(46)73.9(46)73.9(46)65.2(46)71.7(46)65.2(46)67.4(46)73.9(46)67.4(46)82.6(46)73.9(46)71.7(46)76.1(46)73.9(46)56.5(46)78.3(46)56.5(46)63.0(46)69.6(46)71.7(46)63.0(46)37.0(46)73.9(46)69.6(46)43.5(46)
K-MHaS dataset evaluation details

K-MHaS

Korean
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy88.0(250)86.0(250)81.2(250)84.8(250)94.4(250)89.6(250)88.0(250)94.0(250)85.2(250)67.6(250)79.2(250)75.6(250)71.2(250)79.2(250)74.4(250)90.0(250)87.2(250)83.6(250)70.8(250)57.6(250)60.4(250)75.2(250)93.2(250)58.4(250)55.6(250)71.6(250)83.6(250)49.6(250)63.2(250)54.4(250)75.2(250)68.8(250)25.2(250)78.8(250)23.2(250)
Confidence vs. accuracy gap4.8 pp4.5 pp3.3 pp3.7 pp9.7 pp4.1 pp11.3 pp10.1 pp6.0 pp12.1 pp3.5 pp4.6 pp15.6 pp7.1 pp6.7 pp3.0 pp3.5 pp10.0 pp10.7 pp16.1 pp9.9 pp11.2 pp7.3 pp19.8 pp21.9 pp15.4 pp15.3 pp34.0 pp22.0 pp12.0 pp9.4 pp8.1 pp49.3 pp9.3 pp44.0 pp
Median latency376ms532ms460ms339ms466ms509ms463ms464ms261ms455ms385ms211ms274ms609ms294ms247ms659ms363ms379ms432ms451ms472ms338ms280ms504ms356ms548ms479ms248ms695ms548ms396ms422ms587ms581ms
Hate type
Origin82.1(28)85.7(28)78.6(28)82.1(28)96.4(28)92.9(28)89.3(28)100.0(28)75.0(28)53.6(28)71.4(28)64.3(28)60.7(28)71.4(28)67.9(28)96.4(28)92.9(28)89.3(28)64.3(28)50.0(28)53.6(28)71.4(28)100.0(28)50.0(28)46.4(28)78.6(28)75.0(28)39.3(28)57.1(28)42.9(28)82.1(28)67.9(28)10.7(28)78.6(28)25.0(28)
Physical96.4(28)92.9(28)89.3(28)89.3(28)96.4(28)89.3(28)92.9(28)100.0(28)100.0(28)82.1(28)89.3(28)82.1(28)82.1(28)75.0(28)78.6(28)92.9(28)92.9(28)89.3(28)78.6(28)67.9(28)75.0(28)89.3(28)92.9(28)64.3(28)75.0(28)89.3(28)92.9(28)64.3(28)82.1(28)60.7(28)75.0(28)82.1(28)17.9(28)92.9(28)10.7(28)
Politics85.2(27)81.5(27)74.1(27)81.5(27)92.6(27)77.8(27)92.6(27)96.3(27)92.6(27)51.9(27)81.5(27)77.8(27)66.7(27)88.9(27)70.4(27)88.9(27)92.6(27)85.2(27)51.9(27)55.6(27)48.1(27)59.3(27)96.3(27)59.3(27)40.7(27)51.9(27)85.2(27)33.3(27)66.7(27)37.0(27)66.7(27)70.4(27)7.4(27)88.9(27)11.1(27)
Profanity92.9(28)92.9(28)92.9(28)92.9(28)96.4(28)92.9(28)85.7(28)96.4(28)85.7(28)75.0(28)71.4(28)64.3(28)67.9(28)89.3(28)85.7(28)89.3(28)71.4(28)78.6(28)75.0(28)60.7(28)57.1(28)82.1(28)96.4(28)57.1(28)32.1(28)78.6(28)89.3(28)50.0(28)46.4(28)35.7(28)67.9(28)75.0(28)0.0(28)78.6(28)10.7(28)
Age85.7(28)75.0(28)67.9(28)78.6(28)92.9(28)92.9(28)92.9(28)96.4(28)92.9(28)46.4(28)71.4(28)67.9(28)53.6(28)75.0(28)60.7(28)92.9(28)85.7(28)85.7(28)64.3(28)35.7(28)32.1(28)53.6(28)96.4(28)28.6(28)50.0(28)50.0(28)85.7(28)32.1(28)53.6(28)53.6(28)71.4(28)60.7(28)14.3(28)75.0(28)3.6(28)
Gender85.7(28)71.4(28)67.9(28)75.0(28)100.0(28)89.3(28)82.1(28)96.4(28)75.0(28)46.4(28)75.0(28)75.0(28)64.3(28)53.6(28)53.6(28)82.1(28)82.1(28)60.7(28)46.4(28)28.6(28)53.6(28)78.6(28)89.3(28)50.0(28)35.7(28)60.7(28)64.3(28)28.6(28)32.1(28)35.7(28)60.7(28)60.7(28)25.0(28)82.1(28)10.7(28)
Race81.5(27)88.9(27)85.2(27)85.2(27)88.9(27)85.2(27)77.8(27)96.3(27)66.7(27)74.1(27)77.8(27)66.7(27)77.8(27)77.8(27)70.4(27)85.2(27)77.8(27)74.1(27)74.1(27)66.7(27)70.4(27)81.5(27)81.5(27)63.0(27)59.3(27)70.4(27)85.2(27)59.3(27)77.8(27)66.7(27)70.4(27)51.9(27)48.1(27)48.1(27)33.3(27)
Religion96.4(28)96.4(28)89.3(28)92.9(28)100.0(28)96.4(28)96.4(28)100.0(28)92.9(28)85.7(28)85.7(28)89.3(28)75.0(28)85.7(28)85.7(28)96.4(28)92.9(28)96.4(28)89.3(28)60.7(28)60.7(28)78.6(28)100.0(28)57.1(28)64.3(28)78.6(28)85.7(28)46.4(28)60.7(28)60.7(28)96.4(28)75.0(28)7.1(28)78.6(28)3.6(28)
Not hate speech85.7(28)89.3(28)85.7(28)85.7(28)85.7(28)89.3(28)82.1(28)64.3(28)85.7(28)92.9(28)89.3(28)92.9(28)92.9(28)96.4(28)96.4(28)85.7(28)96.4(28)92.9(28)92.9(28)92.9(28)92.9(28)82.1(28)85.7(28)96.4(28)96.4(28)85.7(28)89.3(28)92.9(28)92.9(28)96.4(28)85.7(28)75.0(28)96.4(28)85.7(28)100.0(28)
RTP-LX dataset evaluation details

RTP-LX (한국어)

Korean
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy60.8(125)61.6(125)56.0(125)60.0(125)38.4(125)46.4(125)37.6(125)35.2(125)57.6(125)49.6(125)54.4(125)50.4(125)46.4(125)60.8(125)48.8(125)67.2(125)47.2(125)51.2(125)32.8(125)38.4(125)58.4(125)35.2(125)51.2(125)56.8(125)49.6(125)33.6(125)45.6(125)57.6(125)37.6(125)50.4(125)40.0(125)46.4(125)37.6(125)42.4(125)45.6(125)
Option-order flips8.0% (125)12.0% (125)13.6% (125)20.8% (125)12.0% (125)12.8% (125)13.6% (125)8.8% (125)13.6% (118)12.0% (125)12.8% (125)11.2% (125)18.4% (125)12.8% (125)29.6% (125)19.2% (125)28.8% (125)24.8% (125)23.2% (125)13.6% (125)23.2% (125)12.8% (125)3.2% (125)11.2% (125)44.0% (125)27.2% (125)22.4% (125)18.4% (125)11.2% (125)23.2% (125)36.8% (125)17.7% (124)16.8% (125)16.0% (125)85.6% (125)
Confidence vs. accuracy gap8.2 pp11.0 pp11.1 pp9.8 pp18.1 pp16.1 pp21.3 pp24.1 pp9.1 pp14.9 pp11.4 pp20.2 pp25.5 pp10.7 pp19.2 pp5.8 pp22.6 pp39.9 pp26.8 pp23.0 pp14.9 pp27.8 pp12.2 pp8.1 pp15.9 pp29.6 pp15.1 pp21.7 pp28.8 pp4.5 pp31.1 pp7.4 pp16.4 pp7.5 pp9.9 pp
Level error0.400.430.480.450.690.600.660.750.400.550.500.530.590.470.530.400.530.580.730.570.570.710.620.500.410.860.440.480.730.550.620.920.640.710.66
Distance from human ratings0.3650.3820.3910.3760.5050.4650.4890.5290.3810.4490.4200.4520.4530.3680.4460.3650.4440.5040.5150.4780.4650.5080.4590.3890.4670.5580.4650.4130.5120.4530.5340.5710.5390.5360.543
Median latency386ms521ms467ms342ms471ms507ms469ms466ms340ms453ms371ms205ms244ms596ms292ms233ms741ms4.2s355ms432ms451ms437ms343ms285ms470ms337ms573ms469ms242ms680ms537ms390ms431ms608ms561ms
Human-rated level
Level 196.8(31)100.0(31)100.0(31)100.0(31)96.8(31)93.5(31)93.5(31)93.5(31)96.8(31)100.0(31)96.8(31)96.8(31)96.8(31)93.5(31)100.0(31)93.5(31)96.8(31)96.8(31)100.0(31)100.0(31)96.8(31)96.8(31)100.0(31)100.0(31)100.0(31)96.8(31)96.8(31)100.0(31)100.0(31)100.0(31)96.8(31)83.9(31)100.0(31)90.3(31)100.0(31)
Level 229.0(31)45.2(31)41.9(31)35.5(31)16.1(31)22.6(31)29.0(31)29.0(31)71.0(31)35.5(31)38.7(31)41.9(31)29.0(31)48.4(31)29.0(31)83.9(31)48.4(31)58.1(31)12.9(31)41.9(31)32.3(31)12.9(31)12.9(31)32.3(31)9.7(31)12.9(31)61.3(31)6.5(31)9.7(31)3.2(31)58.1(31)0.0(31)0.0(31)22.6(31)6.5(31)
Level 358.1(31)54.8(31)38.7(31)45.2(31)16.1(31)29.0(31)19.4(31)12.9(31)25.8(31)29.0(31)45.2(31)38.7(31)32.3(31)51.6(31)29.0(31)38.7(31)25.8(31)32.3(31)9.7(31)9.7(31)54.8(31)22.6(31)22.6(31)54.8(31)16.1(31)0.0(31)9.7(31)71.0(31)32.3(31)54.8(31)0.0(31)16.1(31)12.9(31)54.8(31)22.6(31)
Level 459.4(32)46.9(32)43.8(32)59.4(32)25.0(32)40.6(32)9.4(32)6.3(32)37.5(32)34.4(32)37.5(32)25.0(32)28.1(32)50.0(32)37.5(32)53.1(32)18.8(32)18.8(32)9.4(32)3.1(32)50.0(32)9.4(32)68.8(32)40.6(32)71.9(32)25.0(32)15.6(32)53.1(32)9.4(32)43.8(32)6.3(32)84.4(32)37.5(32)3.1(32)53.1(32)

RTP-LX (日本語)

Japanese
Decider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Overall · Accuracy48.8(125)48.8(125)52.0(125)51.2(125)51.2(125)48.8(125)58.4(125)51.2(125)48.0(125)56.0(125)58.4(125)60.8(125)51.2(125)51.2(125)50.4(125)46.4(125)54.4(125)51.2(125)48.8(125)54.4(125)48.0(125)44.8(125)36.0(125)39.2(125)44.0(125)44.0(125)44.8(125)38.4(125)50.4(125)48.0(125)53.6(125)36.0(125)48.8(125)36.0(125)40.8(125)
Option-order flips9.6% (125)19.2% (125)15.2% (125)16.0% (125)16.8% (125)19.2% (125)20.0% (125)20.0% (125)16.8% (113)18.4% (125)10.4% (125)11.2% (125)16.8% (125)17.6% (125)22.4% (125)25.6% (125)40.8% (125)35.2% (125)28.0% (125)14.4% (125)18.4% (125)12.0% (125)11.2% (125)7.2% (125)52.0% (125)27.2% (125)41.6% (125)21.6% (125)10.4% (125)30.4% (125)40.8% (125)42.4% (125)26.4% (125)14.4% (125)79.2% (125)
Confidence vs. accuracy gap8.5 pp6.9 pp4.0 pp8.7 pp12.9 pp7.6 pp5.8 pp11.2 pp8.1 pp9.1 pp9.7 pp9.6 pp23.1 pp14.7 pp8.1 pp24.2 pp12.0 pp36.8 pp12.3 pp8.1 pp6.3 pp16.4 pp13.4 pp17.6 pp12.3 pp18.4 pp13.9 pp21.3 pp13.2 pp16.2 pp15.9 pp8.0 pp17.6 pp6.3 pp8.8 pp
Level error0.600.670.560.560.630.600.560.630.610.550.490.500.600.600.610.670.530.670.730.550.790.650.740.760.670.780.730.850.670.720.730.890.790.900.90
Distance from human ratings0.4140.4280.3920.3970.4300.4090.4030.4340.4260.3770.3630.3730.4460.4060.4240.4250.3830.4570.4470.4150.4860.4260.4920.4590.4610.4690.4700.4890.4330.4670.4720.5400.5180.5490.556
Median latency392ms527ms520ms341ms475ms516ms474ms460ms356ms719ms370ms207ms256ms591ms294ms257ms637ms8.7s364ms431ms516ms526ms343ms281ms477ms347ms553ms487ms251ms677ms558ms395ms422ms599ms566ms
Human-rated level
Level 184.0(25)88.0(25)96.0(25)92.0(25)88.0(25)88.0(25)88.0(25)88.0(25)100.0(25)96.0(25)88.0(25)92.0(25)96.0(25)84.0(25)96.0(25)88.0(25)96.0(25)92.0(25)92.0(25)100.0(25)96.0(25)84.0(25)96.0(25)96.0(25)100.0(25)92.0(25)96.0(25)100.0(25)92.0(25)96.0(25)92.0(25)84.0(25)100.0(25)60.0(25)100.0(25)
Level 232.0(25)60.0(25)36.0(25)44.0(25)24.0(25)20.0(25)40.0(25)20.0(25)64.0(25)32.0(25)40.0(25)28.0(25)36.0(25)44.0(25)36.0(25)76.0(25)28.0(25)48.0(25)12.0(25)44.0(25)24.0(25)8.0(25)0.0(25)20.0(25)16.0(25)0.0(25)52.0(25)8.0(25)8.0(25)4.0(25)48.0(25)0.0(25)4.0(25)0.0(25)16.0(25)
Level 356.0(25)44.0(25)52.0(25)48.0(25)20.0(25)44.0(25)40.0(25)28.0(25)36.0(25)32.0(25)56.0(25)44.0(25)40.0(25)52.0(25)16.0(25)24.0(25)32.0(25)28.0(25)12.0(25)20.0(25)40.0(25)32.0(25)8.0(25)48.0(25)28.0(25)4.0(25)8.0(25)60.0(25)36.0(25)56.0(25)0.0(25)20.0(25)36.0(25)56.0(25)24.0(25)
Level 456.0(25)44.0(25)48.0(25)60.0(25)44.0(25)60.0(25)36.0(25)32.0(25)20.0(25)28.0(25)44.0(25)52.0(25)36.0(25)36.0(25)40.0(25)40.0(25)44.0(25)28.0(25)36.0(25)16.0(25)40.0(25)4.0(25)76.0(25)28.0(25)64.0(25)32.0(25)36.0(25)12.0(25)16.0(25)12.0(25)48.0(25)16.0(25)32.0(25)44.0(25)44.0(25)
Level 516.0(25)8.0(25)28.0(25)12.0(25)80.0(25)32.0(25)88.0(25)88.0(25)20.0(25)92.0(25)64.0(25)88.0(25)48.0(25)40.0(25)64.0(25)4.0(25)72.0(25)60.0(25)92.0(25)92.0(25)40.0(25)96.0(25)0.0(25)4.0(25)12.0(25)92.0(25)32.0(25)12.0(25)100.0(25)72.0(25)80.0(25)60.0(25)72.0(25)20.0(25)20.0(25)

Method

Items
2,000
Datasets
8
Models measured
35
Responses received
116,807

Measurement log

ModelRequests sentResponses receivedActual costPeriod
GPT-6.1 SolRef.2,0002,000$1.652Oct 4
Claude Sonnet 5.5Ref.2,0002,000$3.387Oct 4
Decider V1.33,3403,340$0.014Nov 17
Decider V1.23,3403,340$0.015Oct 24
Drex Max3,3403,340$0.046Oct 24
Perplexity Decider v1.1 27B3,3403,340$0.015Oct 9
d1.2-ultra3,3403,340$0.026Nov 17
D1 Ultra3,3403,340$0.026Oct 16
d1.23,3403,340$0.013Nov 17
d1.13,3403,340$0.013Nov 3
Gemini Decision 1Refused 69 · scored wrong3,3403,267$0.078Nov 12
Mammoth 13,3403,340$0.082Nov 3
Jev 1.143,3403,340$0.073Oct 15
Jev 1.133,3403,340$0.073Oct 3 ~ Oct 4
Mercury Decide3,3403,340$0Oct 3 ~ Oct 5
Microsoft-Decision-13,3403,340$0.023Oct 10
Perplexity Decider v1 27B3,3403,340$0.031Oct 3 ~ Oct 4
GPT-6 Luna DecisionsRefused 4 · scored wrong3,3403,336$0.104Oct 8
Span-023,3403,340$0.0081Oct 24
Solar Decide3,3403,340$0.080Oct 3 ~ Oct 4
D13,3403,340$0.025Oct 3 ~ Oct 4
Elephant 13,3403,340$0.029Nov 3
Decider Flash V1.23,3403,340$0.0076Oct 24
Clef3,3403,340$0.229Oct 3 ~ Oct 4
Kanana 2 V Decide3,3403,340$0Nov 12
Decider Flash V1.33,3403,340$0.0071Nov 17
Drex v2.03,3403,340$0.031Oct 15
Clef Omni3,3403,340$0.159Oct 15
Classify-13,3403,340$0.054Oct 15
Drex v1.53,3403,340$0.021Oct 15
Clef Flash3,3403,340$0.086Oct 3 ~ Oct 4
Kev 4B3,3403,340$0.022Oct 15
Solar Decide Flash3,3403,340$0.080Oct 15
Decision Model PreviewRefused 16 · scored wrong3,3403,324$0Oct 5
Joey 13,3403,340$0.0082Nov 3
Decider 1.13,3403,340$0.026Nov 20
Decider 13,3403,340$0.028Oct 5

How we measure

  1. 1

    A real item from public data

    Drawn from openly licensed Korean and Japanese datasets with a fixed seed, so correct answers are evenly mixed.

  2. 2

    One question

    Asked as yes/no, a choice of options, or a scale score. Every model gets the same items.

  3. 3

    The answer comes from the source

    The correct answer is the label from the dataset's creators. No AI model does the grading.

  4. 4

    Probabilities only

    Models return a probability for each option, with no text. The highest one is graded as the answer, and the probabilities feed the calibration measures.

The details

How to read the numbersWhat accuracy, confidence-vs-accuracy gap, option-order flips, latency and cost mean
Accuracy
The average of the 8 datasets' accuracies, each weighted equally. Only answers to questions asked in the original option order count.
Korean & Japanese
Scores computed from Korean items only and Japanese items only. Accuracy is taken per dataset that has items in that language (7 for Korean, 5 for Japanese) and averaged with equal weight. Because each language covers a different set of datasets, accuracy is not a simple average of the two.
Confidence vs. accuracy gap
When a model says it is 90% sure, is it right 90% of the time? For each confidence band we measure how many points the stated probability is off from actual accuracy, and average them. Closer to 0 means you can trust its probabilities more.
Option-order flips
How often the answer changed when the same question was asked again with the options reversed. Lower means less swayed by option position.
Latency
Time until the answer to one item arrives. Models called through OpenRouter include the routing time.
Cost · 1,000
Based on the amount actually billed. Free tiers without an official price are shown as “free for now”, and open-weight models we ran ourselves (no API on sale) as “no cost”; both are left out of cost comparisons.

On scale items (sentence similarity 0–5, toxicity 1–5) only the exact level counts as correct, so these datasets score lower.

Every metric by modelSamples, probability quality, accuracy by form and language, latency, cost

Sample sizes, probability quality, results by question form and language, latency distribution and cost. Models still in progress are computed from the answers received so far.

MetricDecider V1.3Decider V1.2Drex MaxPerplexity Decider v1.1 27Bd1.2-ultraD1 Ultrad1.2d1.1Gemini Decision 1Mammoth 1Jev 1.14Jev 1.13Mercury DecideMicrosoft-Decision-1Perplexity Decider v1 27BGPT-6 Luna DecisionsSpan-02Solar DecideD1Elephant 1Decider Flash V1.2ClefKanana 2 V DecideDecider Flash V1.3Drex v2.0Clef OmniClassify-1Drex v1.5Clef FlashKev 4BSolar Decide FlashDecision Model PreviewJoey 1Decider 1.1Decider 1
Samples
Requests to send3,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,340
Answers received3,3403,3403,3403,3403,3403,3403,3403,3403,2673,3403,3403,3403,3403,3403,3403,3363,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3403,3243,3403,3403,340
Refused (scored wrong)The provider's content filter refused these items, so there is no answer. Requesting again gives the same result, so they are scored as wrong.0000000069000000400000000000000016000
Items graded2,0002,0002,0002,0002,0002,0002,0002,0001,9982,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,0002,000
Order-comparison pairs1,3401,3401,3401,3401,3401,3401,3401,3401,3071,3401,3401,3401,3401,3401,3401,3381,3401,3401,3401,3401,3401,3401,3401,3401,3401,3401,3401,3401,3401,3401,3401,3331,3401,3401,340
Probability quality
Accuracy82.482.382.281.781.681.580.679.979.878.978.778.677.777.577.277.276.975.375.174.874.273.973.973.872.771.570.569.669.468.067.062.959.759.254.7
Average confidence84.4%83.6%83.9%84.7%80.9%81.6%78.5%80.5%80.4%81.0%81.7%82.3%88.4%85.1%77.8%86.0%86.2%93.4%80.8%79.4%77.1%81.8%75.8%82.1%75.5%80.5%68.4%82.5%79.2%70.9%79.9%74.3%68.8%64.8%63.6%
Confidence gapAverage confidence − accuracy. Positive means it answers more confidently than it is right+2.0 pp+1.3 pp+1.7 pp+3.0 pp−0.7 pp+0.1 pp−2.1 pp+0.6 pp−1.1 pp+2.1 pp+3.0 pp+3.7 pp+10.7 pp+7.6 pp+0.6 pp+8.7 pp+9.3 pp+18.1 pp+5.8 pp+4.6 pp+2.9 pp+7.9 pp+1.9 pp+8.3 pp+2.8 pp+9.0 pp−2.1 pp+12.9 pp+9.8 pp+2.9 pp+12.9 pp+11.0 pp+9.0 pp+5.5 pp+8.9 pp
Average probability on the correct answerHow highly it rated the correct answer, wrong items included77.7%76.6%77.1%77.1%75.0%75.7%72.7%73.4%74.6%73.4%73.1%73.0%75.8%73.8%70.5%73.5%73.8%73.2%71.1%70.0%68.0%70.7%66.8%70.8%65.6%67.3%60.4%66.9%64.4%61.2%63.4%59.1%54.5%51.6%49.6%
Confidence vs. accuracy gap2.1 pp2.3 pp2.4 pp3.4 pp3.2 pp2.1 pp3.0 pp1.9 pp1.4 pp2.3 pp3.4 pp3.7 pp10.7 pp8.6 pp2.3 pp8.7 pp9.3 pp18.1 pp5.9 pp5.3 pp4.7 pp8.0 pp3.8 pp8.4 pp4.5 pp9.0 pp3.7 pp13.1 pp9.9 pp3.7 pp12.9 pp11.0 pp10.2 pp5.7 pp9.7 pp
Brier scoreHow far the whole probability table is from the answer. Lower is better0.2350.2490.2420.2510.2410.2370.2580.2680.2490.2770.2890.2990.3200.3190.2950.3350.3280.4300.3300.3240.3420.3350.3360.3440.3670.3840.3810.4290.4220.4060.4540.4870.5070.5170.547
Level errorOn scale items, the average gap between the model's expected level and the correct level0.510.540.500.520.610.600.600.670.570.630.580.610.590.560.600.610.660.610.690.650.650.710.640.620.590.810.620.710.750.680.790.900.760.840.81
Distance from human ratingsCompared with the rater distributions in RTP-LX and PolyGuard. 0 = same, 1 = opposite0.2740.2850.2710.2770.3050.2970.2990.3200.3050.2910.2830.2930.3040.2860.3120.3110.3140.3490.3240.3080.3250.3190.3310.2990.3430.3520.3650.3350.4050.3790.3630.5100.3710.4110.421
Option-order flips4.2%6.3%7.2%6.9%6.1%7.2%9.6%11.0%5.6%6.4%6.0%6.5%12.5%5.6%14.9%14.3%17.2%19.5%13.7%9.5%8.7%5.4%7.5%7.0%18.5%7.1%32.5%14.5%4.0%19.3%25.9%18.8%13.3%12.4%40.2%
Chosen answer ≠ top probabilityOn multiple-choice items, how often the model's chosen option was not its highest-probability option. Grading uses the top probability0 / 9150 / 9150 / 9150 / 9150 / 9150 / 9150 / 9150 / 9150 / 9010 / 9152 / 9151 / 9150 / 9150 / 9150 / 9151 / 9130 / 9151 / 9150 / 9150 / 9150 / 9150 / 9150 / 9150 / 9150 / 9150 / 9150 / 9150 / 9150 / 9150 / 91518 / 9152 / 9090 / 9150 / 9158 / 915
Accuracy by question form
Yes/no91.1 (660)90.0 (660)88.9 (660)89.4 (660)93.8 (660)91.8 (660)91.7 (660)93.3 (660)86.8 (660)83.9 (660)86.8 (660)85.9 (660)84.2 (660)86.4 (660)84.4 (660)88.0 (660)87.9 (660)86.4 (660)82.7 (660)80.3 (660)79.1 (660)84.8 (660)91.4 (660)78.3 (660)74.2 (660)83.0 (660)82.6 (660)72.0 (660)72.7 (660)72.0 (660)80.9 (660)63.6 (660)63.0 (660)77.7 (660)56.8 (660)
Multiple choice89.3 (915)88.6 (915)89.1 (915)88.0 (915)88.2 (915)89.1 (915)86.9 (915)86.1 (915)88.0 (913)89.2 (915)85.8 (915)86.3 (915)85.6 (915)82.2 (915)85.5 (915)81.7 (915)83.0 (915)78.9 (915)84.3 (915)84.8 (915)80.4 (915)81.6 (915)74.1 (915)82.0 (915)83.9 (915)77.8 (915)73.6 (915)79.6 (915)79.9 (915)76.1 (915)68.6 (915)73.0 (915)65.9 (915)55.5 (915)58.9 (915)
Scale score53.9 (425)56.5 (425)56.7 (425)56.0 (425)48.2 (425)48.9 (425)49.6 (425)45.6 (425)51.3 (425)48.9 (425)50.8 (425)50.8 (425)50.6 (425)53.6 (425)48.2 (425)50.6 (425)46.6 (425)50.4 (425)43.5 (425)44.5 (425)52.9 (425)40.0 (425)46.1 (425)49.2 (425)45.9 (425)39.8 (425)45.2 (425)44.2 (425)41.4 (425)44.2 (425)41.9 (425)40.0 (425)40.9 (425)38.1 (425)42.1 (425)
By language
Korean accuracy84.6 (1,250)84.3 (1,250)83.7 (1,250)83.4 (1,250)84.0 (1,250)83.0 (1,250)81.5 (1,250)81.6 (1,250)82.2 (1,250)78.1 (1,250)79.4 (1,250)78.4 (1,250)78.5 (1,250)78.9 (1,250)78.2 (1,250)81.0 (1,250)78.2 (1,250)77.6 (1,250)74.6 (1,250)73.0 (1,250)73.6 (1,250)73.9 (1,250)77.4 (1,250)74.2 (1,250)72.5 (1,250)70.9 (1,250)72.4 (1,250)69.0 (1,250)68.2 (1,250)67.8 (1,250)68.1 (1,250)64.1 (1,250)56.3 (1,250)60.6 (1,250)52.0 (1,250)
Korean confidence vs. accuracy gap1.4 pp2.7 pp1.6 pp3.0 pp4.3 pp2.8 pp4.0 pp3.4 pp2.4 pp3.0 pp3.0 pp3.1 pp10.0 pp6.9 pp4.3 pp4.8 pp8.5 pp16.0 pp6.6 pp7.1 pp5.3 pp8.5 pp4.9 pp8.3 pp4.2 pp11.1 pp5.5 pp14.7 pp12.0 pp3.8 pp12.9 pp10.4 pp13.1 pp4.8 pp13.2 pp
Korean option-order flips3.6%6.2%7.7%7.7%5.6%8.3%10.2%10.8%6.4%5.9%7.4%8.3%14.0%5.7%15.4%15.7%16.6%20.0%14.7%10.9%9.8%5.5%6.9%8.1%19.6%7.0%31.8%17.2%4.8%20.8%25.9%17.6%12.6%12.6%42.5%
Japanese accuracy78.7 (750)78.8 (750)79.6 (750)78.7 (750)77.5 (750)78.9 (750)78.9 (750)77.1 (750)75.7 (748)80.3 (750)77.6 (750)79.1 (750)76.4 (750)75.2 (750)75.6 (750)70.8 (750)74.5 (750)71.5 (750)76.0 (750)77.6 (750)75.1 (750)73.7 (750)67.9 (750)73.2 (750)72.9 (750)72.4 (750)67.3 (750)70.4 (750)71.3 (750)68.1 (750)65.2 (750)60.9 (750)65.2 (750)56.8 (750)59.1 (750)
Japanese confidence vs. accuracy gap4.4 pp4.3 pp4.3 pp6.0 pp4.3 pp2.0 pp2.1 pp3.1 pp3.6 pp1.9 pp6.1 pp5.6 pp11.8 pp11.4 pp1.9 pp15.4 pp11.0 pp21.6 pp6.0 pp3.4 pp4.4 pp7.2 pp4.4 pp8.6 pp7.0 pp6.4 pp3.6 pp10.6 pp6.4 pp6.9 pp13.0 pp12.6 pp6.2 pp7.8 pp4.0 pp
Japanese option-order flips4.8%6.4%6.6%5.9%6.7%6.1%8.8%11.2%4.6%7.0%4.3%4.5%10.7%5.4%14.4%12.7%17.8%18.9%12.5%7.8%7.5%5.3%8.2%5.8%17.3%7.2%33.3%11.5%3.2%17.4%25.9%20.2%14.1%12.2%37.6%
Latency
Median383ms530ms470ms340ms472ms514ms469ms466ms303ms455ms376ms211ms276ms613ms295ms247ms669ms432ms362ms433ms454ms507ms343ms284ms490ms351ms554ms483ms253ms691ms568ms397ms425ms607ms596ms
Mean483ms647ms546ms412ms485ms581ms480ms501ms325ms652ms389ms225ms292ms626ms314ms319ms950ms4.3s400ms446ms479ms577ms376ms305ms563ms371ms611ms566ms276ms737ms567ms401ms438ms629ms637ms
Within 95%816ms1.0s879ms433ms527ms1.0s522ms632ms476ms1.3s462ms306ms399ms688ms408ms623ms2.5s13.6s611ms508ms611ms1.1s558ms375ms930ms476ms944ms939ms550ms921ms800ms439ms499ms791ms834ms
Within 99%1.5s3.1s1.2s900ms746ms1.6s642ms1.1s593ms2.9s593ms467ms559ms748ms729ms1.3s2.7s16.2s993ms756ms673ms1.5s616ms534ms1.3s723ms1.4s1.3s848ms1.5s1.0s513ms756ms897ms1.5s
Longest response6.5s9.1s2.0s19.7s2.1s3.3s1.2s4.9s6.4s5.3s1.4s586ms1.5s1.9s1.1s3.8s6.1s59.5s3.6s1.5s1.5s2.8s1.4s4.8s6.5s1.6s5.9s10.4s1.3s5.8s5.8s949ms1.6s2.2s2.4s
Cost
Per 1,000$0.0043$0.0045$0.014$0.0046$0.0079$0.0079$0.0038$0.0038$0.023$0.024$0.022$0.022Free for now$0.0070$0.0092$0.031$0.0024$0.024$0.0075$0.0086$0.0023$0.069No cost$0.0021$0.0094$0.048$0.016$0.0063$0.026$0.0066$0.024Free for now$0.0024$0.0077$0.0085
Total measurement cost$0.014$0.015$0.046$0.015$0.026$0.026$0.013$0.013$0.078$0.082$0.073$0.073$0$0.023$0.031$0.104$0.0081$0.080$0.025$0.029$0.0076$0.229$0$0.0071$0.031$0.159$0.054$0.021$0.086$0.022$0.080$0$0.0082$0.026$0.028
Input tokens714,686758,106768,130768,126627,785627,785627,785627,785780,833815,9161,727,2661,727,266568,400556,102768,1261,036,252813,4721,601,606627,785815,916758,106956,111722,955714,686522,7531,057,7521,351,012522,753956,111522,7531,601,606627,158815,916647,871706,963
Datasets and call routesLanguage, item count and license per set; call route and served version per model

Each version fixes its items, and scores are only comparable within the same version. Adding a model keeps the version; adding a dataset starts the next version.

SetLanguageItemsLicense
KoBEST BoolQKorean100CC BY-SA 4.0
KoBEST COPAKorean90CC BY-SA 4.0
KoBEST SentiNegKorean60CC BY-SA 4.0
KLUE NLIKorean150CC BY-SA 4.0
KLUE STSKorean100CC BY-SA 4.0
JGLUE JCommonsenseQAJapanese75CC BY-SA 4.0
JGLUE JNLIJapanese100CC BY-SA 4.0
JGLUE JSTSJapanese75CC BY-SA 4.0
KMMLUKorean125CC BY-ND 4.0
JMMLUJapanese125CC BY-SA 4.0 / CC BY-NC-ND 4.0
M-RewardBench (한국어)Korean125ODC-By 1.0
M-RewardBench (日本語)Japanese125ODC-By 1.0
PolyGuardPrompts (한국어)Korean125CC BY 4.0
PolyGuardPrompts (日本語)Japanese125CC BY 4.0
K-MHaSKorean250CC BY-SA 4.0
RTP-LX (한국어)Korean125ODC-By 1.0
RTP-LX (日本語)Japanese125ODC-By 1.0

KMMLU: Son et al., "KMMLU: Measuring Massive Multitask Language Understanding in Korean" (2024, arXiv:2402.11548), HAERAE. JMMLU: Waseda University NLP Lab — Japanese and world history questions © Step, civics, geography and idiom questions © VIST, MMLU translation supported by RIKEN.

ModelCalled viaVersion that answered
PerplexityDecider V1.3Perplexitypplx-decider-v1.3
PerplexityDecider V1.2OpenRouterperplexity/pplx-decider-v1.2-20261016
Nace AIDrex MaxOpenRouternace-ai/drex-max-20261016
PerplexityPerplexity Decider v1.1 27BPerplexitypplx-decider-v1.1-27b
Liquid AId1.2-ultraOpenRouterliquid/d1.2-ultra-20261116
Liquid AID1 UltraOpenRouterliquid/d1-ultra-20261015
Liquid AId1.2OpenRouterliquid/d1.2-20261116
Liquid AId1.1OpenRouterliquid/d1.1-20261027
GoogleGemini Decision 1Gemini API—
BasetenMammoth 1OpenRouterbaseten/mammoth-1-20261021
TypeSafeJev 1.14OpenRoutertypesafe/jev-1.14-20261014
TypeSafeJev 1.13OpenRoutertypesafe/jev-1.13-20260917
InceptionMercury DecideOpenRouterinception/mercury-decide-20260930
MicrosoftMicrosoft-Decision-1OpenRoutermicrosoft/microsoft-decision-1-20261009
PerplexityPerplexity Decider v1 27BPerplexitypplx-decider-v1-27b
OpenAIGPT-6 Luna DecisionsOpenAIgpt-6-luna
RespanSpan-02OpenRouterrespan/span-02-20261015
UpstageSolar DecideOpenRouterupstage/solar-decide-20260928
Liquid AID1OpenRouterliquid/d1-20260930
BasetenElephant 1OpenRouterbaseten/elephant-1-20261021
PerplexityDecider Flash V1.2OpenRouterperplexity/pplx-decider-flash-v1.2-20261016
CloudflareClefCloudflare Workers AIclef
KakaoKanana 2 V DecideSelf-hosted GPUkakaocorp/kanana-2-v-decide-3b-2611
PerplexityDecider Flash V1.3Perplexitypplx-decider-flash-v1.3
Nace AIDrex v2.0OpenRouternace-ai/drex-v2.0-20261009
CloudflareClef OmniOpenRoutercloudflare/clef-omni-20261009
ScaleDownClassify-1OpenRouterscaledown/classify-1-20260928
Nace AIDrex v1.5OpenRouternace-ai/drex-v1.5-20261009
CloudflareClef FlashCloudflare Workers AIclef-flash
Jared PalmerKev 4BOpenRouterjaredpalmer/kev-4b-20260924
UpstageSolar Decide FlashOpenRouterupstage/solar-decide-flash-20261008
AlibabaDecision Model PreviewAlibaba Cloud Model Studiodecision-model-preview
BasetenJoey 1OpenRouterbaseten/joey-1-20261029
meraGPTDecider 1.1meraGPTdecider-1
meraGPTDecider 1meraGPTstate-decider-1
ReproducibilitySeed, item fingerprints (SHA-256) and source revisions

Items were stratified-sampled from each dataset with seed 20261003. Each set's item fingerprint (SHA-256) lets you check that the items are the same; the source revision is listed only for sets where it was recorded. Contact us if you need the list of source rows for each set. Contact

SetItem fingerprintSource revision
kobest-boolq323717fbfaaa2fc575f3adb5e8fb0983f2b80ee9b45402b70fdf2afefcbbeaa2—
kobest-copaafa227de3bce46f5013ffa5c4887d74b929c68a97c2d7da732f3819c6a69b75c—
kobest-sentineg4af55bf4bbca356dd5f996b367d133a82173aad2fa87ab77cebb4a586df20ebb—
klue-nli78a9c69705d487232845ec18f4c050f7833035b16fd07b7e755aba3ce19ad5cd—
klue-stsc979a7d5258d8d25bf7605bf1f895366e2a1934f7a5b1fb6350921fadd8e0a26—
jglue-jcommonsenseqacb87b85c4b6c94be5e0ae5081bd8ae99603769c7f8ea7a367697a37a5f489ab4—
jglue-jnlie812223e4ac8b7e3de15a2298da954a1e78915f9e46f6b7b2e54d335d4c2eef6—
jglue-jsts8161627fe3f72e5cd5b60a41c24a3dd5e2e7c76abbaa66d509fff54bf2f4d5a9—
kmmlu575b8ade0493fa4fefdf3795db56d8dbf58e746336e1726f196a4a9e491d6c2ed61b3f19e552c576bf5960dd24289763edc36a88
jmmlu0f4f97a840d486d563e55872720874da8bebe9b8144bd1139e014cbb53cbd46e762cbf192c9e4588b574d718f95afd64c96fcbf4
mrewardbench-koa3582c37dfeaa40b5441d7211ca99f24078ed3919d4771c9994d1da8d8f91a42—
mrewardbench-jaaad4a04ac56d89705d4df59e7b7a6d891beed79b9c40aba08cbe478b5a22a6e7—
polyguard-ko5817f4bf61aa1fd9a377af67f4d6a5f68dc5eaff0c407c8c49931c8c0e979c7e—
polyguard-ja47af613e0d45cee908937aecb151279602568a9b5e635b8bbd5dae871ead04f4—
kmhas4cbb45e028936cd7db9099bdb5cfb46d2dc88f18397b876a6839e48977c56266—
rtplx-koa95ac77ce6df6df3b57d922e9e471597111db8be8bd2ac726bd663ed6d9d86bb—
rtplx-jacef1b894731fe06343005940188578dfe98d3a75513d0138cff32757aa4ed0d2—
NotesExcluded models, models still being measured, contamination
  • Left out of the ranking: Tev1 4B Experimental. The provider's terms restrict publishing benchmark results.
  • Items from public datasets may already be in models' training data. Some providers' terms also allow them to use submitted inputs to improve their models.

Frequently asked questions

Q.What is a decision model, and how is it different from an LLM like ChatGPT?

A.An AI model that returns probabilities for a question instead of writing text. It answers in three forms — yes/no, picking an option, or a scale score — and each answer comes with a number for how confident it is. Because it is far faster and cheaper than a text-generating LLM, it is used for judgments repeated at volume, such as sorting customer inquiries, filtering harmful content, or checking AI answers.

Q.Which decision model is most accurate in Korean and Japanese?

A.As of November 20, 2026, Decider V1.3 ranks first overall with 82.4 points. On Korean items the leader is Decider V1.3 (82.6 points); on Japanese items it is Mammoth 1 (80.5 points). The score is the equal-weight average of accuracy across 8 datasets, out of 100.

Q.Are decision models as accurate as general-purpose LLMs like GPT or Claude?

A.In the November 20, 2026 run, on the same 2,000 items the general-purpose LLMs scored GPT-6.1 Sol 83.9 pts and Claude Sonnet 5.5 80.9 pts, and the top decision model, Decider V1.3, scored 82.4. The most accurate LLM leads by 1.5 points. In exchange, the top decision model answers about 4× faster on average than GPT-6.1 Sol and costs about 1/192 as much per 1,000 items. On Japanese items alone, the decision model Mammoth 1 scored 80.5, ahead of the general-purpose LLMs (GPT-6.1 Sol 78.6 pts and Claude Sonnet 5.5 73.9 pts).

Q.What tasks are decision models good for?

A.On par with general-purpose LLMs (top decision model ahead or within 2 points): Checking AI answers, Checking claims against a source, Blocking harmful requests, Catching hate speech, Grading toxicity, Finding similar sentences, Commonsense and cause-effect, Yes/no answers from a document, and Classifying sentiment. Still well behind, where an LLM is more accurate: Checking expert knowledge.

Q.How fast and how cheap are decision models?

A.Among fully measured models, median latency ranged from 211ms to 691ms, and judging 1,000 items cost $0.0021 to $0.069. None of the measured models charges for output tokens, so input length sets the cost.

Q.Can you trust the probabilities (confidence) a decision model returns?

A.It depends on the model. Gemini Decision 1 has the smallest gap between confidence and actual accuracy, 1.4 points on average, so an answer given at 80% confidence is right close to 80% of the time — you can use the probability as a threshold to split automatic handling from human review. Solar Decide has the largest gap, 18.1 points on average, and is overconfident, so calibrate your own threshold before relying on it.

Q.How is this benchmark measured?

A.We drew 2,000 items across 8 datasets from Korean and Japanese public datasets and asked every model the same items. Items with options are asked again with the options reversed, to see whether the order sways the answer. The score is the equal-weight average of per-dataset accuracy, out of 100. As a reference, the general-purpose LLMs GPT-6.1 Sol and Claude Sonnet 5.5 answered the same items too (lowest reasoning setting, not ranked).

Q.Why are the items not published?

A.Publishing them would let future models train on them and inflate their scores. Instead we publish each set's item fingerprint (SHA-256), the sampling seed and the source revision so anyone can check the items are the same, and we share the list of source rows on request.

Curious how other models perform?

If there's a decision model you'd like added to the benchmark, let us know. We'll review it and include it in the next measurement.

Request an evaluation

Version history

  1. 0.2 · 2026-10-04Replaced the knowledge dataset, a translated exam (MMMLU), with exams written in Korea and Japan (KMMLU and JMMLU). The other 15 sets keep the same items as 0.1. Models added: Oct 5 Decision Model Preview, Decider 1; Oct 8 GPT-6 Luna Decisions; Oct 9 Perplexity Decider v1.1 27B; Oct 10 Microsoft-Decision-1; Oct 15 Jev 1.14, Drex v2.0, Clef Omni, Classify-1, Drex v1.5, Kev 4B, Solar Decide Flash; Oct 16 D1 Ultra; Oct 24 Decider V1.2, Drex Max, Span-02, Decider Flash V1.2; Nov 3 d1.1, Mammoth 1, Elephant 1, Joey 1; Nov 12 Gemini Decision 1, Kanana 2 V Decide; Nov 17 Decider V1.3, d1.2-ultra, d1.2, Decider Flash V1.3; Nov 20 Decider 1.1.
  2. 0.1 · 2026-10-03First release. 2,000 items from eight Korean and Japanese public datasets.

How to cite

When citing these results, please include the benchmark version and the source. The result data is published under CC BY 4.0.

AIB DecisionBench-KJ 0.2 (2026-11-20). AIB Inc. https://www.aib.vote/en/benchmarks/decisionbench-kj