Which decision model can you trust most in Korean and Japanese?
AIB DecisionBench-KJ is a decision-model benchmark for comparing AI performance in Korean and Japanese. It measures and analyzes how accurately decision models (Decision AI) judge Korean and Japanese problems, and how far you can trust the probabilities they give for their judgments.
- 35
- Models measured
- 8
- Datasets
- 2,000
- Items
- KO・JA
- Languages
Since TypeSafe released Jev on September 14, 2026, the number of decision models AIB has counted has grown to 42 in under 10 weeks. Instead of generating long text like a typical LLM, a decision model is built to return structured results fast: a yes/no judgment, a multiple-choice pick, or the probability of each option. Its speed and low cost are drawing attention in industries that need the same decision made over and over.
Typical uses include sorting customer inquiries, detecting harmful content, classifying documents and data, and routing branches in automated workflows. The more a job repeats the same judgment across large volumes of input, the more speed, cost and the reliability of the judgment probability matter alongside accuracy.
AIB DecisionBench-KJ compares leading decision models under identical conditions on 2,000 items from 8 public datasets. Centered on Korean and Japanese decision-model performance, it analyzes accuracy and stability per dataset along with practical fit, so you can see which model suits which job.
The top decision model, Decider V1.3, scores 82.4, 1.5 points below
GPT-6.1 Sol (83.9), but handles 1,000 decisions 4× faster (8m 3s vs 34m 25s) at 1/192 of the cost.
Gemini Decision 1 has the most accurate probabilities (confidence off from accuracy by 1.4 pts on average), and Decider Flash V1.3 is the cheapest ($0.0021 per 1,000).
Overall ranking
- Korean
- 86.1
- Japanese
- 78.6
- Understanding & classification
- 77.7
- Review & fact check
- 89.5
- Harmful content
- 79.7
- Expert knowledge
- 92.8
- Latency
- 1.9s
- Within 95% | 99%
- 3.4s | 4.3s
- Longest response
- 10.3s
- Time · 1,000
- 34m 25s
- Cost · 1,000
- $0.826
- Korean
- 82.8
- Japanese
- 73.9
- Understanding & classification
- 78.3
- Review & fact check
- 88.8
- Harmful content
- 73.5
- Expert knowledge
- 88.4
- Latency
- 1.4s
- Within 95% | 99%
- 3.6s | 4.6s
- Longest response
- 7.5s
- Time · 1,000
- 31m 13s
- Cost · 1,000
- $1.693
- 🥇
PerplexityDecider V1.382.4±1.6
- Korean
- 82.6
- Japanese
- 78.2
- Understanding & classification
- 78.5
- Review & fact check
- 92.8
- Harmful content
- 77.2
- Expert knowledge
- 78.8
- Latency
- 383ms
- Within 95% | 99%
- 816ms | 1.5s
- Longest response
- 6.5s
- Time · 1,000
- 8m 3s
- Cost · 1,000
- $0.0043
- Korean
- 82.3
- Japanese
- 78.2
- Understanding & classification
- 80.5
- Review & fact check
- 92.8
- Harmful content
- 76.5
- Expert knowledge
- 76.8
- Latency
- 530ms
- Within 95% | 99%
- 1.0s | 3.1s
- Longest response
- 9.1s
- Time · 1,000
- 10m 47s
- Cost · 1,000
- $0.0045
- Korean
- 81.7
- Japanese
- 79.2
- Understanding & classification
- 82.0
- Review & fact check
- 93.0
- Harmful content
- 75.1
- Expert knowledge
- 77.6
- Latency
- 470ms
- Within 95% | 99%
- 879ms | 1.2s
- Longest response
- 2.0s
- Time · 1,000
- 9m 6s
- Cost · 1,000
- $0.014
- Korean
- 81.5
- Japanese
- 78.1
- Understanding & classification
- 80.0
- Review & fact check
- 92.2
- Harmful content
- 76.1
- Expert knowledge
- 75.6
- Latency
- 340ms
- Within 95% | 99%
- 433ms | 900ms
- Longest response
- 19.7s
- Time · 1,000
- 6m 52s
- Cost · 1,000
- $0.0046
- Korean
- 80.7
- Japanese
- 77.5
- Understanding & classification
- 78.7
- Review & fact check
- 91.2
- Harmful content
- 76.4
- Expert knowledge
- 78.4
- Latency
- 472ms
- Within 95% | 99%
- 527ms | 746ms
- Longest response
- 2.1s
- Time · 1,000
- 8m 5s
- Cost · 1,000
- $0.0079
- Korean
- 80.6
- Japanese
- 78.9
- Understanding & classification
- 77.5
- Review & fact check
- 90.7
- Harmful content
- 75.6
- Expert knowledge
- 83.2
- Latency
- 514ms
- Within 95% | 99%
- 1.0s | 1.6s
- Longest response
- 3.3s
- Time · 1,000
- 9m 41s
- Cost · 1,000
- $0.0079
- Korean
- 78.8
- Japanese
- 79.3
- Understanding & classification
- 77.3
- Review & fact check
- 88.7
- Harmful content
- 75.6
- Expert knowledge
- 81.2
- Latency
- 469ms
- Within 95% | 99%
- 522ms | 642ms
- Longest response
- 1.2s
- Time · 1,000
- 8m 0s
- Cost · 1,000
- $0.0038
- Korean
- 78.3
- Japanese
- 76.9
- Understanding & classification
- 76.5
- Review & fact check
- 88.7
- Harmful content
- 75.3
- Expert knowledge
- 78.0
- Latency
- 466ms
- Within 95% | 99%
- 632ms | 1.1s
- Longest response
- 4.9s
- Time · 1,000
- 8m 21s
- Cost · 1,000
- $0.0038
- Korean
- 80.3
- Japanese
- 75.3
- Understanding & classification
- 76.7
- Review & fact check
- 88.3
- Harmful content
- 72.9
- Expert knowledge
- 84.7
- Latency
- 303ms
- Within 95% | 99%
- 476ms | 593ms
- Longest response
- 6.4s
- Time · 1,000
- 5m 25s
- Cost · 1,000
- $0.023
- Korean
- 77.3
- Japanese
- 80.5
- Understanding & classification
- 74.0
- Review & fact check
- 92.7
- Harmful content
- 70.3
- Expert knowledge
- 79.6
- Latency
- 455ms
- Within 95% | 99%
- 1.3s | 2.9s
- Longest response
- 5.3s
- Time · 1,000
- 10m 52s
- Cost · 1,000
- $0.024
- Korean
- 78.0
- Japanese
- 78.2
- Understanding & classification
- 73.5
- Review & fact check
- 87.7
- Harmful content
- 74.3
- Expert knowledge
- 78.8
- Latency
- 376ms
- Within 95% | 99%
- 462ms | 593ms
- Longest response
- 1.4s
- Time · 1,000
- 6m 29s
- Cost · 1,000
- $0.022
- Korean
- 77.2
- Japanese
- 79.4
- Understanding & classification
- 74.0
- Review & fact check
- 88.8
- Harmful content
- 73.1
- Expert knowledge
- 78.4
- Latency
- 211ms
- Within 95% | 99%
- 306ms | 467ms
- Longest response
- 586ms
- Time · 1,000
- 3m 45s
- Cost · 1,000
- $0.022
- Korean
- 76.7
- Japanese
- 75.8
- Understanding & classification
- 77.8
- Review & fact check
- 91.0
- Harmful content
- 69.3
- Expert knowledge
- 70.8
- Latency
- 276ms
- Within 95% | 99%
- 399ms | 559ms
- Longest response
- 1.5s
- Time · 1,000
- 4m 52s
- Cost · 1,000
- Free for now
- Korean
- 77.0
- Japanese
- 75.0
- Understanding & classification
- 76.5
- Review & fact check
- 85.8
- Harmful content
- 73.5
- Expert knowledge
- 71.2
- Latency
- 613ms
- Within 95% | 99%
- 688ms | 748ms
- Longest response
- 1.9s
- Time · 1,000
- 10m 26s
- Cost · 1,000
- $0.0070
- Korean
- 76.2
- Japanese
- 75.2
- Understanding & classification
- 75.2
- Review & fact check
- 90.2
- Harmful content
- 69.7
- Expert knowledge
- 71.6
- Latency
- 295ms
- Within 95% | 99%
- 408ms | 729ms
- Longest response
- 1.1s
- Time · 1,000
- 5m 14s
- Cost · 1,000
- $0.0092
- Korean
- 79.6
- Japanese
- 71.5
- Understanding & classification
- 72.8
- Review & fact check
- 82.7
- Harmful content
- 75.5
- Expert knowledge
- 76.4
- Latency
- 247ms
- Within 95% | 99%
- 623ms | 1.3s
- Longest response
- 3.8s
- Time · 1,000
- 5m 19s
- Cost · 1,000
- $0.031
- Korean
- 75.8
- Japanese
- 74.6
- Understanding & classification
- 72.7
- Review & fact check
- 85.2
- Harmful content
- 73.3
- Expert knowledge
- 74.0
- Latency
- 669ms
- Within 95% | 99%
- 2.5s | 2.7s
- Longest response
- 6.1s
- Time · 1,000
- 15m 50s
- Cost · 1,000
- $0.0024
- Korean
- 75.7
- Japanese
- 72.1
- Understanding & classification
- 76.0
- Review & fact check
- 78.0
- Harmful content
- 72.7
- Expert knowledge
- 75.6
- Latency
- 432ms
- Within 95% | 99%
- 13.6s | 16.2s
- Longest response
- 59.5s
- Time · 1,000
- 1h 11m
- Cost · 1,000
- $0.024
- Korean
- 72.5
- Japanese
- 75.2
- Understanding & classification
- 75.5
- Review & fact check
- 87.3
- Harmful content
- 65.7
- Expert knowledge
- 73.2
- Latency
- 362ms
- Within 95% | 99%
- 611ms | 993ms
- Longest response
- 3.6s
- Time · 1,000
- 6m 40s
- Cost · 1,000
- $0.0075
- Korean
- 72.2
- Japanese
- 77.5
- Understanding & classification
- 73.0
- Review & fact check
- 89.0
- Harmful content
- 65.1
- Expert knowledge
- 72.4
- Latency
- 433ms
- Within 95% | 99%
- 508ms | 756ms
- Longest response
- 1.5s
- Time · 1,000
- 7m 26s
- Cost · 1,000
- $0.0086
- Korean
- 71.9
- Japanese
- 74.3
- Understanding & classification
- 76.0
- Review & fact check
- 86.8
- Harmful content
- 66.7
- Expert knowledge
- 63.2
- Latency
- 454ms
- Within 95% | 99%
- 611ms | 673ms
- Longest response
- 1.5s
- Time · 1,000
- 7m 59s
- Cost · 1,000
- $0.0023
- Korean
- 72.1
- Japanese
- 73.1
- Understanding & classification
- 72.0
- Review & fact check
- 82.8
- Harmful content
- 67.2
- Expert knowledge
- 75.2
- Latency
- 507ms
- Within 95% | 99%
- 1.1s | 1.5s
- Longest response
- 2.8s
- Time · 1,000
- 9m 37s
- Cost · 1,000
- $0.069
- Korean
- 73.0
- Japanese
- 66.7
- Understanding & classification
- 74.0
- Review & fact check
- 82.3
- Harmful content
- 74.9
- Expert knowledge
- 50.0
- Latency
- 343ms
- Within 95% | 99%
- 558ms | 616ms
- Longest response
- 1.4s
- Time · 1,000
- 6m 16s
- Cost · 1,000
- No cost
- #24
PerplexityDecider Flash V1.373.8±1.8
- Korean
- 73.0
- Japanese
- 72.5
- Understanding & classification
- 75.8
- Review & fact check
- 88.0
- Harmful content
- 64.3
- Expert knowledge
- 65.2
- Latency
- 284ms
- Within 95% | 99%
- 375ms | 534ms
- Longest response
- 4.8s
- Time · 1,000
- 5m 5s
- Cost · 1,000
- $0.0021
- Korean
- 71.7
- Japanese
- 72.3
- Understanding & classification
- 73.5
- Review & fact check
- 86.2
- Harmful content
- 60.7
- Expert knowledge
- 74.8
- Latency
- 490ms
- Within 95% | 99%
- 930ms | 1.3s
- Longest response
- 6.5s
- Time · 1,000
- 9m 23s
- Cost · 1,000
- $0.0094
- Korean
- 68.7
- Japanese
- 72.2
- Understanding & classification
- 71.5
- Review & fact check
- 79.0
- Harmful content
- 65.3
- Expert knowledge
- 71.6
- Latency
- 351ms
- Within 95% | 99%
- 476ms | 723ms
- Longest response
- 1.6s
- Time · 1,000
- 6m 11s
- Cost · 1,000
- $0.048
- Korean
- 68.5
- Japanese
- 66.5
- Understanding & classification
- 72.0
- Review & fact check
- 77.8
- Harmful content
- 68.4
- Expert knowledge
- 56.8
- Latency
- 554ms
- Within 95% | 99%
- 944ms | 1.4s
- Longest response
- 5.9s
- Time · 1,000
- 10m 11s
- Cost · 1,000
- $0.016
- Korean
- 68.2
- Japanese
- 69.4
- Understanding & classification
- 71.3
- Review & fact check
- 84.2
- Harmful content
- 59.2
- Expert knowledge
- 62.8
- Latency
- 483ms
- Within 95% | 99%
- 939ms | 1.3s
- Longest response
- 10.4s
- Time · 1,000
- 9m 26s
- Cost · 1,000
- $0.0063
- Korean
- 65.9
- Japanese
- 71.3
- Understanding & classification
- 71.5
- Review & fact check
- 78.3
- Harmful content
- 58.3
- Expert knowledge
- 77.6
- Latency
- 253ms
- Within 95% | 99%
- 550ms | 848ms
- Longest response
- 1.3s
- Time · 1,000
- 4m 36s
- Cost · 1,000
- $0.026
- Korean
- 66.2
- Japanese
- 67.8
- Understanding & classification
- 69.2
- Review & fact check
- 82.5
- Harmful content
- 59.6
- Expert knowledge
- 56.0
- Latency
- 691ms
- Within 95% | 99%
- 921ms | 1.5s
- Longest response
- 5.8s
- Time · 1,000
- 12m 17s
- Cost · 1,000
- $0.0066
- Korean
- 65.9
- Japanese
- 65.8
- Understanding & classification
- 68.2
- Review & fact check
- 68.7
- Harmful content
- 67.5
- Expert knowledge
- 59.6
- Latency
- 568ms
- Within 95% | 99%
- 800ms | 1.0s
- Longest response
- 5.8s
- Time · 1,000
- 9m 27s
- Cost · 1,000
- $0.024
- Korean
- 59.7
- Japanese
- 58.7
- Understanding & classification
- 66.5
- Review & fact check
- 80.8
- Harmful content
- 50.4
- Expert knowledge
- 51.6
- Latency
- 397ms
- Within 95% | 99%
- 439ms | 513ms
- Longest response
- 949ms
- Time · 1,000
- 6m 41s
- Cost · 1,000
- Free for now
- Korean
- 56.0
- Japanese
- 65.0
- Understanding & classification
- 61.5
- Review & fact check
- 78.3
- Harmful content
- 50.8
- Expert knowledge
- 38.4
- Latency
- 425ms
- Within 95% | 99%
- 499ms | 756ms
- Longest response
- 1.6s
- Time · 1,000
- 7m 18s
- Cost · 1,000
- $0.0024
- #34
meraGPTDecider 1.159.2±2.1
- Korean
- 57.6
- Japanese
- 56.5
- Understanding & classification
- 56.2
- Review & fact check
- 64.7
- Harmful content
- 64.3
- Expert knowledge
- 35.2
- Latency
- 607ms
- Within 95% | 99%
- 791ms | 897ms
- Longest response
- 2.2s
- Time · 1,000
- 10m 29s
- Cost · 1,000
- $0.0077
- Korean
- 52.3
- Japanese
- 58.4
- Understanding & classification
- 61.5
- Review & fact check
- 67.3
- Harmful content
- 45.2
- Expert knowledge
- 41.6
- Latency
- 596ms
- Within 95% | 99%
- 834ms | 1.5s
- Longest response
- 2.4s
- Time · 1,000
- 10m 37s
- Cost · 1,000
- $0.0085
| Rank | Model | Accuracy | Korean | Japanese | Understanding & classification | Review & fact check | Harmful content | Expert knowledge | Latency | Within 95% | Within 99% | Longest response | Time · 1,000 | Cost · 1,000 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Ref. | 83.9±1.5 | 86.1 | 78.6 | 77.7 | 89.5 | 79.7 | 92.8 | 1.9s | 3.4s | 4.3s | 10.3s | 34m 25s | $0.826 | |
| Ref. | 80.9±1.6 | 82.8 | 73.9 | 78.3 | 88.8 | 73.5 | 88.4 | 1.4s | 3.6s | 4.6s | 7.5s | 31m 13s | $1.693 | |
| 🥇 | 82.4±1.6 | 82.6 | 78.2 | 78.5 | 92.8 | 77.2 | 78.8 | 383ms | 816ms | 1.5s | 6.5s | 8m 3s | $0.0043 | |
| 🥈 | 82.3±1.5 | 82.3 | 78.2 | 80.5 | 92.8 | 76.5 | 76.8 | 530ms | 1.0s | 3.1s | 9.1s | 10m 47s | $0.0045 | |
| 🥉 | 82.2±1.6 | 81.7 | 79.2 | 82.0 | 93.0 | 75.1 | 77.6 | 470ms | 879ms | 1.2s | 2.0s | 9m 6s | $0.014 | |
| #4 | 81.7±1.6 | 81.5 | 78.1 | 80.0 | 92.2 | 76.1 | 75.6 | 340ms | 433ms | 900ms | 19.7s | 6m 52s | $0.0046 | |
| #5 | 81.6±1.5 | 80.7 | 77.5 | 78.7 | 91.2 | 76.4 | 78.4 | 472ms | 527ms | 746ms | 2.1s | 8m 5s | $0.0079 | |
| #6 | 81.5±1.5 | 80.6 | 78.9 | 77.5 | 90.7 | 75.6 | 83.2 | 514ms | 1.0s | 1.6s | 3.3s | 9m 41s | $0.0079 | |
| #7 | 80.6±1.6 | 78.8 | 79.3 | 77.3 | 88.7 | 75.6 | 81.2 | 469ms | 522ms | 642ms | 1.2s | 8m 0s | $0.0038 | |
| #8 | 79.9±1.6 | 78.3 | 76.9 | 76.5 | 88.7 | 75.3 | 78.0 | 466ms | 632ms | 1.1s | 4.9s | 8m 21s | $0.0038 | |
| #9 | 79.8±1.7 | 80.3 | 75.3 | 76.7 | 88.3 | 72.9 | 84.7 | 303ms | 476ms | 593ms | 6.4s | 5m 25s | $0.023 | |
| #10 | 78.9±1.7 | 77.3 | 80.5 | 74.0 | 92.7 | 70.3 | 79.6 | 455ms | 1.3s | 2.9s | 5.3s | 10m 52s | $0.024 | |
| #11 | 78.7±1.7 | 78.0 | 78.2 | 73.5 | 87.7 | 74.3 | 78.8 | 376ms | 462ms | 593ms | 1.4s | 6m 29s | $0.022 | |
| #12 | 78.6±1.8 | 77.2 | 79.4 | 74.0 | 88.8 | 73.1 | 78.4 | 211ms | 306ms | 467ms | 586ms | 3m 45s | $0.022 | |
| #13 | 77.7±1.7 | 76.7 | 75.8 | 77.8 | 91.0 | 69.3 | 70.8 | 276ms | 399ms | 559ms | 1.5s | 4m 52s | Free for now | |
| #14 | 77.5±1.8 | 77.0 | 75.0 | 76.5 | 85.8 | 73.5 | 71.2 | 613ms | 688ms | 748ms | 1.9s | 10m 26s | $0.0070 | |
| #15 | 77.2±1.7 | 76.2 | 75.2 | 75.2 | 90.2 | 69.7 | 71.6 | 295ms | 408ms | 729ms | 1.1s | 5m 14s | $0.0092 | |
| #16 | 77.2±1.8 | 79.6 | 71.5 | 72.8 | 82.7 | 75.5 | 76.4 | 247ms | 623ms | 1.3s | 3.8s | 5m 19s | $0.031 | |
| #17 | 76.9±1.8 | 75.8 | 74.6 | 72.7 | 85.2 | 73.3 | 74.0 | 669ms | 2.5s | 2.7s | 6.1s | 15m 50s | $0.0024 | |
| #18 | 75.3±1.8 | 75.7 | 72.1 | 76.0 | 78.0 | 72.7 | 75.6 | 432ms | 13.6s | 16.2s | 59.5s | 1h 11m | $0.024 | |
| #19 | 75.1±1.8 | 72.5 | 75.2 | 75.5 | 87.3 | 65.7 | 73.2 | 362ms | 611ms | 993ms | 3.6s | 6m 40s | $0.0075 | |
| #20 | 74.8±1.8 | 72.2 | 77.5 | 73.0 | 89.0 | 65.1 | 72.4 | 433ms | 508ms | 756ms | 1.5s | 7m 26s | $0.0086 | |
| #21 | 74.2±1.9 | 71.9 | 74.3 | 76.0 | 86.8 | 66.7 | 63.2 | 454ms | 611ms | 673ms | 1.5s | 7m 59s | $0.0023 | |
| #22 | 73.9±1.8 | 72.1 | 73.1 | 72.0 | 82.8 | 67.2 | 75.2 | 507ms | 1.1s | 1.5s | 2.8s | 9m 37s | $0.069 | |
| #23 | 73.9±1.8 | 73.0 | 66.7 | 74.0 | 82.3 | 74.9 | 50.0 | 343ms | 558ms | 616ms | 1.4s | 6m 16s | No cost | |
| #24 | 73.8±1.8 | 73.0 | 72.5 | 75.8 | 88.0 | 64.3 | 65.2 | 284ms | 375ms | 534ms | 4.8s | 5m 5s | $0.0021 | |
| #25 | 72.7±1.9 | 71.7 | 72.3 | 73.5 | 86.2 | 60.7 | 74.8 | 490ms | 930ms | 1.3s | 6.5s | 9m 23s | $0.0094 | |
| #26 | 71.5±1.9 | 68.7 | 72.2 | 71.5 | 79.0 | 65.3 | 71.6 | 351ms | 476ms | 723ms | 1.6s | 6m 11s | $0.048 | |
| #27 | 70.5±1.9 | 68.5 | 66.5 | 72.0 | 77.8 | 68.4 | 56.8 | 554ms | 944ms | 1.4s | 5.9s | 10m 11s | $0.016 | |
| #28 | 69.6±2.0 | 68.2 | 69.4 | 71.3 | 84.2 | 59.2 | 62.8 | 483ms | 939ms | 1.3s | 10.4s | 9m 26s | $0.0063 | |
| #29 | 69.4±1.9 | 65.9 | 71.3 | 71.5 | 78.3 | 58.3 | 77.6 | 253ms | 550ms | 848ms | 1.3s | 4m 36s | $0.026 | |
| #30 | JJared PalmerKev 4B | 68.0±2.0 | 66.2 | 67.8 | 69.2 | 82.5 | 59.6 | 56.0 | 691ms | 921ms | 1.5s | 5.8s | 12m 17s | $0.0066 |
| #31 | 67.0±2.0 | 65.9 | 65.8 | 68.2 | 68.7 | 67.5 | 59.6 | 568ms | 800ms | 1.0s | 5.8s | 9m 27s | $0.024 | |
| #32 | 62.9±2.0 | 59.7 | 58.7 | 66.5 | 80.8 | 50.4 | 51.6 | 397ms | 439ms | 513ms | 949ms | 6m 41s | Free for now | |
| #33 | 59.7±1.9 | 56.0 | 65.0 | 61.5 | 78.3 | 50.8 | 38.4 | 425ms | 499ms | 756ms | 1.6s | 7m 18s | $0.0024 | |
| #34 | 59.2±2.1 | 57.6 | 56.5 | 56.2 | 64.7 | 64.3 | 35.2 | 607ms | 791ms | 897ms | 2.2s | 10m 29s | $0.0077 | |
| #35 | 54.7±2.1 | 52.3 | 58.4 | 61.5 | 67.3 | 45.2 | 41.6 | 596ms | 834ms | 1.5s | 2.4s | 10m 37s | $0.0085 |
Use cases
Drex Max89.2%
Mammoth 189.2%
- M-RewardBench (한국어)
- M-RewardBench (日本語)
Decider V1.294.0%
Drex Max94.0%
- KLUE NLI
- JGLUE JNLI
d1.290.8%
Mammoth 190.4%
- PolyGuardPrompts (한국어)
- PolyGuardPrompts (日本語)
d1.194.0%Kanana 2 V Decide93.2%
- K-MHaS
Jev 1.1456.4%
Microsoft-Decision-156.0%
- RTP-LX (한국어)
- RTP-LX (日本語)
D1 Ultra83.2%
d1.281.2%
- KMMLU
- JMMLU
Decider V1.258.3%
Perplexity Decider v1.1 27B56.6%
- KLUE STS
- JGLUE JSTS
Drex Max98.8%
d1.2-ultra98.8%
- KoBEST COPA
- JGLUE JCommonsenseQA
Drex Max100.0%
d1.2100.0%
- KoBEST BoolQ
Decider V1.298.3%
Drex Max98.3%
- KoBEST SentiNeg
Reliability
d1.2-ultraoff by 3.2 pts on average · well calibrated
D1 Ultraoff by 2.1 pts on average · well calibrated
d1.2off by 3.0 pts on average · well calibrated
d1.1off by 1.9 pts on average · well calibrated
Gemini Decision 1off by 1.4 pts on average · well calibrated
Mammoth 1off by 2.3 pts on average · well calibrated
Jev 1.14off by 3.4 pts on average · well calibrated
Jev 1.13off by 3.7 pts on average · well calibrated
Microsoft-Decision-1off by 8.6 pts on average · overconfident
Span-02off by 9.3 pts on average · overconfident
D1off by 5.9 pts on average · overconfident
Elephant 1off by 5.3 pts on average · overconfident
Clefoff by 8.0 pts on average · overconfident
Clef Omnioff by 9.0 pts on average · overconfident
Classify-1off by 3.7 pts on average · well calibrated
Clef Flashoff by 9.9 pts on average · overconfident
Joey 1off by 10.2 pts on average · overconfidentEach dot = a confidence range · size = number of answers in it
Datasets
KoBEST dataset evaluation details
KoBEST BoolQ
Korean| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 100.0(100) | 99.0(100) | 100.0(100) | 99.0(100) | 99.0(100) | 99.0(100) | 100.0(100) | 100.0(100) | 99.0(100) | 100.0(100) | 99.0(100) | 100.0(100) | 100.0(100) | 100.0(100) | 99.0(100) | 98.0(100) | 98.0(100) | 94.0(100) | 96.0(100) | 100.0(100) | 98.0(100) | 99.0(100) | 93.0(100) | 99.0(100) | 97.0(100) | 96.0(100) | 87.0(100) | 93.0(100) | 94.0(100) | 94.0(100) | 86.0(100) | 91.0(100) | 89.0(100) | 75.0(100) | 89.0(100) |
| Confidence vs. accuracy gap | 2.4 pp | 2.7 pp | 1.2 pp | 2.2 pp | 3.8 pp | 3.1 pp | 4.3 pp | 3.3 pp | 3.8 pp | 2.5 pp | 8.7 pp | 8.5 pp | 2.0 pp | 4.0 pp | 4.3 pp | 2.0 pp | 2.8 pp | 3.1 pp | 3.2 pp | 4.4 pp | 5.2 pp | 3.3 pp | 5.6 pp | 1.3 pp | 9.1 pp | 2.7 pp | 12.2 pp | 5.5 pp | 4.4 pp | 3.1 pp | 7.6 pp | 7.9 pp | 5.8 pp | 10.9 pp | 4.4 pp |
| Median latency | 385ms | 546ms | 464ms | 347ms | 475ms | 507ms | 469ms | 475ms | 257ms | 461ms | 375ms | 222ms | 296ms | 622ms | 300ms | 268ms | 675ms | 318ms | 359ms | 444ms | 450ms | 427ms | 446ms | 279ms | 478ms | 355ms | 544ms | 479ms | 244ms | 700ms | 552ms | 403ms | 430ms | 588ms | 604ms |
| Gold answer | |||||||||||||||||||||||||||||||||||
| Yes | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 98.0(50) | 100.0(50) | 100.0(50) | 98.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 96.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 94.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 96.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 96.0(50) | 96.0(50) | 90.0(50) | 94.0(50) |
| No | 100.0(50) | 98.0(50) | 100.0(50) | 98.0(50) | 98.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 98.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 98.0(50) | 100.0(50) | 96.0(50) | 88.0(50) | 92.0(50) | 100.0(50) | 96.0(50) | 98.0(50) | 92.0(50) | 98.0(50) | 94.0(50) | 92.0(50) | 78.0(50) | 86.0(50) | 88.0(50) | 88.0(50) | 72.0(50) | 86.0(50) | 82.0(50) | 60.0(50) | 84.0(50) |
KoBEST COPA
Korean| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 98.9(90) | 97.8(90) | 98.9(90) | 97.8(90) | 98.9(90) | 97.8(90) | 95.6(90) | 97.8(90) | 98.9(90) | 97.8(90) | 97.8(90) | 97.8(90) | 96.7(90) | 96.7(90) | 97.8(90) | 96.7(90) | 97.8(90) | 95.6(90) | 96.7(90) | 96.7(90) | 95.6(90) | 96.7(90) | 94.4(90) | 96.7(90) | 96.7(90) | 95.6(90) | 92.2(90) | 96.7(90) | 96.7(90) | 92.2(90) | 95.6(90) | 88.9(90) | 71.1(90) | 64.4(90) | 72.2(90) |
| Option-order flips | 0.0% (90) | 1.1% (90) | 0.0% (90) | 1.1% (90) | 0.0% (90) | 1.1% (90) | 3.3% (90) | 1.1% (90) | 0.0% (90) | 0.0% (90) | 0.0% (90) | 0.0% (90) | 1.1% (90) | 0.0% (90) | 1.1% (90) | 2.2% (90) | 0.0% (90) | 5.6% (90) | 2.2% (90) | 0.0% (90) | 2.2% (90) | 0.0% (90) | 7.8% (90) | 2.2% (90) | 1.1% (90) | 0.0% (90) | 8.9% (90) | 2.2% (90) | 0.0% (90) | 4.4% (90) | 5.6% (90) | 11.1% (90) | 7.8% (90) | 13.3% (90) | 16.7% (90) |
| Confidence vs. accuracy gap | 1.5 pp | 1.2 pp | 1.5 pp | 1.7 pp | 2.6 pp | 1.0 pp | 3.1 pp | 1.0 pp | 2.3 pp | 2.5 pp | 1.8 pp | 2.7 pp | 2.5 pp | 2.7 pp | 5.7 pp | 3.0 pp | 2.2 pp | 1.7 pp | 2.5 pp | 2.3 pp | 3.2 pp | 2.2 pp | 4.3 pp | 2.1 pp | 2.3 pp | 5.3 pp | 6.8 pp | 1.8 pp | 4.1 pp | 5.9 pp | 2.9 pp | 5.7 pp | 13.3 pp | 8.9 pp | 5.6 pp |
| Median latency | 383ms | 519ms | 474ms | 340ms | 469ms | 502ms | 468ms | 462ms | 394ms | 441ms | 378ms | 218ms | 273ms | 618ms | 302ms | 226ms | 531ms | 318ms | 364ms | 432ms | 449ms | 507ms | 395ms | 281ms | 575ms | 354ms | 558ms | 574ms | 255ms | 683ms | 554ms | 394ms | 425ms | 613ms | 605ms |
| Asks for | |||||||||||||||||||||||||||||||||||
| Cause | 100.0(44) | 97.7(44) | 100.0(44) | 100.0(44) | 100.0(44) | 100.0(44) | 97.7(44) | 100.0(44) | 100.0(44) | 100.0(44) | 100.0(44) | 100.0(44) | 100.0(44) | 97.7(44) | 100.0(44) | 97.7(44) | 100.0(44) | 95.5(44) | 97.7(44) | 100.0(44) | 97.7(44) | 97.7(44) | 95.5(44) | 97.7(44) | 100.0(44) | 97.7(44) | 88.6(44) | 97.7(44) | 100.0(44) | 97.7(44) | 95.5(44) | 93.2(44) | 77.3(44) | 68.2(44) | 77.3(44) |
| Effect | 97.8(46) | 97.8(46) | 97.8(46) | 95.7(46) | 97.8(46) | 95.7(46) | 93.5(46) | 95.7(46) | 97.8(46) | 95.7(46) | 95.7(46) | 95.7(46) | 93.5(46) | 95.7(46) | 95.7(46) | 95.7(46) | 95.7(46) | 95.7(46) | 95.7(46) | 93.5(46) | 93.5(46) | 95.7(46) | 93.5(46) | 95.7(46) | 93.5(46) | 93.5(46) | 95.7(46) | 95.7(46) | 93.5(46) | 87.0(46) | 95.7(46) | 84.8(46) | 65.2(46) | 60.9(46) | 67.4(46) |
| Position of the correct option | |||||||||||||||||||||||||||||||||||
| First option | 97.8(45) | 97.8(45) | 97.8(45) | 97.8(45) | 97.8(45) | 97.8(45) | 93.3(45) | 97.8(45) | 97.8(45) | 97.8(45) | 97.8(45) | 97.8(45) | 97.8(45) | 95.6(45) | 97.8(45) | 95.6(45) | 97.8(45) | 97.8(45) | 95.6(45) | 95.6(45) | 95.6(45) | 93.3(45) | 91.1(45) | 95.6(45) | 95.6(45) | 93.3(45) | 93.3(45) | 95.6(45) | 95.6(45) | 91.1(45) | 95.6(45) | 86.7(45) | 68.9(45) | 75.6(45) | 73.3(45) |
| Second option | 100.0(45) | 97.8(45) | 100.0(45) | 97.8(45) | 100.0(45) | 97.8(45) | 97.8(45) | 97.8(45) | 100.0(45) | 97.8(45) | 97.8(45) | 97.8(45) | 95.6(45) | 97.8(45) | 97.8(45) | 97.8(45) | 97.8(45) | 93.3(45) | 97.8(45) | 97.8(45) | 95.6(45) | 100.0(45) | 97.8(45) | 97.8(45) | 97.8(45) | 97.8(45) | 91.1(45) | 97.8(45) | 97.8(45) | 93.3(45) | 95.6(45) | 91.1(45) | 73.3(45) | 53.3(45) | 71.1(45) |
KoBEST SentiNeg
Korean| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 98.3(60) | 98.3(60) | 98.3(60) | 98.3(60) | 98.3(60) | 98.3(60) | 96.7(60) | 98.3(60) | 98.3(60) | 98.3(60) | 96.7(60) | 96.7(60) | 96.7(60) | 98.3(60) | 98.3(60) | 98.3(60) | 98.3(60) | 98.3(60) | 98.3(60) | 96.7(60) | 95.0(60) | 95.0(60) | 95.0(60) | 93.3(60) | 91.7(60) | 98.3(60) | 96.7(60) | 96.7(60) | 98.3(60) | 95.0(60) | 98.3(60) | 90.0(60) | 90.0(60) | 90.0(60) | 91.7(60) |
| Confidence vs. accuracy gap | 2.5 pp | 2.3 pp | 2.0 pp | 1.4 pp | 2.3 pp | 0.9 pp | 1.7 pp | 1.4 pp | 1.8 pp | 2.9 pp | 6.1 pp | 6.4 pp | 2.2 pp | 2.6 pp | 3.4 pp | 5.0 pp | 1.1 pp | 1.0 pp | 1.2 pp | 1.7 pp | 4.4 pp | 0.8 pp | 10.4 pp | 5.1 pp | 10.6 pp | 0.5 pp | 9.9 pp | 2.0 pp | 2.3 pp | 6.9 pp | 1.7 pp | 5.9 pp | 5.7 pp | 7.4 pp | 10.6 pp |
| Median latency | 378ms | 578ms | 458ms | 344ms | 472ms | 510ms | 467ms | 454ms | 366ms | 445ms | 370ms | 217ms | 273ms | 618ms | 303ms | 220ms | 514ms | 367ms | 349ms | 425ms | 450ms | 611ms | 339ms | 282ms | 572ms | 351ms | 562ms | 473ms | 245ms | 689ms | 565ms | 392ms | 420ms | 614ms | 586ms |
| Gold answer | |||||||||||||||||||||||||||||||||||
| Positive | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 93.3(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 96.7(30) | 93.3(30) | 96.7(30) | 90.0(30) | 90.0(30) | 93.3(30) | 83.3(30) | 96.7(30) | 96.7(30) | 93.3(30) | 96.7(30) | 93.3(30) | 96.7(30) | 90.0(30) | 86.7(30) | 83.3(30) | 83.3(30) |
| Negative | 100.0(30) | 100.0(30) | 100.0(30) | 100.0(30) | 100.0(30) | 100.0(30) | 96.7(30) | 100.0(30) | 100.0(30) | 100.0(30) | 96.7(30) | 96.7(30) | 100.0(30) | 100.0(30) | 100.0(30) | 100.0(30) | 100.0(30) | 100.0(30) | 100.0(30) | 100.0(30) | 93.3(30) | 100.0(30) | 100.0(30) | 93.3(30) | 100.0(30) | 100.0(30) | 96.7(30) | 100.0(30) | 100.0(30) | 96.7(30) | 100.0(30) | 90.0(30) | 93.3(30) | 96.7(30) | 100.0(30) |
KLUE dataset evaluation details
KLUE NLI
Korean| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 98.7(150) | 98.7(150) | 99.3(150) | 98.7(150) | 98.7(150) | 97.3(150) | 96.0(150) | 95.3(150) | 96.0(150) | 98.0(150) | 92.0(150) | 92.7(150) | 96.7(150) | 89.3(150) | 96.0(150) | 82.0(150) | 88.0(150) | 80.0(150) | 92.0(150) | 91.3(150) | 94.0(150) | 81.3(150) | 93.3(150) | 93.3(150) | 92.0(150) | 78.7(150) | 88.7(150) | 94.0(150) | 79.3(150) | 86.7(150) | 66.0(150) | 87.3(150) | 82.7(150) | 64.0(150) | 57.3(150) |
| Option-order flips | 1.3% (150) | 0.0% (150) | 1.3% (150) | 0.0% (150) | 0.0% (150) | 3.3% (150) | 2.0% (150) | 4.0% (150) | 0.0% (150) | 0.0% (150) | 1.3% (150) | 0.7% (150) | 1.3% (150) | 0.0% (150) | 0.7% (150) | 8.7% (150) | 4.7% (150) | 9.3% (150) | 2.7% (150) | 1.3% (150) | 0.7% (150) | 0.0% (150) | 0.0% (150) | 0.7% (150) | 4.7% (150) | 0.0% (150) | 7.3% (150) | 1.3% (150) | 0.0% (150) | 6.0% (150) | 12.0% (150) | 1.4% (147) | 2.0% (150) | 4.7% (150) | 19.3% (150) |
| Confidence vs. accuracy gap | 3.7 pp | 2.3 pp | 3.2 pp | 3.9 pp | 10.3 pp | 10.9 pp | 11.5 pp | 11.9 pp | 5.4 pp | 3.6 pp | 3.6 pp | 3.2 pp | 3.6 pp | 4.2 pp | 5.4 pp | 2.4 pp | 5.4 pp | 15.1 pp | 2.8 pp | 2.4 pp | 4.9 pp | 7.9 pp | 7.4 pp | 3.6 pp | 7.9 pp | 9.3 pp | 8.2 pp | 5.6 pp | 4.6 pp | 9.4 pp | 14.2 pp | 8.6 pp | 7.2 pp | 3.7 pp | 8.5 pp |
| Median latency | 382ms | 530ms | 466ms | 347ms | 470ms | 505ms | 464ms | 461ms | 273ms | 449ms | 376ms | 206ms | 280ms | 602ms | 302ms | 269ms | 961ms | 401ms | 349ms | 431ms | 453ms | 472ms | 340ms | 285ms | 473ms | 355ms | 544ms | 465ms | 270ms | 700ms | 539ms | 393ms | 422ms | 596ms | 585ms |
| Gold relation | |||||||||||||||||||||||||||||||||||
| Entailment | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 100.0(50) | 96.0(50) | 98.0(50) | 98.0(50) | 98.0(50) | 100.0(50) | 94.0(50) | 94.0(50) | 100.0(50) | 90.0(50) | 98.0(50) | 80.0(50) | 86.0(50) | 76.0(50) | 92.0(50) | 96.0(50) | 98.0(50) | 84.0(50) | 98.0(50) | 96.0(50) | 90.0(50) | 94.0(50) | 88.0(50) | 98.0(50) | 84.0(50) | 90.0(50) | 58.0(50) | 92.0(50) | 90.0(50) | 70.0(50) | 54.0(50) |
| Neutral | 98.0(50) | 96.0(50) | 98.0(50) | 98.0(50) | 96.0(50) | 96.0(50) | 96.0(50) | 94.0(50) | 92.0(50) | 94.0(50) | 90.0(50) | 94.0(50) | 90.0(50) | 92.0(50) | 96.0(50) | 96.0(50) | 94.0(50) | 90.0(50) | 94.0(50) | 86.0(50) | 90.0(50) | 86.0(50) | 88.0(50) | 92.0(50) | 94.0(50) | 58.0(50) | 94.0(50) | 92.0(50) | 78.0(50) | 88.0(50) | 66.0(50) | 80.0(50) | 80.0(50) | 54.0(50) | 84.0(50) |
| Contradiction | 98.0(50) | 100.0(50) | 100.0(50) | 98.0(50) | 100.0(50) | 100.0(50) | 94.0(50) | 94.0(50) | 98.0(50) | 100.0(50) | 92.0(50) | 90.0(50) | 100.0(50) | 86.0(50) | 94.0(50) | 70.0(50) | 84.0(50) | 74.0(50) | 90.0(50) | 92.0(50) | 94.0(50) | 74.0(50) | 94.0(50) | 92.0(50) | 92.0(50) | 84.0(50) | 84.0(50) | 92.0(50) | 76.0(50) | 82.0(50) | 74.0(50) | 90.0(50) | 78.0(50) | 68.0(50) | 34.0(50) |
KLUE STS
Korean| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 52.0(100) | 58.0(100) | 62.0(100) | 53.0(100) | 56.0(100) | 48.0(100) | 51.0(100) | 44.0(100) | 46.0(100) | 37.0(100) | 38.0(100) | 35.0(100) | 52.0(100) | 46.0(100) | 44.0(100) | 41.0(100) | 34.0(100) | 48.0(100) | 39.0(100) | 36.0(100) | 49.0(100) | 33.0(100) | 51.0(100) | 49.0(100) | 41.0(100) | 32.0(100) | 41.0(100) | 30.0(100) | 33.0(100) | 37.0(100) | 30.0(100) | 29.0(100) | 34.0(100) | 26.0(100) | 31.0(100) |
| Option-order flips | 1.0% (100) | 8.0% (100) | 15.0% (100) | 6.0% (100) | 9.0% (100) | 17.0% (100) | 16.0% (100) | 25.0% (100) | 29.0% (100) | 13.0% (100) | 13.0% (100) | 17.0% (100) | 34.0% (100) | 22.0% (100) | 27.0% (100) | 30.0% (100) | 39.0% (100) | 25.0% (100) | 33.0% (100) | 32.0% (100) | 7.0% (100) | 23.0% (100) | 3.0% (100) | 14.0% (100) | 24.0% (100) | 16.0% (100) | 33.0% (100) | 48.0% (100) | 20.0% (100) | 44.0% (100) | 28.0% (100) | 19.0% (100) | 16.0% (100) | 7.0% (100) | 90.0% (100) |
| Confidence vs. accuracy gap | 11.4 pp | 10.1 pp | 7.1 pp | 13.0 pp | 11.6 pp | 9.5 pp | 8.3 pp | 9.6 pp | 3.1 pp | 13.1 pp | 22.1 pp | 26.3 pp | 19.3 pp | 24.8 pp | 14.0 pp | 29.8 pp | 29.6 pp | 49.1 pp | 14.1 pp | 17.5 pp | 8.5 pp | 16.2 pp | 15.4 pp | 8.0 pp | 6.4 pp | 23.7 pp | 6.2 pp | 27.3 pp | 19.3 pp | 18.8 pp | 29.1 pp | 21.7 pp | 11.4 pp | 21.2 pp | 9.2 pp |
| Level error | 0.55 | 0.52 | 0.48 | 0.59 | 0.54 | 0.66 | 0.57 | 0.71 | 0.73 | 0.88 | 0.82 | 0.88 | 0.57 | 0.68 | 0.61 | 0.76 | 0.97 | 0.63 | 0.73 | 0.89 | 0.66 | 0.85 | 0.56 | 0.66 | 0.72 | 0.91 | 0.73 | 0.88 | 0.90 | 0.75 | 1.13 | 1.03 | 0.93 | 1.08 | 0.99 |
| Median latency | 383ms | 528ms | 476ms | 346ms | 471ms | 511ms | 468ms | 464ms | 343ms | 840ms | 387ms | 208ms | 279ms | 602ms | 303ms | 237ms | 911ms | 6.5s | 365ms | 433ms | 452ms | 525ms | 342ms | 285ms | 497ms | 351ms | 556ms | 466ms | 257ms | 697ms | 571ms | 396ms | 423ms | 608ms | 611ms |
| Human-rated level | |||||||||||||||||||||||||||||||||||
| Level 0 | 70.6(17) | 76.5(17) | 70.6(17) | 70.6(17) | 70.6(17) | 47.1(17) | 70.6(17) | 64.7(17) | 5.9(17) | 52.9(17) | 11.8(17) | 5.9(17) | 88.2(17) | 47.1(17) | 94.1(17) | 11.8(17) | 35.3(17) | 52.9(17) | 64.7(17) | 76.5(17) | 47.1(17) | 23.5(17) | 70.6(17) | 47.1(17) | 76.5(17) | 52.9(17) | 64.7(17) | 35.3(17) | 47.1(17) | 58.8(17) | 29.4(17) | 35.3(17) | 64.7(17) | 17.6(17) | 29.4(17) |
| Level 1 | 41.2(17) | 29.4(17) | 41.2(17) | 29.4(17) | 70.6(17) | 70.6(17) | 23.5(17) | 29.4(17) | 76.5(17) | 29.4(17) | 41.2(17) | 41.2(17) | 29.4(17) | 64.7(17) | 5.9(17) | 58.8(17) | 29.4(17) | 52.9(17) | 0.0(17) | 0.0(17) | 52.9(17) | 35.3(17) | 41.2(17) | 58.8(17) | 11.8(17) | 11.8(17) | 17.6(17) | 23.5(17) | 5.9(17) | 5.9(17) | 29.4(17) | 0.0(17) | 17.6(17) | 0.0(17) | 0.0(17) |
| Level 2 | 37.5(16) | 50.0(16) | 62.5(16) | 37.5(16) | 37.5(16) | 43.8(16) | 50.0(16) | 18.8(16) | 25.0(16) | 6.3(16) | 18.8(16) | 18.8(16) | 18.8(16) | 25.0(16) | 12.5(16) | 62.5(16) | 0.0(16) | 12.5(16) | 31.3(16) | 0.0(16) | 6.3(16) | 0.0(16) | 25.0(16) | 25.0(16) | 18.8(16) | 0.0(16) | 18.8(16) | 12.5(16) | 0.0(16) | 25.0(16) | 0.0(16) | 0.0(16) | 6.3(16) | 25.0(16) | 25.0(16) |
| Level 3 | 31.3(16) | 43.8(16) | 50.0(16) | 37.5(16) | 56.3(16) | 31.3(16) | 68.8(16) | 37.5(16) | 37.5(16) | 18.8(16) | 25.0(16) | 18.8(16) | 50.0(16) | 37.5(16) | 43.8(16) | 68.8(16) | 18.8(16) | 56.3(16) | 25.0(16) | 50.0(16) | 50.0(16) | 37.5(16) | 68.8(16) | 37.5(16) | 18.8(16) | 25.0(16) | 37.5(16) | 18.8(16) | 43.8(16) | 43.8(16) | 18.8(16) | 31.3(16) | 6.3(16) | 0.0(16) | 18.8(16) |
| Level 4 | 52.9(17) | 76.5(17) | 88.2(17) | 70.6(17) | 88.2(17) | 76.5(17) | 76.5(17) | 100.0(17) | 64.7(17) | 64.7(17) | 76.5(17) | 58.8(17) | 64.7(17) | 17.6(17) | 70.6(17) | 47.1(17) | 88.2(17) | 88.2(17) | 41.2(17) | 47.1(17) | 88.2(17) | 76.5(17) | 64.7(17) | 76.5(17) | 41.2(17) | 76.5(17) | 100.0(17) | 82.4(17) | 82.4(17) | 88.2(17) | 100.0(17) | 64.7(17) | 64.7(17) | 52.9(17) | 47.1(17) |
| Level 5 | 76.5(17) | 70.6(17) | 58.8(17) | 70.6(17) | 11.8(17) | 17.6(17) | 17.6(17) | 11.8(17) | 64.7(17) | 47.1(17) | 52.9(17) | 64.7(17) | 58.8(17) | 82.4(17) | 35.3(17) | 0.0(17) | 29.4(17) | 23.5(17) | 70.6(17) | 41.2(17) | 47.1(17) | 23.5(17) | 35.3(17) | 47.1(17) | 76.5(17) | 23.5(17) | 5.9(17) | 5.9(17) | 17.6(17) | 0.0(17) | 0.0(17) | 41.2(17) | 41.2(17) | 58.8(17) | 64.7(17) |
JGLUE dataset evaluation details
JGLUE JCommonsenseQA
Japanese| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 98.7(75) | 97.3(75) | 98.7(75) | 98.7(75) | 98.7(75) | 98.7(75) | 98.7(75) | 97.3(75) | 97.3(75) | 97.3(75) | 97.3(75) | 97.3(75) | 97.3(75) | 96.0(75) | 97.3(75) | 96.0(75) | 97.3(75) | 97.3(75) | 97.3(75) | 98.7(75) | 92.0(75) | 98.7(75) | 89.3(75) | 94.7(75) | 98.7(75) | 92.0(75) | 90.7(75) | 96.0(75) | 98.7(75) | 96.0(75) | 89.3(75) | 86.7(75) | 82.7(75) | 65.3(75) | 73.3(75) |
| Option-order flips | 1.3% (75) | 5.3% (75) | 1.3% (75) | 0.0% (75) | 2.7% (75) | 0.0% (75) | 2.7% (75) | 1.3% (75) | 0.0% (75) | 1.3% (75) | 0.0% (75) | 0.0% (75) | 0.0% (75) | 0.0% (75) | 4.0% (75) | 5.3% (75) | 2.7% (75) | 5.3% (75) | 1.3% (75) | 0.0% (75) | 5.3% (75) | 0.0% (75) | 6.7% (75) | 2.7% (75) | 2.7% (75) | 0.0% (75) | 16.0% (75) | 2.7% (75) | 0.0% (75) | 14.7% (75) | 18.7% (75) | 13.3% (75) | 5.3% (75) | 5.3% (75) | 14.7% (75) |
| Confidence vs. accuracy gap | 2.9 pp | 4.3 pp | 3.2 pp | 2.5 pp | 4.5 pp | 3.9 pp | 5.1 pp | 3.3 pp | 5.6 pp | 2.3 pp | 1.6 pp | 1.9 pp | 2.6 pp | 3.7 pp | 6.3 pp | 5.3 pp | 3.9 pp | 3.9 pp | 2.5 pp | 3.7 pp | 4.0 pp | 7.7 pp | 7.1 pp | 3.8 pp | 6.7 pp | 10.5 pp | 15.9 pp | 3.4 pp | 8.0 pp | 12.6 pp | 9.4 pp | 6.6 pp | 12.0 pp | 12.0 pp | 14.0 pp |
| Median latency | 377ms | 534ms | 497ms | 328ms | 468ms | 517ms | 466ms | 466ms | 380ms | 445ms | 375ms | 217ms | 293ms | 647ms | 292ms | 286ms | 542ms | 420ms | 371ms | 434ms | 449ms | 491ms | 435ms | 283ms | 648ms | 346ms | 550ms | 560ms | 252ms | 685ms | 551ms | 392ms | 427ms | 611ms | 611ms |
| Position of the correct option | |||||||||||||||||||||||||||||||||||
| Option 1 | 100.0(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 86.7(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 100.0(15) | 93.3(15) | 100.0(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 100.0(15) | 93.3(15) | 93.3(15) | 100.0(15) | 100.0(15) | 93.3(15) | 86.7(15) | 86.7(15) | 80.0(15) | 73.3(15) | 86.7(15) |
| Option 2 | 100.0(15) | 93.3(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 93.3(15) | 100.0(15) | 93.3(15) | 100.0(15) | 100.0(15) | 100.0(15) | 86.7(15) | 100.0(15) | 73.3(15) | 93.3(15) | 100.0(15) | 93.3(15) | 80.0(15) | 86.7(15) | 100.0(15) | 100.0(15) | 86.7(15) | 73.3(15) | 80.0(15) | 73.3(15) | 60.0(15) |
| Option 3 | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 93.3(15) | 100.0(15) | 100.0(15) | 93.3(15) | 100.0(15) | 100.0(15) | 100.0(15) | 93.3(15) | 93.3(15) | 100.0(15) | 93.3(15) | 66.7(15) | 73.3(15) |
| Option 4 | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 93.3(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 86.7(15) | 100.0(15) | 86.7(15) | 93.3(15) | 100.0(15) | 93.3(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 86.7(15) | 93.3(15) | 93.3(15) | 66.7(15) | 80.0(15) |
| Option 5 | 93.3(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 100.0(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 100.0(15) | 93.3(15) | 100.0(15) | 86.7(15) | 100.0(15) | 86.7(15) | 93.3(15) | 93.3(15) | 93.3(15) | 100.0(15) | 100.0(15) | 93.3(15) | 93.3(15) | 86.7(15) | 80.0(15) | 93.3(15) | 93.3(15) | 93.3(15) | 93.3(15) | 80.0(15) | 66.7(15) | 46.7(15) | 66.7(15) |
JGLUE JNLI
Japanese| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 88.0(100) | 87.0(100) | 86.0(100) | 84.0(100) | 82.0(100) | 83.0(100) | 79.0(100) | 80.0(100) | 82.0(100) | 86.0(100) | 76.0(100) | 78.0(100) | 85.0(100) | 77.0(100) | 84.0(100) | 64.0(100) | 75.0(100) | 60.0(100) | 83.0(100) | 84.0(100) | 85.0(100) | 81.0(100) | 81.0(100) | 81.0(100) | 79.0(100) | 74.0(100) | 73.0(100) | 79.0(100) | 72.0(100) | 75.0(100) | 58.0(100) | 77.0(100) | 71.0(100) | 59.0(100) | 61.0(100) |
| Option-order flips | 2.0% (100) | 0.0% (100) | 0.0% (100) | 1.0% (100) | 4.0% (100) | 3.0% (100) | 5.0% (100) | 9.0% (100) | 0.0% (100) | 2.0% (100) | 2.0% (100) | 0.0% (100) | 4.0% (100) | 2.0% (100) | 2.0% (100) | 2.0% (100) | 5.0% (100) | 9.0% (100) | 4.0% (100) | 0.0% (100) | 1.0% (100) | 0.0% (100) | 0.0% (100) | 2.0% (100) | 3.0% (100) | 0.0% (100) | 6.0% (100) | 7.0% (100) | 0.0% (100) | 4.0% (100) | 13.0% (100) | 3.0% (100) | 0.0% (100) | 0.0% (100) | 19.0% (100) |
| Confidence vs. accuracy gap | 5.9 pp | 5.8 pp | 7.0 pp | 7.7 pp | 8.7 pp | 6.6 pp | 6.9 pp | 5.7 pp | 9.8 pp | 2.6 pp | 12.7 pp | 13.6 pp | 10.4 pp | 16.5 pp | 7.0 pp | 27.9 pp | 13.1 pp | 33.5 pp | 7.6 pp | 9.9 pp | 4.7 pp | 3.0 pp | 5.7 pp | 10.2 pp | 9.6 pp | 8.6 pp | 12.4 pp | 13.5 pp | 6.4 pp | 17.7 pp | 22.7 pp | 18.3 pp | 13.6 pp | 13.6 pp | 7.3 pp |
| Median latency | 384ms | 528ms | 466ms | 331ms | 467ms | 511ms | 466ms | 463ms | 277ms | 448ms | 376ms | 211ms | 341ms | 644ms | 310ms | 299ms | 520ms | 384ms | 355ms | 432ms | 457ms | 537ms | 338ms | 288ms | 475ms | 346ms | 546ms | 468ms | 311ms | 693ms | 555ms | 395ms | 425ms | 609ms | 617ms |
| Gold relation | |||||||||||||||||||||||||||||||||||
| Entailment | 87.9(33) | 90.9(33) | 93.9(33) | 90.9(33) | 78.8(33) | 84.8(33) | 78.8(33) | 81.8(33) | 93.9(33) | 90.9(33) | 78.8(33) | 81.8(33) | 81.8(33) | 90.9(33) | 84.8(33) | 75.8(33) | 81.8(33) | 72.7(33) | 81.8(33) | 90.9(33) | 90.9(33) | 84.8(33) | 81.8(33) | 81.8(33) | 84.8(33) | 93.9(33) | 90.9(33) | 84.8(33) | 81.8(33) | 81.8(33) | 72.7(33) | 81.8(33) | 75.8(33) | 57.6(33) | 81.8(33) |
| Neutral | 82.4(34) | 85.3(34) | 82.4(34) | 82.4(34) | 88.2(34) | 91.2(34) | 88.2(34) | 91.2(34) | 94.1(34) | 82.4(34) | 97.1(34) | 97.1(34) | 88.2(34) | 94.1(34) | 97.1(34) | 97.1(34) | 100.0(34) | 85.3(34) | 94.1(34) | 94.1(34) | 85.3(34) | 94.1(34) | 91.2(34) | 91.2(34) | 97.1(34) | 76.5(34) | 82.4(34) | 94.1(34) | 88.2(34) | 97.1(34) | 76.5(34) | 85.3(34) | 79.4(34) | 79.4(34) | 50.0(34) |
| Contradiction | 93.9(33) | 84.8(33) | 81.8(33) | 78.8(33) | 78.8(33) | 72.7(33) | 69.7(33) | 66.7(33) | 57.6(33) | 84.8(33) | 51.5(33) | 54.5(33) | 84.8(33) | 45.5(33) | 69.7(33) | 18.2(33) | 42.4(33) | 21.2(33) | 72.7(33) | 66.7(33) | 78.8(33) | 63.6(33) | 69.7(33) | 69.7(33) | 54.5(33) | 51.5(33) | 45.5(33) | 57.6(33) | 45.5(33) | 45.5(33) | 24.2(33) | 63.6(33) | 57.6(33) | 39.4(33) | 51.5(33) |
JGLUE JSTS
Japanese| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 53.3(75) | 58.7(75) | 58.7(75) | 61.3(75) | 49.3(75) | 54.7(75) | 53.3(75) | 56.0(75) | 53.3(75) | 52.0(75) | 49.3(75) | 56.0(75) | 54.7(75) | 56.0(75) | 49.3(75) | 42.7(75) | 49.3(75) | 50.7(75) | 58.7(75) | 49.3(75) | 57.3(75) | 49.3(75) | 48.0(75) | 53.3(75) | 49.3(75) | 53.3(75) | 50.7(75) | 50.7(75) | 44.0(75) | 37.3(75) | 41.3(75) | 50.7(75) | 42.7(75) | 50.7(75) | 53.3(75) |
| Option-order flips | 4.0% (75) | 8.0% (75) | 14.7% (75) | 4.0% (75) | 6.7% (75) | 4.0% (75) | 12.0% (75) | 18.7% (75) | 12.0% (75) | 13.3% (75) | 12.0% (75) | 16.0% (75) | 32.0% (75) | 9.3% (75) | 28.0% (75) | 25.3% (75) | 38.7% (75) | 25.3% (75) | 18.7% (75) | 16.0% (75) | 9.3% (75) | 24.0% (75) | 1.3% (75) | 9.3% (75) | 21.3% (75) | 14.7% (75) | 21.3% (75) | 22.7% (75) | 9.3% (75) | 25.3% (75) | 24.0% (75) | 12.0% (75) | 28.0% (75) | 13.3% (75) | 84.0% (75) |
| Confidence vs. accuracy gap | 16.3 pp | 14.4 pp | 12.3 pp | 12.4 pp | 11.5 pp | 8.1 pp | 6.1 pp | 13.8 pp | 18.9 pp | 13.2 pp | 22.3 pp | 21.7 pp | 20.6 pp | 21.3 pp | 14.6 pp | 35.3 pp | 22.1 pp | 45.3 pp | 11.5 pp | 13.2 pp | 11.1 pp | 15.2 pp | 19.1 pp | 13.8 pp | 19.3 pp | 11.5 pp | 10.0 pp | 19.5 pp | 18.2 pp | 18.0 pp | 21.3 pp | 22.2 pp | 12.9 pp | 9.2 pp | 9.7 pp |
| Level error | 0.50 | 0.51 | 0.48 | 0.48 | 0.56 | 0.54 | 0.59 | 0.56 | 0.57 | 0.59 | 0.56 | 0.57 | 0.58 | 0.47 | 0.67 | 0.65 | 0.67 | 0.56 | 0.51 | 0.62 | 0.56 | 0.64 | 0.60 | 0.53 | 0.57 | 0.65 | 0.58 | 0.65 | 0.70 | 0.71 | 0.72 | 0.69 | 0.69 | 0.66 | 0.69 |
| Median latency | 386ms | 519ms | 467ms | 328ms | 468ms | 506ms | 466ms | 462ms | 265ms | 448ms | 374ms | 219ms | 324ms | 606ms | 293ms | 357ms | 788ms | 6.8s | 383ms | 431ms | 456ms | 485ms | 340ms | 288ms | 480ms | 350ms | 545ms | 460ms | 294ms | 692ms | 577ms | 397ms | 426ms | 610ms | 635ms |
| Human-rated level | |||||||||||||||||||||||||||||||||||
| Level 0 | 83.3(12) | 91.7(12) | 91.7(12) | 91.7(12) | 83.3(12) | 83.3(12) | 83.3(12) | 83.3(12) | 83.3(12) | 91.7(12) | 75.0(12) | 58.3(12) | 100.0(12) | 83.3(12) | 100.0(12) | 75.0(12) | 83.3(12) | 100.0(12) | 100.0(12) | 91.7(12) | 83.3(12) | 75.0(12) | 91.7(12) | 83.3(12) | 100.0(12) | 91.7(12) | 100.0(12) | 91.7(12) | 83.3(12) | 75.0(12) | 83.3(12) | 83.3(12) | 100.0(12) | 83.3(12) | 83.3(12) |
| Level 1 | 15.4(13) | 7.7(13) | 15.4(13) | 0.0(13) | 23.1(13) | 23.1(13) | 7.7(13) | 7.7(13) | 30.8(13) | 0.0(13) | 15.4(13) | 23.1(13) | 30.8(13) | 15.4(13) | 15.4(13) | 23.1(13) | 23.1(13) | 23.1(13) | 0.0(13) | 0.0(13) | 7.7(13) | 30.8(13) | 0.0(13) | 7.7(13) | 0.0(13) | 0.0(13) | 0.0(13) | 0.0(13) | 0.0(13) | 0.0(13) | 30.8(13) | 0.0(13) | 7.7(13) | 46.2(13) | 15.4(13) |
| Level 2 | 41.7(12) | 50.0(12) | 58.3(12) | 66.7(12) | 41.7(12) | 58.3(12) | 83.3(12) | 66.7(12) | 50.0(12) | 33.3(12) | 33.3(12) | 50.0(12) | 41.7(12) | 41.7(12) | 25.0(12) | 66.7(12) | 0.0(12) | 41.7(12) | 41.7(12) | 25.0(12) | 58.3(12) | 16.7(12) | 50.0(12) | 41.7(12) | 16.7(12) | 25.0(12) | 41.7(12) | 33.3(12) | 8.3(12) | 25.0(12) | 8.3(12) | 0.0(12) | 0.0(12) | 25.0(12) | 41.7(12) |
| Level 3 | 61.5(13) | 76.9(13) | 53.8(13) | 76.9(13) | 61.5(13) | 61.5(13) | 61.5(13) | 69.2(13) | 46.2(13) | 76.9(13) | 69.2(13) | 61.5(13) | 46.2(13) | 61.5(13) | 61.5(13) | 38.5(13) | 76.9(13) | 53.8(13) | 84.6(13) | 100.0(13) | 76.9(13) | 84.6(13) | 61.5(13) | 84.6(13) | 69.2(13) | 69.2(13) | 61.5(13) | 76.9(13) | 76.9(13) | 38.5(13) | 38.5(13) | 76.9(13) | 23.1(13) | 15.4(13) | 61.5(13) |
| Level 4 | 41.7(12) | 50.0(12) | 50.0(12) | 58.3(12) | 41.7(12) | 58.3(12) | 41.7(12) | 66.7(12) | 58.3(12) | 58.3(12) | 41.7(12) | 66.7(12) | 33.3(12) | 50.0(12) | 41.7(12) | 25.0(12) | 75.0(12) | 50.0(12) | 58.3(12) | 33.3(12) | 66.7(12) | 58.3(12) | 33.3(12) | 58.3(12) | 41.7(12) | 66.7(12) | 58.3(12) | 58.3(12) | 58.3(12) | 58.3(12) | 66.7(12) | 83.3(12) | 66.7(12) | 83.3(12) | 41.7(12) |
| Level 5 | 76.9(13) | 76.9(13) | 84.6(13) | 76.9(13) | 46.2(13) | 46.2(13) | 46.2(13) | 46.2(13) | 53.8(13) | 53.8(13) | 61.5(13) | 76.9(13) | 76.9(13) | 84.6(13) | 53.8(13) | 30.8(13) | 38.5(13) | 38.5(13) | 69.2(13) | 46.2(13) | 53.8(13) | 30.8(13) | 53.8(13) | 46.2(13) | 69.2(13) | 69.2(13) | 46.2(13) | 46.2(13) | 38.5(13) | 30.8(13) | 23.1(13) | 61.5(13) | 61.5(13) | 53.8(13) | 76.9(13) |
KMMLU · JMMLU dataset evaluation details
KMMLU
Korean| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 70.4(125) | 68.8(125) | 68.0(125) | 68.0(125) | 72.8(125) | 75.2(125) | 73.6(125) | 70.4(125) | 78.4(125) | 70.4(125) | 68.8(125) | 69.6(125) | 64.0(125) | 57.6(125) | 61.6(125) | 67.2(125) | 63.2(125) | 68.8(125) | 63.2(125) | 64.8(125) | 48.0(125) | 67.2(125) | 36.8(125) | 54.4(125) | 64.8(125) | 59.2(125) | 44.0(125) | 48.0(125) | 66.4(125) | 44.8(125) | 50.4(125) | 36.8(125) | 25.6(125) | 24.8(125) | 31.2(125) |
| Option-order flips | 8.8% (125) | 15.2% (125) | 13.6% (125) | 16.0% (125) | 12.0% (125) | 12.8% (125) | 23.2% (125) | 20.0% (125) | 0.0% (125) | 11.2% (125) | 16.0% (125) | 19.2% (125) | 25.6% (125) | 1.6% (125) | 32.0% (125) | 30.6% (124) | 26.4% (125) | 32.0% (125) | 26.4% (125) | 20.8% (125) | 22.4% (125) | 0.0% (125) | 26.4% (125) | 20.8% (125) | 35.2% (125) | 0.0% (125) | 52.8% (125) | 35.2% (125) | 0.0% (125) | 48.8% (125) | 40.8% (125) | 49.6% (125) | 33.6% (125) | 31.2% (125) | 47.2% (125) |
| Confidence vs. accuracy gap | 11.4 pp | 11.7 pp | 12.7 pp | 11.5 pp | 7.9 pp | 4.7 pp | 6.8 pp | 6.2 pp | 6.1 pp | 12.5 pp | 9.9 pp | 6.0 pp | 15.6 pp | 19.1 pp | 4.9 pp | 7.1 pp | 14.0 pp | 19.6 pp | 9.4 pp | 8.6 pp | 22.5 pp | 9.0 pp | 14.1 pp | 17.5 pp | 6.8 pp | 11.4 pp | 5.8 pp | 25.8 pp | 9.1 pp | 5.8 pp | 17.1 pp | 14.6 pp | 17.4 pp | 28.3 pp | 26.2 pp |
| Median latency | 379ms | 528ms | 459ms | 343ms | 469ms | 507ms | 465ms | 478ms | 274ms | 451ms | 379ms | 206ms | 269ms | 605ms | 293ms | 251ms | 840ms | 8.2s | 355ms | 437ms | 446ms | 482ms | 376ms | 287ms | 482ms | 346ms | 552ms | 506ms | 247ms | 702ms | 561ms | 395ms | 431ms | 607ms | 606ms |
| Field | |||||||||||||||||||||||||||||||||||
| Humanities & social sciences | 77.4(31) | 77.4(31) | 71.0(31) | 77.4(31) | 80.6(31) | 90.3(31) | 80.6(31) | 71.0(31) | 80.6(31) | 71.0(31) | 71.0(31) | 67.7(31) | 77.4(31) | 61.3(31) | 71.0(31) | 80.6(31) | 80.6(31) | 87.1(31) | 80.6(31) | 71.0(31) | 58.1(31) | 80.6(31) | 38.7(31) | 61.3(31) | 77.4(31) | 71.0(31) | 48.4(31) | 45.2(31) | 67.7(31) | 45.2(31) | 74.2(31) | 35.5(31) | 16.1(31) | 19.4(31) | 32.3(31) |
| STEM | 56.3(32) | 62.5(32) | 59.4(32) | 56.3(32) | 65.6(32) | 75.0(32) | 68.8(32) | 75.0(32) | 90.6(32) | 71.9(32) | 68.8(32) | 65.6(32) | 62.5(32) | 59.4(32) | 62.5(32) | 65.6(32) | 65.6(32) | 68.8(32) | 59.4(32) | 68.8(32) | 34.4(32) | 56.3(32) | 43.8(32) | 50.0(32) | 68.8(32) | 59.4(32) | 40.6(32) | 50.0(32) | 65.6(32) | 50.0(32) | 46.9(32) | 50.0(32) | 31.3(32) | 28.1(32) | 34.4(32) |
| Applied science | 71.9(32) | 65.6(32) | 75.0(32) | 62.5(32) | 71.9(32) | 59.4(32) | 65.6(32) | 59.4(32) | 68.8(32) | 75.0(32) | 71.9(32) | 65.6(32) | 62.5(32) | 50.0(32) | 56.3(32) | 50.0(32) | 50.0(32) | 65.6(32) | 53.1(32) | 56.3(32) | 46.9(32) | 59.4(32) | 28.1(32) | 46.9(32) | 53.1(32) | 56.3(32) | 34.4(32) | 40.6(32) | 62.5(32) | 40.6(32) | 28.1(32) | 40.6(32) | 25.0(32) | 31.3(32) | 31.3(32) |
| Other licenses | 76.7(30) | 70.0(30) | 66.7(30) | 76.7(30) | 73.3(30) | 76.7(30) | 80.0(30) | 76.7(30) | 73.3(30) | 63.3(30) | 63.3(30) | 80.0(30) | 53.3(30) | 60.0(30) | 56.7(30) | 73.3(30) | 56.7(30) | 53.3(30) | 60.0(30) | 63.3(30) | 53.3(30) | 73.3(30) | 36.7(30) | 60.0(30) | 60.0(30) | 50.0(30) | 53.3(30) | 56.7(30) | 70.0(30) | 43.3(30) | 53.3(30) | 20.0(30) | 30.0(30) | 20.0(30) | 26.7(30) |
| Test-taker accuracy | |||||||||||||||||||||||||||||||||||
| Under 40% (hard) | 66.7(63) | 63.5(63) | 65.1(63) | 63.5(63) | 63.5(63) | 66.7(63) | 61.9(63) | 61.9(63) | 76.2(63) | 69.8(63) | 66.7(63) | 65.1(63) | 54.0(63) | 42.9(63) | 55.6(63) | 60.3(63) | 52.4(63) | 55.6(63) | 54.0(63) | 58.7(63) | 34.9(63) | 61.9(63) | 31.7(63) | 39.7(63) | 50.8(63) | 52.4(63) | 39.7(63) | 36.5(63) | 55.6(63) | 42.9(63) | 38.1(63) | 34.9(63) | 28.6(63) | 25.4(63) | 25.4(63) |
| 40–70% | 76.4(55) | 76.4(55) | 72.7(55) | 74.5(55) | 83.6(55) | 83.6(55) | 85.5(55) | 78.2(55) | 85.5(55) | 69.1(55) | 72.7(55) | 76.4(55) | 76.4(55) | 70.9(55) | 69.1(55) | 78.2(55) | 74.5(55) | 81.8(55) | 70.9(55) | 69.1(55) | 61.8(55) | 70.9(55) | 36.4(55) | 70.9(55) | 81.8(55) | 67.3(55) | 49.1(55) | 60.0(55) | 78.2(55) | 43.6(55) | 61.8(55) | 36.4(55) | 25.5(55) | 21.8(55) | 36.4(55) |
| 70% or more (easy) | 66.7(3) | 66.7(3) | 66.7(3) | 66.7(3) | 66.7(3) | 66.7(3) | 100.0(3) | 100.0(3) | 33.3(3) | 100.0(3) | 66.7(3) | 66.7(3) | 66.7(3) | 100.0(3) | 66.7(3) | 33.3(3) | 100.0(3) | 100.0(3) | 100.0(3) | 66.7(3) | 66.7(3) | 100.0(3) | 66.7(3) | 66.7(3) | 66.7(3) | 66.7(3) | 66.7(3) | 66.7(3) | 66.7(3) | 100.0(3) | 66.7(3) | 33.3(3) | 0.0(3) | 0.0(3) | 66.7(3) |
| Position of the correct option | |||||||||||||||||||||||||||||||||||
| A | 70.0(30) | 63.3(30) | 60.0(30) | 63.3(30) | 70.0(30) | 76.7(30) | 70.0(30) | 63.3(30) | 73.3(30) | 60.0(30) | 53.3(30) | 53.3(30) | 56.7(30) | 53.3(30) | 50.0(30) | 63.3(30) | 56.7(30) | 66.7(30) | 53.3(30) | 53.3(30) | 56.7(30) | 60.0(30) | 33.3(30) | 60.0(30) | 66.7(30) | 50.0(30) | 53.3(30) | 50.0(30) | 70.0(30) | 26.7(30) | 43.3(30) | 33.3(30) | 20.0(30) | 23.3(30) | 33.3(30) |
| B | 65.6(32) | 65.6(32) | 68.8(32) | 71.9(32) | 81.3(32) | 71.9(32) | 81.3(32) | 68.8(32) | 87.5(32) | 78.1(32) | 75.0(32) | 84.4(32) | 56.3(32) | 65.6(32) | 62.5(32) | 65.6(32) | 71.9(32) | 65.6(32) | 62.5(32) | 62.5(32) | 46.9(32) | 59.4(32) | 31.3(32) | 50.0(32) | 59.4(32) | 59.4(32) | 40.6(32) | 40.6(32) | 62.5(32) | 37.5(32) | 50.0(32) | 28.1(32) | 21.9(32) | 34.4(32) | 34.4(32) |
| C | 68.8(32) | 68.8(32) | 62.5(32) | 56.3(32) | 65.6(32) | 62.5(32) | 62.5(32) | 68.8(32) | 71.9(32) | 68.8(32) | 75.0(32) | 65.6(32) | 65.6(32) | 65.6(32) | 65.6(32) | 68.8(32) | 62.5(32) | 78.1(32) | 78.1(32) | 68.8(32) | 40.6(32) | 68.8(32) | 46.9(32) | 53.1(32) | 68.8(32) | 56.3(32) | 53.1(32) | 56.3(32) | 59.4(32) | 62.5(32) | 71.9(32) | 46.9(32) | 37.5(32) | 18.8(32) | 40.6(32) |
| D | 77.4(31) | 77.4(31) | 80.6(31) | 80.6(31) | 74.2(31) | 90.3(31) | 80.6(31) | 80.6(31) | 80.6(31) | 74.2(31) | 71.0(31) | 74.2(31) | 77.4(31) | 45.2(31) | 67.7(31) | 71.0(31) | 61.3(31) | 64.5(31) | 58.1(31) | 74.2(31) | 48.4(31) | 80.6(31) | 35.5(31) | 54.8(31) | 64.5(31) | 71.0(31) | 29.0(31) | 45.2(31) | 74.2(31) | 51.6(31) | 35.5(31) | 38.7(31) | 22.6(31) | 22.6(31) | 16.1(31) |
JMMLU
Japanese| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 87.2(125) | 84.8(125) | 87.2(125) | 83.2(125) | 84.0(125) | 91.2(125) | 88.8(125) | 85.6(125) | 91.1(123) | 88.8(125) | 88.8(125) | 87.2(125) | 77.6(125) | 84.8(125) | 81.6(125) | 85.6(125) | 84.8(125) | 82.4(125) | 83.2(125) | 80.0(125) | 78.4(125) | 83.2(125) | 63.2(125) | 76.0(125) | 84.8(125) | 84.0(125) | 69.6(125) | 77.6(125) | 88.8(125) | 67.2(125) | 68.8(125) | 66.4(125) | 51.2(125) | 45.6(125) | 52.0(125) |
| Option-order flips | 5.6% (125) | 4.8% (125) | 4.8% (125) | 8.0% (125) | 3.2% (125) | 3.2% (125) | 4.8% (125) | 8.8% (125) | 0.0% (123) | 5.6% (125) | 2.4% (125) | 1.6% (125) | 11.2% (125) | 1.6% (125) | 15.2% (125) | 5.7% (124) | 12.0% (125) | 12.8% (125) | 15.2% (125) | 12.0% (125) | 6.4% (125) | 0.0% (125) | 23.2% (125) | 12.0% (125) | 9.6% (125) | 0.0% (125) | 28.8% (125) | 13.6% (125) | 0.0% (125) | 24.8% (125) | 27.2% (125) | 30.4% (125) | 22.4% (125) | 32.0% (125) | 34.4% (125) |
| Confidence vs. accuracy gap | 8.6 pp | 7.1 pp | 7.1 pp | 8.7 pp | 6.2 pp | 3.7 pp | 4.6 pp | 5.9 pp | 6.3 pp | 3.4 pp | 3.0 pp | 4.3 pp | 13.7 pp | 11.0 pp | 4.9 pp | 5.2 pp | 6.5 pp | 11.4 pp | 9.4 pp | 5.7 pp | 9.7 pp | 6.4 pp | 9.4 pp | 10.1 pp | 4.6 pp | 7.9 pp | 9.2 pp | 9.5 pp | 4.8 pp | 6.4 pp | 11.5 pp | 12.8 pp | 7.9 pp | 15.8 pp | 12.0 pp |
| Median latency | 379ms | 542ms | 477ms | 335ms | 468ms | 503ms | 467ms | 463ms | 245ms | 457ms | 377ms | 209ms | 304ms | 623ms | 291ms | 294ms | 651ms | 338ms | 369ms | 431ms | 466ms | 503ms | 384ms | 284ms | 490ms | 352ms | 584ms | 490ms | 277ms | 694ms | 568ms | 398ms | 426ms | 593ms | 601ms |
| Question origin | |||||||||||||||||||||||||||||||||||
| Japan-specific | 88.9(63) | 88.9(63) | 92.1(63) | 92.1(63) | 88.9(63) | 96.8(63) | 93.7(63) | 88.9(63) | 96.8(63) | 92.1(63) | 92.1(63) | 90.5(63) | 84.1(63) | 81.0(63) | 88.9(63) | 93.7(63) | 88.9(63) | 87.3(63) | 92.1(63) | 82.5(63) | 84.1(63) | 85.7(63) | 74.6(63) | 79.4(63) | 92.1(63) | 87.3(63) | 76.2(63) | 82.5(63) | 87.3(63) | 73.0(63) | 82.5(63) | 77.8(63) | 52.4(63) | 54.0(63) | 60.3(63) |
| Translated from MMLU | 85.5(62) | 80.6(62) | 82.3(62) | 74.2(62) | 79.0(62) | 85.5(62) | 83.9(62) | 82.3(62) | 85.0(60) | 85.5(62) | 85.5(62) | 83.9(62) | 71.0(62) | 88.7(62) | 74.2(62) | 77.4(62) | 80.6(62) | 77.4(62) | 74.2(62) | 77.4(62) | 72.6(62) | 80.6(62) | 51.6(62) | 72.6(62) | 77.4(62) | 80.6(62) | 62.9(62) | 72.6(62) | 90.3(62) | 61.3(62) | 54.8(62) | 54.8(62) | 50.0(62) | 37.1(62) | 43.5(62) |
| Japan-specific subjects | |||||||||||||||||||||||||||||||||||
| Japanese history | 61.5(13) | 76.9(13) | 76.9(13) | 76.9(13) | 69.2(13) | 92.3(13) | 84.6(13) | 76.9(13) | 92.3(13) | 76.9(13) | 84.6(13) | 84.6(13) | 61.5(13) | 46.2(13) | 76.9(13) | 84.6(13) | 69.2(13) | 61.5(13) | 84.6(13) | 53.8(13) | 46.2(13) | 69.2(13) | 53.8(13) | 46.2(13) | 84.6(13) | 69.2(13) | 53.8(13) | 61.5(13) | 69.2(13) | 46.2(13) | 76.9(13) | 61.5(13) | 30.8(13) | 38.5(13) | 23.1(13) |
| World history | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 92.3(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 84.6(13) | 100.0(13) | 100.0(13) | 100.0(13) | 100.0(13) | 92.3(13) | 100.0(13) | 84.6(13) | 92.3(13) | 92.3(13) | 38.5(13) | 53.8(13) | 84.6(13) |
| Civics | 92.3(13) | 92.3(13) | 100.0(13) | 92.3(13) | 100.0(13) | 100.0(13) | 100.0(13) | 84.6(13) | 100.0(13) | 100.0(13) | 92.3(13) | 92.3(13) | 92.3(13) | 84.6(13) | 84.6(13) | 100.0(13) | 92.3(13) | 92.3(13) | 92.3(13) | 84.6(13) | 100.0(13) | 92.3(13) | 84.6(13) | 92.3(13) | 92.3(13) | 92.3(13) | 76.9(13) | 84.6(13) | 92.3(13) | 84.6(13) | 69.2(13) | 84.6(13) | 76.9(13) | 61.5(13) | 61.5(13) |
| Japanese geography | 91.7(12) | 91.7(12) | 91.7(12) | 91.7(12) | 83.3(12) | 91.7(12) | 91.7(12) | 91.7(12) | 91.7(12) | 91.7(12) | 91.7(12) | 83.3(12) | 83.3(12) | 83.3(12) | 83.3(12) | 91.7(12) | 91.7(12) | 91.7(12) | 91.7(12) | 83.3(12) | 83.3(12) | 75.0(12) | 58.3(12) | 75.0(12) | 91.7(12) | 83.3(12) | 66.7(12) | 83.3(12) | 83.3(12) | 58.3(12) | 75.0(12) | 66.7(12) | 50.0(12) | 66.7(12) | 58.3(12) |
| Idioms | 100.0(12) | 83.3(12) | 91.7(12) | 100.0(12) | 91.7(12) | 100.0(12) | 91.7(12) | 91.7(12) | 100.0(12) | 91.7(12) | 91.7(12) | 91.7(12) | 83.3(12) | 91.7(12) | 100.0(12) | 91.7(12) | 91.7(12) | 100.0(12) | 91.7(12) | 91.7(12) | 91.7(12) | 91.7(12) | 91.7(12) | 83.3(12) | 91.7(12) | 91.7(12) | 83.3(12) | 91.7(12) | 91.7(12) | 91.7(12) | 100.0(12) | 83.3(12) | 66.7(12) | 50.0(12) | 75.0(12) |
| Position of the correct option | |||||||||||||||||||||||||||||||||||
| A | 90.0(30) | 90.0(30) | 86.7(30) | 83.3(30) | 83.3(30) | 90.0(30) | 90.0(30) | 90.0(30) | 89.7(29) | 93.3(30) | 93.3(30) | 90.0(30) | 76.7(30) | 86.7(30) | 80.0(30) | 90.0(30) | 76.7(30) | 86.7(30) | 73.3(30) | 86.7(30) | 76.7(30) | 86.7(30) | 63.3(30) | 76.7(30) | 83.3(30) | 83.3(30) | 83.3(30) | 76.7(30) | 90.0(30) | 53.3(30) | 70.0(30) | 66.7(30) | 50.0(30) | 40.0(30) | 56.7(30) |
| B | 81.3(32) | 75.0(32) | 84.4(32) | 78.1(32) | 75.0(32) | 87.5(32) | 87.5(32) | 75.0(32) | 87.1(31) | 81.3(32) | 90.6(32) | 90.6(32) | 68.8(32) | 81.3(32) | 75.0(32) | 78.1(32) | 78.1(32) | 75.0(32) | 84.4(32) | 71.9(32) | 75.0(32) | 71.9(32) | 62.5(32) | 62.5(32) | 84.4(32) | 84.4(32) | 62.5(32) | 75.0(32) | 84.4(32) | 65.6(32) | 71.9(32) | 53.1(32) | 40.6(32) | 59.4(32) | 43.8(32) |
| C | 93.5(31) | 87.1(31) | 96.8(31) | 87.1(31) | 90.3(31) | 100.0(31) | 93.5(31) | 93.5(31) | 93.5(31) | 96.8(31) | 90.3(31) | 87.1(31) | 90.3(31) | 87.1(31) | 87.1(31) | 87.1(31) | 96.8(31) | 90.3(31) | 93.5(31) | 77.4(31) | 83.9(31) | 90.3(31) | 54.8(31) | 90.3(31) | 87.1(31) | 87.1(31) | 58.1(31) | 77.4(31) | 93.5(31) | 71.0(31) | 87.1(31) | 74.2(31) | 71.0(31) | 41.9(31) | 54.8(31) |
| D | 84.4(32) | 87.5(32) | 81.3(32) | 84.4(32) | 87.5(32) | 87.5(32) | 84.4(32) | 84.4(32) | 93.8(32) | 84.4(32) | 81.3(32) | 81.3(32) | 75.0(32) | 84.4(32) | 84.4(32) | 87.5(32) | 87.5(32) | 78.1(32) | 81.3(32) | 84.4(32) | 78.1(32) | 84.4(32) | 71.9(32) | 75.0(32) | 84.4(32) | 81.3(32) | 75.0(32) | 81.3(32) | 87.5(32) | 78.1(32) | 46.9(32) | 71.9(32) | 43.8(32) | 40.6(32) | 53.1(32) |
M-RewardBench dataset evaluation details
M-RewardBench (한국어)
Korean| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 91.2(125) | 90.4(125) | 92.0(125) | 90.4(125) | 88.8(125) | 88.0(125) | 84.0(125) | 84.8(125) | 82.4(125) | 90.4(125) | 86.4(125) | 87.2(125) | 88.0(125) | 83.2(125) | 88.0(125) | 87.2(125) | 86.4(125) | 78.4(125) | 85.6(125) | 86.4(125) | 80.0(125) | 80.0(125) | 70.4(125) | 83.2(125) | 80.8(125) | 76.0(125) | 69.6(125) | 78.4(125) | 72.8(125) | 83.2(125) | 68.0(125) | 72.8(125) | 73.6(125) | 60.0(125) | 65.6(125) |
| Option-order flips | 1.6% (125) | 0.8% (125) | 3.2% (125) | 1.6% (125) | 0.8% (125) | 3.2% (125) | 4.0% (125) | 7.2% (125) | 0.0% (117) | 0.0% (125) | 1.6% (125) | 2.4% (125) | 6.4% (125) | 0.8% (125) | 3.2% (125) | 4.0% (125) | 3.2% (125) | 22.4% (125) | 3.2% (125) | 0.8% (125) | 2.4% (125) | 0.0% (125) | 1.6% (125) | 0.8% (125) | 7.2% (125) | 0.0% (125) | 64.8% (125) | 3.2% (125) | 0.0% (125) | 1.6% (125) | 29.6% (125) | 8.1% (123) | 0.8% (125) | 4.0% (125) | 3.2% (125) |
| Confidence vs. accuracy gap | 4.4 pp | 4.6 pp | 3.6 pp | 6.1 pp | 6.8 pp | 4.3 pp | 4.9 pp | 5.5 pp | 6.7 pp | 5.5 pp | 6.1 pp | 3.2 pp | 9.7 pp | 11.6 pp | 3.1 pp | 5.3 pp | 10.0 pp | 18.5 pp | 1.7 pp | 5.0 pp | 6.8 pp | 15.2 pp | 7.8 pp | 10.3 pp | 7.5 pp | 18.1 pp | 3.6 pp | 14.6 pp | 18.7 pp | 3.4 pp | 18.1 pp | 13.7 pp | 10.3 pp | 12.4 pp | 11.3 pp |
| Median latency | 383ms | 537ms | 471ms | 340ms | 488ms | 585ms | 483ms | 474ms | 354ms | 499ms | 373ms | 217ms | 275ms | 632ms | 297ms | 224ms | 581ms | 520ms | 360ms | 440ms | 454ms | 576ms | 358ms | 285ms | 482ms | 356ms | 556ms | 485ms | 244ms | 694ms | 614ms | 414ms | 425ms | 637ms | 596ms |
| Category | |||||||||||||||||||||||||||||||||||
| Chat | 96.9(32) | 100.0(32) | 100.0(32) | 100.0(32) | 93.8(32) | 90.6(32) | 96.9(32) | 93.8(32) | 93.8(32) | 87.5(32) | 96.9(32) | 96.9(32) | 90.6(32) | 93.8(32) | 93.8(32) | 90.6(32) | 96.9(32) | 87.5(32) | 93.8(32) | 84.4(32) | 96.9(32) | 96.9(32) | 93.8(32) | 93.8(32) | 87.5(32) | 87.5(32) | 78.1(32) | 96.9(32) | 93.8(32) | 96.9(32) | 84.4(32) | 78.1(32) | 87.5(32) | 90.6(32) | 87.5(32) |
| Chat Hard | 80.6(31) | 80.6(31) | 74.2(31) | 74.2(31) | 77.4(31) | 74.2(31) | 74.2(31) | 77.4(31) | 77.4(31) | 83.9(31) | 74.2(31) | 74.2(31) | 71.0(31) | 67.7(31) | 74.2(31) | 77.4(31) | 74.2(31) | 54.8(31) | 67.7(31) | 77.4(31) | 54.8(31) | 51.6(31) | 38.7(31) | 67.7(31) | 54.8(31) | 45.2(31) | 54.8(31) | 54.8(31) | 48.4(31) | 67.7(31) | 41.9(31) | 45.2(31) | 58.1(31) | 32.3(31) | 35.5(31) |
| Safety | 90.3(31) | 80.6(31) | 96.8(31) | 90.3(31) | 90.3(31) | 90.3(31) | 71.0(31) | 74.2(31) | 64.5(31) | 96.8(31) | 83.9(31) | 87.1(31) | 96.8(31) | 80.6(31) | 93.5(31) | 90.3(31) | 80.6(31) | 83.9(31) | 87.1(31) | 87.1(31) | 71.0(31) | 87.1(31) | 71.0(31) | 80.6(31) | 87.1(31) | 80.6(31) | 83.9(31) | 77.4(31) | 71.0(31) | 77.4(31) | 83.9(31) | 83.9(31) | 64.5(31) | 64.5(31) | 71.0(31) |
| Reasoning | 96.8(31) | 100.0(31) | 96.8(31) | 96.8(31) | 93.5(31) | 96.8(31) | 93.5(31) | 93.5(31) | 93.5(31) | 93.5(31) | 90.3(31) | 90.3(31) | 93.5(31) | 90.3(31) | 90.3(31) | 90.3(31) | 93.5(31) | 87.1(31) | 93.5(31) | 96.8(31) | 96.8(31) | 83.9(31) | 77.4(31) | 90.3(31) | 93.5(31) | 90.3(31) | 61.3(31) | 83.9(31) | 77.4(31) | 90.3(31) | 61.3(31) | 83.9(31) | 83.9(31) | 51.6(31) | 67.7(31) |
| Position of the correct option | |||||||||||||||||||||||||||||||||||
| Better answer is A | 85.5(62) | 88.7(62) | 88.7(62) | 85.5(62) | 83.9(62) | 85.5(62) | 83.9(62) | 83.9(62) | 83.9(62) | 85.5(62) | 83.9(62) | 83.9(62) | 88.7(62) | 79.0(62) | 83.9(62) | 88.7(62) | 80.6(62) | 64.5(62) | 83.9(62) | 80.6(62) | 79.0(62) | 75.8(62) | 71.0(62) | 80.6(62) | 74.2(62) | 69.4(62) | 56.5(62) | 69.4(62) | 71.0(62) | 79.0(62) | 45.2(62) | 66.1(62) | 69.4(62) | 56.5(62) | 58.1(62) |
| Better answer is B | 96.8(63) | 92.1(63) | 95.2(63) | 95.2(63) | 93.7(63) | 90.5(63) | 84.1(63) | 85.7(63) | 81.0(63) | 95.2(63) | 88.9(63) | 90.5(63) | 87.3(63) | 87.3(63) | 92.1(63) | 85.7(63) | 92.1(63) | 92.1(63) | 87.3(63) | 92.1(63) | 81.0(63) | 84.1(63) | 69.8(63) | 85.7(63) | 87.3(63) | 82.5(63) | 82.5(63) | 87.3(63) | 74.6(63) | 87.3(63) | 90.5(63) | 79.4(63) | 77.8(63) | 63.5(63) | 73.0(63) |
M-RewardBench (日本語)
Japanese| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 85.6(125) | 88.0(125) | 86.4(125) | 87.2(125) | 85.6(125) | 84.8(125) | 83.2(125) | 82.4(125) | 81.6(125) | 88.0(125) | 84.0(125) | 85.6(125) | 84.8(125) | 80.0(125) | 83.2(125) | 81.6(125) | 78.4(125) | 76.8(125) | 80.0(125) | 84.0(125) | 77.6(125) | 76.0(125) | 73.6(125) | 83.2(125) | 81.6(125) | 72.8(125) | 69.6(125) | 75.2(125) | 75.2(125) | 73.6(125) | 67.2(125) | 76.0(125) | 75.2(125) | 66.4(125) | 68.8(125) |
| Option-order flips | 4.0% (125) | 0.0% (125) | 3.2% (125) | 2.4% (125) | 4.8% (125) | 3.2% (125) | 6.4% (125) | 8.0% (125) | 0.0% (121) | 0.8% (125) | 0.0% (125) | 0.0% (125) | 3.2% (125) | 0.8% (125) | 13.6% (125) | 12.0% (125) | 7.2% (125) | 20.8% (125) | 4.0% (125) | 3.2% (125) | 3.2% (125) | 0.0% (125) | 1.6% (125) | 0.8% (125) | 8.0% (125) | 0.0% (125) | 68.8% (125) | 1.6% (125) | 0.0% (125) | 4.8% (125) | 25.6% (125) | 10.5% (124) | 1.6% (125) | 3.2% (125) | 0.0% (125) |
| Confidence vs. accuracy gap | 5.1 pp | 5.1 pp | 5.4 pp | 7.9 pp | 4.9 pp | 4.4 pp | 5.2 pp | 8.3 pp | 4.8 pp | 5.8 pp | 10.8 pp | 11.7 pp | 8.9 pp | 9.7 pp | 3.1 pp | 7.1 pp | 15.9 pp | 17.4 pp | 7.6 pp | 6.4 pp | 7.5 pp | 17.9 pp | 9.2 pp | 7.0 pp | 9.4 pp | 19.3 pp | 4.9 pp | 15.0 pp | 14.1 pp | 10.8 pp | 16.4 pp | 11.1 pp | 5.9 pp | 14.4 pp | 11.2 pp |
| Median latency | 391ms | 541ms | 469ms | 339ms | 484ms | 526ms | 478ms | 477ms | 357ms | 481ms | 383ms | 212ms | 277ms | 624ms | 295ms | 228ms | 613ms | 421ms | 368ms | 432ms | 462ms | 575ms | 360ms | 284ms | 492ms | 363ms | 550ms | 495ms | 248ms | 689ms | 639ms | 407ms | 424ms | 628ms | 618ms |
| Category | |||||||||||||||||||||||||||||||||||
| Chat | 96.8(31) | 96.8(31) | 96.8(31) | 93.5(31) | 96.8(31) | 90.3(31) | 90.3(31) | 93.5(31) | 90.3(31) | 93.5(31) | 93.5(31) | 96.8(31) | 93.5(31) | 93.5(31) | 90.3(31) | 83.9(31) | 93.5(31) | 90.3(31) | 93.5(31) | 93.5(31) | 96.8(31) | 90.3(31) | 90.3(31) | 96.8(31) | 96.8(31) | 90.3(31) | 77.4(31) | 90.3(31) | 96.8(31) | 93.5(31) | 90.3(31) | 96.8(31) | 90.3(31) | 96.8(31) | 90.3(31) |
| Chat Hard | 62.5(32) | 75.0(32) | 62.5(32) | 75.0(32) | 62.5(32) | 62.5(32) | 62.5(32) | 56.3(32) | 62.5(32) | 68.8(32) | 59.4(32) | 59.4(32) | 65.6(32) | 56.3(32) | 65.6(32) | 56.3(32) | 53.1(32) | 56.3(32) | 50.0(32) | 62.5(32) | 50.0(32) | 50.0(32) | 50.0(32) | 62.5(32) | 53.1(32) | 46.9(32) | 34.4(32) | 40.6(32) | 53.1(32) | 46.9(32) | 50.0(32) | 46.9(32) | 46.9(32) | 34.4(32) | 50.0(32) |
| Safety | 87.1(31) | 80.6(31) | 87.1(31) | 87.1(31) | 83.9(31) | 90.3(31) | 87.1(31) | 87.1(31) | 74.2(31) | 90.3(31) | 87.1(31) | 90.3(31) | 93.5(31) | 80.6(31) | 87.1(31) | 93.5(31) | 80.6(31) | 74.2(31) | 83.9(31) | 87.1(31) | 77.4(31) | 80.6(31) | 74.2(31) | 80.6(31) | 87.1(31) | 71.0(31) | 90.3(31) | 83.9(31) | 74.2(31) | 74.2(31) | 67.7(31) | 80.6(31) | 83.9(31) | 67.7(31) | 77.4(31) |
| Reasoning | 96.8(31) | 100.0(31) | 100.0(31) | 93.5(31) | 100.0(31) | 96.8(31) | 93.5(31) | 93.5(31) | 100.0(31) | 100.0(31) | 96.8(31) | 96.8(31) | 87.1(31) | 90.3(31) | 90.3(31) | 93.5(31) | 87.1(31) | 87.1(31) | 93.5(31) | 93.5(31) | 87.1(31) | 83.9(31) | 80.6(31) | 93.5(31) | 90.3(31) | 83.9(31) | 77.4(31) | 87.1(31) | 77.4(31) | 80.6(31) | 61.3(31) | 80.6(31) | 80.6(31) | 67.7(31) | 58.1(31) |
| Position of the correct option | |||||||||||||||||||||||||||||||||||
| Better answer is A | 87.3(63) | 87.3(63) | 88.9(63) | 87.3(63) | 85.7(63) | 85.7(63) | 85.7(63) | 85.7(63) | 84.1(63) | 90.5(63) | 81.0(63) | 84.1(63) | 85.7(63) | 77.8(63) | 82.5(63) | 87.3(63) | 69.8(63) | 61.9(63) | 81.0(63) | 82.5(63) | 76.2(63) | 71.4(63) | 73.0(63) | 85.7(63) | 79.4(63) | 65.1(63) | 58.7(63) | 66.7(63) | 69.8(63) | 71.4(63) | 46.0(63) | 69.8(63) | 77.8(63) | 68.3(63) | 60.3(63) |
| Better answer is B | 83.9(62) | 88.7(62) | 83.9(62) | 87.1(62) | 85.5(62) | 83.9(62) | 80.6(62) | 79.0(62) | 79.0(62) | 85.5(62) | 87.1(62) | 87.1(62) | 83.9(62) | 82.3(62) | 83.9(62) | 75.8(62) | 87.1(62) | 91.9(62) | 79.0(62) | 85.5(62) | 79.0(62) | 80.6(62) | 74.2(62) | 80.6(62) | 83.9(62) | 80.6(62) | 80.6(62) | 83.9(62) | 80.6(62) | 75.8(62) | 88.7(62) | 82.3(62) | 72.6(62) | 64.5(62) | 77.4(62) |
PolyGuard dataset evaluation details
PolyGuardPrompts (한국어)
Korean| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 88.8(125) | 88.8(125) | 91.2(125) | 88.8(125) | 90.4(125) | 88.8(125) | 92.8(125) | 90.4(125) | 84.0(125) | 90.4(125) | 88.8(125) | 89.6(125) | 90.4(125) | 88.0(125) | 87.2(125) | 82.4(125) | 82.4(125) | 84.8(125) | 87.2(125) | 91.2(125) | 84.0(125) | 88.0(125) | 88.8(125) | 85.6(125) | 84.0(125) | 84.0(125) | 76.0(125) | 80.0(125) | 64.8(125) | 70.4(125) | 84.0(125) | 39.2(125) | 84.0(125) | 73.6(125) | 70.4(125) |
| Confidence vs. accuracy gap | 7.6 pp | 8.3 pp | 7.0 pp | 9.3 pp | 6.3 pp | 6.6 pp | 6.7 pp | 7.1 pp | 6.0 pp | 2.7 pp | 4.6 pp | 3.7 pp | 7.8 pp | 4.9 pp | 6.9 pp | 14.0 pp | 11.7 pp | 9.3 pp | 8.4 pp | 2.0 pp | 9.6 pp | 6.9 pp | 3.1 pp | 9.7 pp | 5.3 pp | 8.3 pp | 3.3 pp | 12.6 pp | 20.5 pp | 5.3 pp | 9.6 pp | 32.6 pp | 6.0 pp | 10.8 pp | 5.7 pp |
| Distance from human ratings | 0.160 | 0.165 | 0.147 | 0.159 | 0.135 | 0.154 | 0.148 | 0.155 | 0.207 | 0.173 | 0.171 | 0.172 | 0.140 | 0.182 | 0.182 | 0.207 | 0.219 | 0.202 | 0.167 | 0.168 | 0.194 | 0.158 | 0.190 | 0.189 | 0.216 | 0.200 | 0.271 | 0.229 | 0.364 | 0.325 | 0.201 | 0.483 | 0.222 | 0.291 | 0.297 |
| Median latency | 403ms | 530ms | 470ms | 338ms | 482ms | 804ms | 478ms | 466ms | 267ms | 445ms | 372ms | 210ms | 269ms | 610ms | 295ms | 247ms | 967ms | 318ms | 360ms | 431ms | 451ms | 573ms | 339ms | 281ms | 505ms | 355ms | 537ms | 470ms | 241ms | 686ms | 578ms | 396ms | 423ms | 616ms | 581ms |
| Gold label | |||||||||||||||||||||||||||||||||||
| Harmful request | 80.6(62) | 80.6(62) | 85.5(62) | 80.6(62) | 83.9(62) | 82.3(62) | 88.7(62) | 87.1(62) | 69.4(62) | 83.9(62) | 85.5(62) | 87.1(62) | 88.7(62) | 82.3(62) | 79.0(62) | 67.7(62) | 67.7(62) | 74.2(62) | 82.3(62) | 85.5(62) | 82.3(62) | 83.9(62) | 85.5(62) | 82.3(62) | 72.6(62) | 74.2(62) | 61.3(62) | 64.5(62) | 56.5(62) | 64.5(62) | 71.0(62) | 43.5(62) | 71.0(62) | 56.5(62) | 53.2(62) |
| Benign request | 96.8(63) | 96.8(63) | 96.8(63) | 96.8(63) | 96.8(63) | 95.2(63) | 96.8(63) | 93.7(63) | 98.4(63) | 96.8(63) | 92.1(63) | 92.1(63) | 92.1(63) | 93.7(63) | 95.2(63) | 96.8(63) | 96.8(63) | 95.2(63) | 92.1(63) | 96.8(63) | 85.7(63) | 92.1(63) | 92.1(63) | 88.9(63) | 95.2(63) | 93.7(63) | 90.5(63) | 95.2(63) | 73.0(63) | 76.2(63) | 96.8(63) | 34.9(63) | 96.8(63) | 90.5(63) | 87.3(63) |
| Jailbreak attempt | |||||||||||||||||||||||||||||||||||
| Adversarial | 88.9(63) | 87.3(63) | 92.1(63) | 90.5(63) | 90.5(63) | 87.3(63) | 92.1(63) | 92.1(63) | 77.8(63) | 90.5(63) | 85.7(63) | 87.3(63) | 90.5(63) | 85.7(63) | 84.1(63) | 81.0(63) | 79.4(63) | 81.0(63) | 85.7(63) | 92.1(63) | 79.4(63) | 85.7(63) | 85.7(63) | 85.7(63) | 84.1(63) | 79.4(63) | 69.8(63) | 74.6(63) | 73.0(63) | 74.6(63) | 82.5(63) | 39.7(63) | 84.1(63) | 66.7(63) | 65.1(63) |
| Plain | 88.7(62) | 90.3(62) | 90.3(62) | 87.1(62) | 90.3(62) | 90.3(62) | 93.5(62) | 88.7(62) | 90.3(62) | 90.3(62) | 91.9(62) | 91.9(62) | 90.3(62) | 90.3(62) | 90.3(62) | 83.9(62) | 85.5(62) | 88.7(62) | 88.7(62) | 90.3(62) | 88.7(62) | 90.3(62) | 91.9(62) | 85.5(62) | 83.9(62) | 88.7(62) | 82.3(62) | 85.5(62) | 56.5(62) | 66.1(62) | 85.5(62) | 38.7(62) | 83.9(62) | 80.6(62) | 75.8(62) |
| Rater agreement | |||||||||||||||||||||||||||||||||||
| 3 of 3 agree | 93.4(91) | 92.3(91) | 95.6(91) | 93.4(91) | 95.6(91) | 93.4(91) | 97.8(91) | 95.6(91) | 86.8(91) | 95.6(91) | 93.4(91) | 94.5(91) | 95.6(91) | 90.1(91) | 90.1(91) | 85.7(91) | 84.6(91) | 87.9(91) | 91.2(91) | 96.7(91) | 91.2(91) | 92.3(91) | 93.4(91) | 91.2(91) | 87.9(91) | 86.8(91) | 81.3(91) | 85.7(91) | 67.0(91) | 70.3(91) | 89.0(91) | 37.4(91) | 87.9(91) | 74.7(91) | 73.6(91) |
| 2 of 3 agree | 76.5(34) | 79.4(34) | 79.4(34) | 76.5(34) | 76.5(34) | 76.5(34) | 79.4(34) | 76.5(34) | 76.5(34) | 76.5(34) | 76.5(34) | 76.5(34) | 76.5(34) | 82.4(34) | 79.4(34) | 73.5(34) | 76.5(34) | 76.5(34) | 76.5(34) | 76.5(34) | 64.7(34) | 76.5(34) | 76.5(34) | 70.6(34) | 73.5(34) | 76.5(34) | 61.8(34) | 64.7(34) | 58.8(34) | 70.6(34) | 70.6(34) | 44.1(34) | 73.5(34) | 70.6(34) | 61.8(34) |
PolyGuardPrompts (日本語)
Japanese| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 88.8(125) | 88.0(125) | 88.8(125) | 87.2(125) | 89.6(125) | 90.4(125) | 88.8(125) | 87.2(125) | 77.6(125) | 90.4(125) | 85.6(125) | 86.4(125) | 85.6(125) | 82.4(125) | 83.2(125) | 76.8(125) | 81.6(125) | 81.6(125) | 84.0(125) | 91.2(125) | 88.8(125) | 84.8(125) | 87.2(125) | 87.2(125) | 75.2(125) | 87.2(125) | 76.8(125) | 80.0(125) | 70.4(125) | 80.0(125) | 76.8(125) | 43.2(125) | 84.0(125) | 76.0(125) | 68.0(125) |
| Confidence vs. accuracy gap | 7.7 pp | 10.7 pp | 6.5 pp | 11.1 pp | 6.9 pp | 3.9 pp | 3.9 pp | 3.1 pp | 11.0 pp | 3.9 pp | 2.5 pp | 6.5 pp | 9.0 pp | 9.6 pp | 4.7 pp | 19.7 pp | 12.5 pp | 12.2 pp | 6.5 pp | 2.9 pp | 6.6 pp | 9.0 pp | 4.8 pp | 6.8 pp | 13.2 pp | 5.1 pp | 4.6 pp | 9.6 pp | 10.6 pp | 12.7 pp | 10.5 pp | 31.3 pp | 5.9 pp | 7.6 pp | 7.8 pp |
| Distance from human ratings | 0.156 | 0.166 | 0.153 | 0.175 | 0.150 | 0.160 | 0.155 | 0.163 | 0.220 | 0.165 | 0.177 | 0.175 | 0.174 | 0.185 | 0.198 | 0.248 | 0.210 | 0.234 | 0.167 | 0.171 | 0.156 | 0.182 | 0.185 | 0.159 | 0.227 | 0.180 | 0.254 | 0.210 | 0.311 | 0.271 | 0.244 | 0.446 | 0.206 | 0.268 | 0.289 |
| Median latency | 399ms | 534ms | 474ms | 346ms | 484ms | 996ms | 479ms | 469ms | 261ms | 450ms | 371ms | 220ms | 248ms | 619ms | 292ms | 260ms | 802ms | 326ms | 377ms | 428ms | 447ms | 568ms | 344ms | 283ms | 541ms | 358ms | 561ms | 513ms | 248ms | 691ms | 613ms | 395ms | 426ms | 606ms | 569ms |
| Gold label | |||||||||||||||||||||||||||||||||||
| Harmful request | 81.0(63) | 79.4(63) | 82.5(63) | 76.2(63) | 82.5(63) | 85.7(63) | 82.5(63) | 81.0(63) | 57.1(63) | 84.1(63) | 77.8(63) | 81.0(63) | 81.0(63) | 69.8(63) | 73.0(63) | 57.1(63) | 63.5(63) | 65.1(63) | 77.8(63) | 82.5(63) | 82.5(63) | 79.4(63) | 79.4(63) | 77.8(63) | 54.0(63) | 79.4(63) | 60.3(63) | 61.9(63) | 52.4(63) | 68.3(63) | 55.6(63) | 36.5(63) | 69.8(63) | 54.0(63) | 61.9(63) |
| Benign request | 96.8(62) | 96.8(62) | 95.2(62) | 98.4(62) | 96.8(62) | 95.2(62) | 95.2(62) | 93.5(62) | 98.4(62) | 96.8(62) | 93.5(62) | 91.9(62) | 90.3(62) | 95.2(62) | 93.5(62) | 96.8(62) | 100.0(62) | 98.4(62) | 90.3(62) | 100.0(62) | 95.2(62) | 90.3(62) | 95.2(62) | 96.8(62) | 96.8(62) | 95.2(62) | 93.5(62) | 98.4(62) | 88.7(62) | 91.9(62) | 98.4(62) | 50.0(62) | 98.4(62) | 98.4(62) | 74.2(62) |
| Jailbreak attempt | |||||||||||||||||||||||||||||||||||
| Adversarial | 87.1(62) | 85.5(62) | 87.1(62) | 87.1(62) | 87.1(62) | 88.7(62) | 85.5(62) | 83.9(62) | 72.6(62) | 88.7(62) | 83.9(62) | 87.1(62) | 83.9(62) | 83.9(62) | 82.3(62) | 75.8(62) | 79.0(62) | 82.3(62) | 82.3(62) | 90.3(62) | 87.1(62) | 82.3(62) | 88.7(62) | 88.7(62) | 74.2(62) | 87.1(62) | 71.0(62) | 77.4(62) | 74.2(62) | 82.3(62) | 74.2(62) | 50.0(62) | 80.6(62) | 72.6(62) | 64.5(62) |
| Plain | 90.5(63) | 90.5(63) | 90.5(63) | 87.3(63) | 92.1(63) | 92.1(63) | 92.1(63) | 90.5(63) | 82.5(63) | 92.1(63) | 87.3(63) | 85.7(63) | 87.3(63) | 81.0(63) | 84.1(63) | 77.8(63) | 84.1(63) | 81.0(63) | 85.7(63) | 92.1(63) | 90.5(63) | 87.3(63) | 85.7(63) | 85.7(63) | 76.2(63) | 87.3(63) | 82.5(63) | 82.5(63) | 66.7(63) | 77.8(63) | 79.4(63) | 36.5(63) | 87.3(63) | 79.4(63) | 71.4(63) |
| Rater agreement | |||||||||||||||||||||||||||||||||||
| 3 of 3 agree | 96.2(79) | 96.2(79) | 96.2(79) | 94.9(79) | 96.2(79) | 94.9(79) | 98.7(79) | 93.7(79) | 86.1(79) | 97.5(79) | 93.7(79) | 93.7(79) | 92.4(79) | 92.4(79) | 89.9(79) | 83.5(79) | 89.9(79) | 86.1(79) | 93.7(79) | 96.2(79) | 97.5(79) | 92.4(79) | 93.7(79) | 94.9(79) | 86.1(79) | 92.4(79) | 88.6(79) | 89.9(79) | 70.9(79) | 84.8(79) | 84.8(79) | 46.8(79) | 89.9(79) | 79.7(79) | 82.3(79) |
| 2 of 3 agree | 76.1(46) | 73.9(46) | 76.1(46) | 73.9(46) | 78.3(46) | 82.6(46) | 71.7(46) | 76.1(46) | 63.0(46) | 78.3(46) | 71.7(46) | 73.9(46) | 73.9(46) | 65.2(46) | 71.7(46) | 65.2(46) | 67.4(46) | 73.9(46) | 67.4(46) | 82.6(46) | 73.9(46) | 71.7(46) | 76.1(46) | 73.9(46) | 56.5(46) | 78.3(46) | 56.5(46) | 63.0(46) | 69.6(46) | 71.7(46) | 63.0(46) | 37.0(46) | 73.9(46) | 69.6(46) | 43.5(46) |
K-MHaS dataset evaluation details
K-MHaS
Korean| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 88.0(250) | 86.0(250) | 81.2(250) | 84.8(250) | 94.4(250) | 89.6(250) | 88.0(250) | 94.0(250) | 85.2(250) | 67.6(250) | 79.2(250) | 75.6(250) | 71.2(250) | 79.2(250) | 74.4(250) | 90.0(250) | 87.2(250) | 83.6(250) | 70.8(250) | 57.6(250) | 60.4(250) | 75.2(250) | 93.2(250) | 58.4(250) | 55.6(250) | 71.6(250) | 83.6(250) | 49.6(250) | 63.2(250) | 54.4(250) | 75.2(250) | 68.8(250) | 25.2(250) | 78.8(250) | 23.2(250) |
| Confidence vs. accuracy gap | 4.8 pp | 4.5 pp | 3.3 pp | 3.7 pp | 9.7 pp | 4.1 pp | 11.3 pp | 10.1 pp | 6.0 pp | 12.1 pp | 3.5 pp | 4.6 pp | 15.6 pp | 7.1 pp | 6.7 pp | 3.0 pp | 3.5 pp | 10.0 pp | 10.7 pp | 16.1 pp | 9.9 pp | 11.2 pp | 7.3 pp | 19.8 pp | 21.9 pp | 15.4 pp | 15.3 pp | 34.0 pp | 22.0 pp | 12.0 pp | 9.4 pp | 8.1 pp | 49.3 pp | 9.3 pp | 44.0 pp |
| Median latency | 376ms | 532ms | 460ms | 339ms | 466ms | 509ms | 463ms | 464ms | 261ms | 455ms | 385ms | 211ms | 274ms | 609ms | 294ms | 247ms | 659ms | 363ms | 379ms | 432ms | 451ms | 472ms | 338ms | 280ms | 504ms | 356ms | 548ms | 479ms | 248ms | 695ms | 548ms | 396ms | 422ms | 587ms | 581ms |
| Hate type | |||||||||||||||||||||||||||||||||||
| Origin | 82.1(28) | 85.7(28) | 78.6(28) | 82.1(28) | 96.4(28) | 92.9(28) | 89.3(28) | 100.0(28) | 75.0(28) | 53.6(28) | 71.4(28) | 64.3(28) | 60.7(28) | 71.4(28) | 67.9(28) | 96.4(28) | 92.9(28) | 89.3(28) | 64.3(28) | 50.0(28) | 53.6(28) | 71.4(28) | 100.0(28) | 50.0(28) | 46.4(28) | 78.6(28) | 75.0(28) | 39.3(28) | 57.1(28) | 42.9(28) | 82.1(28) | 67.9(28) | 10.7(28) | 78.6(28) | 25.0(28) |
| Physical | 96.4(28) | 92.9(28) | 89.3(28) | 89.3(28) | 96.4(28) | 89.3(28) | 92.9(28) | 100.0(28) | 100.0(28) | 82.1(28) | 89.3(28) | 82.1(28) | 82.1(28) | 75.0(28) | 78.6(28) | 92.9(28) | 92.9(28) | 89.3(28) | 78.6(28) | 67.9(28) | 75.0(28) | 89.3(28) | 92.9(28) | 64.3(28) | 75.0(28) | 89.3(28) | 92.9(28) | 64.3(28) | 82.1(28) | 60.7(28) | 75.0(28) | 82.1(28) | 17.9(28) | 92.9(28) | 10.7(28) |
| Politics | 85.2(27) | 81.5(27) | 74.1(27) | 81.5(27) | 92.6(27) | 77.8(27) | 92.6(27) | 96.3(27) | 92.6(27) | 51.9(27) | 81.5(27) | 77.8(27) | 66.7(27) | 88.9(27) | 70.4(27) | 88.9(27) | 92.6(27) | 85.2(27) | 51.9(27) | 55.6(27) | 48.1(27) | 59.3(27) | 96.3(27) | 59.3(27) | 40.7(27) | 51.9(27) | 85.2(27) | 33.3(27) | 66.7(27) | 37.0(27) | 66.7(27) | 70.4(27) | 7.4(27) | 88.9(27) | 11.1(27) |
| Profanity | 92.9(28) | 92.9(28) | 92.9(28) | 92.9(28) | 96.4(28) | 92.9(28) | 85.7(28) | 96.4(28) | 85.7(28) | 75.0(28) | 71.4(28) | 64.3(28) | 67.9(28) | 89.3(28) | 85.7(28) | 89.3(28) | 71.4(28) | 78.6(28) | 75.0(28) | 60.7(28) | 57.1(28) | 82.1(28) | 96.4(28) | 57.1(28) | 32.1(28) | 78.6(28) | 89.3(28) | 50.0(28) | 46.4(28) | 35.7(28) | 67.9(28) | 75.0(28) | 0.0(28) | 78.6(28) | 10.7(28) |
| Age | 85.7(28) | 75.0(28) | 67.9(28) | 78.6(28) | 92.9(28) | 92.9(28) | 92.9(28) | 96.4(28) | 92.9(28) | 46.4(28) | 71.4(28) | 67.9(28) | 53.6(28) | 75.0(28) | 60.7(28) | 92.9(28) | 85.7(28) | 85.7(28) | 64.3(28) | 35.7(28) | 32.1(28) | 53.6(28) | 96.4(28) | 28.6(28) | 50.0(28) | 50.0(28) | 85.7(28) | 32.1(28) | 53.6(28) | 53.6(28) | 71.4(28) | 60.7(28) | 14.3(28) | 75.0(28) | 3.6(28) |
| Gender | 85.7(28) | 71.4(28) | 67.9(28) | 75.0(28) | 100.0(28) | 89.3(28) | 82.1(28) | 96.4(28) | 75.0(28) | 46.4(28) | 75.0(28) | 75.0(28) | 64.3(28) | 53.6(28) | 53.6(28) | 82.1(28) | 82.1(28) | 60.7(28) | 46.4(28) | 28.6(28) | 53.6(28) | 78.6(28) | 89.3(28) | 50.0(28) | 35.7(28) | 60.7(28) | 64.3(28) | 28.6(28) | 32.1(28) | 35.7(28) | 60.7(28) | 60.7(28) | 25.0(28) | 82.1(28) | 10.7(28) |
| Race | 81.5(27) | 88.9(27) | 85.2(27) | 85.2(27) | 88.9(27) | 85.2(27) | 77.8(27) | 96.3(27) | 66.7(27) | 74.1(27) | 77.8(27) | 66.7(27) | 77.8(27) | 77.8(27) | 70.4(27) | 85.2(27) | 77.8(27) | 74.1(27) | 74.1(27) | 66.7(27) | 70.4(27) | 81.5(27) | 81.5(27) | 63.0(27) | 59.3(27) | 70.4(27) | 85.2(27) | 59.3(27) | 77.8(27) | 66.7(27) | 70.4(27) | 51.9(27) | 48.1(27) | 48.1(27) | 33.3(27) |
| Religion | 96.4(28) | 96.4(28) | 89.3(28) | 92.9(28) | 100.0(28) | 96.4(28) | 96.4(28) | 100.0(28) | 92.9(28) | 85.7(28) | 85.7(28) | 89.3(28) | 75.0(28) | 85.7(28) | 85.7(28) | 96.4(28) | 92.9(28) | 96.4(28) | 89.3(28) | 60.7(28) | 60.7(28) | 78.6(28) | 100.0(28) | 57.1(28) | 64.3(28) | 78.6(28) | 85.7(28) | 46.4(28) | 60.7(28) | 60.7(28) | 96.4(28) | 75.0(28) | 7.1(28) | 78.6(28) | 3.6(28) |
| Not hate speech | 85.7(28) | 89.3(28) | 85.7(28) | 85.7(28) | 85.7(28) | 89.3(28) | 82.1(28) | 64.3(28) | 85.7(28) | 92.9(28) | 89.3(28) | 92.9(28) | 92.9(28) | 96.4(28) | 96.4(28) | 85.7(28) | 96.4(28) | 92.9(28) | 92.9(28) | 92.9(28) | 92.9(28) | 82.1(28) | 85.7(28) | 96.4(28) | 96.4(28) | 85.7(28) | 89.3(28) | 92.9(28) | 92.9(28) | 96.4(28) | 85.7(28) | 75.0(28) | 96.4(28) | 85.7(28) | 100.0(28) |
RTP-LX dataset evaluation details
RTP-LX (한국어)
Korean| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 60.8(125) | 61.6(125) | 56.0(125) | 60.0(125) | 38.4(125) | 46.4(125) | 37.6(125) | 35.2(125) | 57.6(125) | 49.6(125) | 54.4(125) | 50.4(125) | 46.4(125) | 60.8(125) | 48.8(125) | 67.2(125) | 47.2(125) | 51.2(125) | 32.8(125) | 38.4(125) | 58.4(125) | 35.2(125) | 51.2(125) | 56.8(125) | 49.6(125) | 33.6(125) | 45.6(125) | 57.6(125) | 37.6(125) | 50.4(125) | 40.0(125) | 46.4(125) | 37.6(125) | 42.4(125) | 45.6(125) |
| Option-order flips | 8.0% (125) | 12.0% (125) | 13.6% (125) | 20.8% (125) | 12.0% (125) | 12.8% (125) | 13.6% (125) | 8.8% (125) | 13.6% (118) | 12.0% (125) | 12.8% (125) | 11.2% (125) | 18.4% (125) | 12.8% (125) | 29.6% (125) | 19.2% (125) | 28.8% (125) | 24.8% (125) | 23.2% (125) | 13.6% (125) | 23.2% (125) | 12.8% (125) | 3.2% (125) | 11.2% (125) | 44.0% (125) | 27.2% (125) | 22.4% (125) | 18.4% (125) | 11.2% (125) | 23.2% (125) | 36.8% (125) | 17.7% (124) | 16.8% (125) | 16.0% (125) | 85.6% (125) |
| Confidence vs. accuracy gap | 8.2 pp | 11.0 pp | 11.1 pp | 9.8 pp | 18.1 pp | 16.1 pp | 21.3 pp | 24.1 pp | 9.1 pp | 14.9 pp | 11.4 pp | 20.2 pp | 25.5 pp | 10.7 pp | 19.2 pp | 5.8 pp | 22.6 pp | 39.9 pp | 26.8 pp | 23.0 pp | 14.9 pp | 27.8 pp | 12.2 pp | 8.1 pp | 15.9 pp | 29.6 pp | 15.1 pp | 21.7 pp | 28.8 pp | 4.5 pp | 31.1 pp | 7.4 pp | 16.4 pp | 7.5 pp | 9.9 pp |
| Level error | 0.40 | 0.43 | 0.48 | 0.45 | 0.69 | 0.60 | 0.66 | 0.75 | 0.40 | 0.55 | 0.50 | 0.53 | 0.59 | 0.47 | 0.53 | 0.40 | 0.53 | 0.58 | 0.73 | 0.57 | 0.57 | 0.71 | 0.62 | 0.50 | 0.41 | 0.86 | 0.44 | 0.48 | 0.73 | 0.55 | 0.62 | 0.92 | 0.64 | 0.71 | 0.66 |
| Distance from human ratings | 0.365 | 0.382 | 0.391 | 0.376 | 0.505 | 0.465 | 0.489 | 0.529 | 0.381 | 0.449 | 0.420 | 0.452 | 0.453 | 0.368 | 0.446 | 0.365 | 0.444 | 0.504 | 0.515 | 0.478 | 0.465 | 0.508 | 0.459 | 0.389 | 0.467 | 0.558 | 0.465 | 0.413 | 0.512 | 0.453 | 0.534 | 0.571 | 0.539 | 0.536 | 0.543 |
| Median latency | 386ms | 521ms | 467ms | 342ms | 471ms | 507ms | 469ms | 466ms | 340ms | 453ms | 371ms | 205ms | 244ms | 596ms | 292ms | 233ms | 741ms | 4.2s | 355ms | 432ms | 451ms | 437ms | 343ms | 285ms | 470ms | 337ms | 573ms | 469ms | 242ms | 680ms | 537ms | 390ms | 431ms | 608ms | 561ms |
| Human-rated level | |||||||||||||||||||||||||||||||||||
| Level 1 | 96.8(31) | 100.0(31) | 100.0(31) | 100.0(31) | 96.8(31) | 93.5(31) | 93.5(31) | 93.5(31) | 96.8(31) | 100.0(31) | 96.8(31) | 96.8(31) | 96.8(31) | 93.5(31) | 100.0(31) | 93.5(31) | 96.8(31) | 96.8(31) | 100.0(31) | 100.0(31) | 96.8(31) | 96.8(31) | 100.0(31) | 100.0(31) | 100.0(31) | 96.8(31) | 96.8(31) | 100.0(31) | 100.0(31) | 100.0(31) | 96.8(31) | 83.9(31) | 100.0(31) | 90.3(31) | 100.0(31) |
| Level 2 | 29.0(31) | 45.2(31) | 41.9(31) | 35.5(31) | 16.1(31) | 22.6(31) | 29.0(31) | 29.0(31) | 71.0(31) | 35.5(31) | 38.7(31) | 41.9(31) | 29.0(31) | 48.4(31) | 29.0(31) | 83.9(31) | 48.4(31) | 58.1(31) | 12.9(31) | 41.9(31) | 32.3(31) | 12.9(31) | 12.9(31) | 32.3(31) | 9.7(31) | 12.9(31) | 61.3(31) | 6.5(31) | 9.7(31) | 3.2(31) | 58.1(31) | 0.0(31) | 0.0(31) | 22.6(31) | 6.5(31) |
| Level 3 | 58.1(31) | 54.8(31) | 38.7(31) | 45.2(31) | 16.1(31) | 29.0(31) | 19.4(31) | 12.9(31) | 25.8(31) | 29.0(31) | 45.2(31) | 38.7(31) | 32.3(31) | 51.6(31) | 29.0(31) | 38.7(31) | 25.8(31) | 32.3(31) | 9.7(31) | 9.7(31) | 54.8(31) | 22.6(31) | 22.6(31) | 54.8(31) | 16.1(31) | 0.0(31) | 9.7(31) | 71.0(31) | 32.3(31) | 54.8(31) | 0.0(31) | 16.1(31) | 12.9(31) | 54.8(31) | 22.6(31) |
| Level 4 | 59.4(32) | 46.9(32) | 43.8(32) | 59.4(32) | 25.0(32) | 40.6(32) | 9.4(32) | 6.3(32) | 37.5(32) | 34.4(32) | 37.5(32) | 25.0(32) | 28.1(32) | 50.0(32) | 37.5(32) | 53.1(32) | 18.8(32) | 18.8(32) | 9.4(32) | 3.1(32) | 50.0(32) | 9.4(32) | 68.8(32) | 40.6(32) | 71.9(32) | 25.0(32) | 15.6(32) | 53.1(32) | 9.4(32) | 43.8(32) | 6.3(32) | 84.4(32) | 37.5(32) | 3.1(32) | 53.1(32) |
RTP-LX (日本語)
Japanese| PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall · Accuracy | 48.8(125) | 48.8(125) | 52.0(125) | 51.2(125) | 51.2(125) | 48.8(125) | 58.4(125) | 51.2(125) | 48.0(125) | 56.0(125) | 58.4(125) | 60.8(125) | 51.2(125) | 51.2(125) | 50.4(125) | 46.4(125) | 54.4(125) | 51.2(125) | 48.8(125) | 54.4(125) | 48.0(125) | 44.8(125) | 36.0(125) | 39.2(125) | 44.0(125) | 44.0(125) | 44.8(125) | 38.4(125) | 50.4(125) | 48.0(125) | 53.6(125) | 36.0(125) | 48.8(125) | 36.0(125) | 40.8(125) |
| Option-order flips | 9.6% (125) | 19.2% (125) | 15.2% (125) | 16.0% (125) | 16.8% (125) | 19.2% (125) | 20.0% (125) | 20.0% (125) | 16.8% (113) | 18.4% (125) | 10.4% (125) | 11.2% (125) | 16.8% (125) | 17.6% (125) | 22.4% (125) | 25.6% (125) | 40.8% (125) | 35.2% (125) | 28.0% (125) | 14.4% (125) | 18.4% (125) | 12.0% (125) | 11.2% (125) | 7.2% (125) | 52.0% (125) | 27.2% (125) | 41.6% (125) | 21.6% (125) | 10.4% (125) | 30.4% (125) | 40.8% (125) | 42.4% (125) | 26.4% (125) | 14.4% (125) | 79.2% (125) |
| Confidence vs. accuracy gap | 8.5 pp | 6.9 pp | 4.0 pp | 8.7 pp | 12.9 pp | 7.6 pp | 5.8 pp | 11.2 pp | 8.1 pp | 9.1 pp | 9.7 pp | 9.6 pp | 23.1 pp | 14.7 pp | 8.1 pp | 24.2 pp | 12.0 pp | 36.8 pp | 12.3 pp | 8.1 pp | 6.3 pp | 16.4 pp | 13.4 pp | 17.6 pp | 12.3 pp | 18.4 pp | 13.9 pp | 21.3 pp | 13.2 pp | 16.2 pp | 15.9 pp | 8.0 pp | 17.6 pp | 6.3 pp | 8.8 pp |
| Level error | 0.60 | 0.67 | 0.56 | 0.56 | 0.63 | 0.60 | 0.56 | 0.63 | 0.61 | 0.55 | 0.49 | 0.50 | 0.60 | 0.60 | 0.61 | 0.67 | 0.53 | 0.67 | 0.73 | 0.55 | 0.79 | 0.65 | 0.74 | 0.76 | 0.67 | 0.78 | 0.73 | 0.85 | 0.67 | 0.72 | 0.73 | 0.89 | 0.79 | 0.90 | 0.90 |
| Distance from human ratings | 0.414 | 0.428 | 0.392 | 0.397 | 0.430 | 0.409 | 0.403 | 0.434 | 0.426 | 0.377 | 0.363 | 0.373 | 0.446 | 0.406 | 0.424 | 0.425 | 0.383 | 0.457 | 0.447 | 0.415 | 0.486 | 0.426 | 0.492 | 0.459 | 0.461 | 0.469 | 0.470 | 0.489 | 0.433 | 0.467 | 0.472 | 0.540 | 0.518 | 0.549 | 0.556 |
| Median latency | 392ms | 527ms | 520ms | 341ms | 475ms | 516ms | 474ms | 460ms | 356ms | 719ms | 370ms | 207ms | 256ms | 591ms | 294ms | 257ms | 637ms | 8.7s | 364ms | 431ms | 516ms | 526ms | 343ms | 281ms | 477ms | 347ms | 553ms | 487ms | 251ms | 677ms | 558ms | 395ms | 422ms | 599ms | 566ms |
| Human-rated level | |||||||||||||||||||||||||||||||||||
| Level 1 | 84.0(25) | 88.0(25) | 96.0(25) | 92.0(25) | 88.0(25) | 88.0(25) | 88.0(25) | 88.0(25) | 100.0(25) | 96.0(25) | 88.0(25) | 92.0(25) | 96.0(25) | 84.0(25) | 96.0(25) | 88.0(25) | 96.0(25) | 92.0(25) | 92.0(25) | 100.0(25) | 96.0(25) | 84.0(25) | 96.0(25) | 96.0(25) | 100.0(25) | 92.0(25) | 96.0(25) | 100.0(25) | 92.0(25) | 96.0(25) | 92.0(25) | 84.0(25) | 100.0(25) | 60.0(25) | 100.0(25) |
| Level 2 | 32.0(25) | 60.0(25) | 36.0(25) | 44.0(25) | 24.0(25) | 20.0(25) | 40.0(25) | 20.0(25) | 64.0(25) | 32.0(25) | 40.0(25) | 28.0(25) | 36.0(25) | 44.0(25) | 36.0(25) | 76.0(25) | 28.0(25) | 48.0(25) | 12.0(25) | 44.0(25) | 24.0(25) | 8.0(25) | 0.0(25) | 20.0(25) | 16.0(25) | 0.0(25) | 52.0(25) | 8.0(25) | 8.0(25) | 4.0(25) | 48.0(25) | 0.0(25) | 4.0(25) | 0.0(25) | 16.0(25) |
| Level 3 | 56.0(25) | 44.0(25) | 52.0(25) | 48.0(25) | 20.0(25) | 44.0(25) | 40.0(25) | 28.0(25) | 36.0(25) | 32.0(25) | 56.0(25) | 44.0(25) | 40.0(25) | 52.0(25) | 16.0(25) | 24.0(25) | 32.0(25) | 28.0(25) | 12.0(25) | 20.0(25) | 40.0(25) | 32.0(25) | 8.0(25) | 48.0(25) | 28.0(25) | 4.0(25) | 8.0(25) | 60.0(25) | 36.0(25) | 56.0(25) | 0.0(25) | 20.0(25) | 36.0(25) | 56.0(25) | 24.0(25) |
| Level 4 | 56.0(25) | 44.0(25) | 48.0(25) | 60.0(25) | 44.0(25) | 60.0(25) | 36.0(25) | 32.0(25) | 20.0(25) | 28.0(25) | 44.0(25) | 52.0(25) | 36.0(25) | 36.0(25) | 40.0(25) | 40.0(25) | 44.0(25) | 28.0(25) | 36.0(25) | 16.0(25) | 40.0(25) | 4.0(25) | 76.0(25) | 28.0(25) | 64.0(25) | 32.0(25) | 36.0(25) | 12.0(25) | 16.0(25) | 12.0(25) | 48.0(25) | 16.0(25) | 32.0(25) | 44.0(25) | 44.0(25) |
| Level 5 | 16.0(25) | 8.0(25) | 28.0(25) | 12.0(25) | 80.0(25) | 32.0(25) | 88.0(25) | 88.0(25) | 20.0(25) | 92.0(25) | 64.0(25) | 88.0(25) | 48.0(25) | 40.0(25) | 64.0(25) | 4.0(25) | 72.0(25) | 60.0(25) | 92.0(25) | 92.0(25) | 40.0(25) | 96.0(25) | 0.0(25) | 4.0(25) | 12.0(25) | 92.0(25) | 32.0(25) | 12.0(25) | 100.0(25) | 72.0(25) | 80.0(25) | 60.0(25) | 72.0(25) | 20.0(25) | 20.0(25) |
Method
Measurement log
| Model | Requests sent | Responses received | Actual cost | Period |
|---|---|---|---|---|
| 2,000 | 2,000 | $1.652 | Oct 4 | |
| 2,000 | 2,000 | $3.387 | Oct 4 | |
| 3,340 | 3,340 | $0.014 | Nov 17 | |
| 3,340 | 3,340 | $0.015 | Oct 24 | |
| 3,340 | 3,340 | $0.046 | Oct 24 | |
| 3,340 | 3,340 | $0.015 | Oct 9 | |
| 3,340 | 3,340 | $0.026 | Nov 17 | |
| 3,340 | 3,340 | $0.026 | Oct 16 | |
| 3,340 | 3,340 | $0.013 | Nov 17 | |
| 3,340 | 3,340 | $0.013 | Nov 3 | |
| 3,340 | 3,267 | $0.078 | Nov 12 | |
| 3,340 | 3,340 | $0.082 | Nov 3 | |
| 3,340 | 3,340 | $0.073 | Oct 15 | |
| 3,340 | 3,340 | $0.073 | Oct 3 ~ Oct 4 | |
| 3,340 | 3,340 | $0 | Oct 3 ~ Oct 5 | |
| 3,340 | 3,340 | $0.023 | Oct 10 | |
| 3,340 | 3,340 | $0.031 | Oct 3 ~ Oct 4 | |
| 3,340 | 3,336 | $0.104 | Oct 8 | |
| 3,340 | 3,340 | $0.0081 | Oct 24 | |
| 3,340 | 3,340 | $0.080 | Oct 3 ~ Oct 4 | |
| 3,340 | 3,340 | $0.025 | Oct 3 ~ Oct 4 | |
| 3,340 | 3,340 | $0.029 | Nov 3 | |
| 3,340 | 3,340 | $0.0076 | Oct 24 | |
| 3,340 | 3,340 | $0.229 | Oct 3 ~ Oct 4 | |
| 3,340 | 3,340 | $0 | Nov 12 | |
| 3,340 | 3,340 | $0.0071 | Nov 17 | |
| 3,340 | 3,340 | $0.031 | Oct 15 | |
| 3,340 | 3,340 | $0.159 | Oct 15 | |
| 3,340 | 3,340 | $0.054 | Oct 15 | |
| 3,340 | 3,340 | $0.021 | Oct 15 | |
| 3,340 | 3,340 | $0.086 | Oct 3 ~ Oct 4 | |
| JKev 4B | 3,340 | 3,340 | $0.022 | Oct 15 |
| 3,340 | 3,340 | $0.080 | Oct 15 | |
| 3,340 | 3,324 | $0 | Oct 5 | |
| 3,340 | 3,340 | $0.0082 | Nov 3 | |
| 3,340 | 3,340 | $0.026 | Nov 20 | |
| 3,340 | 3,340 | $0.028 | Oct 5 |
How we measure
- 1
A real item from public data
Drawn from openly licensed Korean and Japanese datasets with a fixed seed, so correct answers are evenly mixed.
- 2
One question
Asked as yes/no, a choice of options, or a scale score. Every model gets the same items.
- 3
The answer comes from the source
The correct answer is the label from the dataset's creators. No AI model does the grading.
- 4
Probabilities only
Models return a probability for each option, with no text. The highest one is graded as the answer, and the probabilities feed the calibration measures.
The details
How to read the numbersWhat accuracy, confidence-vs-accuracy gap, option-order flips, latency and cost mean
- Accuracy
- The average of the 8 datasets' accuracies, each weighted equally. Only answers to questions asked in the original option order count.
- Korean & Japanese
- Scores computed from Korean items only and Japanese items only. Accuracy is taken per dataset that has items in that language (7 for Korean, 5 for Japanese) and averaged with equal weight. Because each language covers a different set of datasets, accuracy is not a simple average of the two.
- Confidence vs. accuracy gap
- When a model says it is 90% sure, is it right 90% of the time? For each confidence band we measure how many points the stated probability is off from actual accuracy, and average them. Closer to 0 means you can trust its probabilities more.
- Option-order flips
- How often the answer changed when the same question was asked again with the options reversed. Lower means less swayed by option position.
- Latency
- Time until the answer to one item arrives. Models called through OpenRouter include the routing time.
- Cost · 1,000
- Based on the amount actually billed. Free tiers without an official price are shown as “free for now”, and open-weight models we ran ourselves (no API on sale) as “no cost”; both are left out of cost comparisons.
On scale items (sentence similarity 0–5, toxicity 1–5) only the exact level counts as correct, so these datasets score lower.
Every metric by modelSamples, probability quality, accuracy by form and language, latency, cost
Sample sizes, probability quality, results by question form and language, latency distribution and cost. Models still in progress are computed from the answers received so far.
| Metric | PDecider V1.3 | PDecider Flash V1.3 | JKev 4B | MDecider 1.1 | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Samples | |||||||||||||||||||||||||||||||||||
| Requests to send | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 |
| Answers received | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,267 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,336 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,340 | 3,324 | 3,340 | 3,340 | 3,340 |
| Refused (scored wrong)The provider's content filter refused these items, so there is no answer. Requesting again gives the same result, so they are scored as wrong. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 69 | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 16 | 0 | 0 | 0 |
| Items graded | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 1,998 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 |
| Order-comparison pairs | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,307 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,338 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,340 | 1,333 | 1,340 | 1,340 | 1,340 |
| Probability quality | |||||||||||||||||||||||||||||||||||
| Accuracy | 82.4 | 82.3 | 82.2 | 81.7 | 81.6 | 81.5 | 80.6 | 79.9 | 79.8 | 78.9 | 78.7 | 78.6 | 77.7 | 77.5 | 77.2 | 77.2 | 76.9 | 75.3 | 75.1 | 74.8 | 74.2 | 73.9 | 73.9 | 73.8 | 72.7 | 71.5 | 70.5 | 69.6 | 69.4 | 68.0 | 67.0 | 62.9 | 59.7 | 59.2 | 54.7 |
| Average confidence | 84.4% | 83.6% | 83.9% | 84.7% | 80.9% | 81.6% | 78.5% | 80.5% | 80.4% | 81.0% | 81.7% | 82.3% | 88.4% | 85.1% | 77.8% | 86.0% | 86.2% | 93.4% | 80.8% | 79.4% | 77.1% | 81.8% | 75.8% | 82.1% | 75.5% | 80.5% | 68.4% | 82.5% | 79.2% | 70.9% | 79.9% | 74.3% | 68.8% | 64.8% | 63.6% |
| Confidence gapAverage confidence − accuracy. Positive means it answers more confidently than it is right | +2.0 pp | +1.3 pp | +1.7 pp | +3.0 pp | −0.7 pp | +0.1 pp | −2.1 pp | +0.6 pp | −1.1 pp | +2.1 pp | +3.0 pp | +3.7 pp | +10.7 pp | +7.6 pp | +0.6 pp | +8.7 pp | +9.3 pp | +18.1 pp | +5.8 pp | +4.6 pp | +2.9 pp | +7.9 pp | +1.9 pp | +8.3 pp | +2.8 pp | +9.0 pp | −2.1 pp | +12.9 pp | +9.8 pp | +2.9 pp | +12.9 pp | +11.0 pp | +9.0 pp | +5.5 pp | +8.9 pp |
| Average probability on the correct answerHow highly it rated the correct answer, wrong items included | 77.7% | 76.6% | 77.1% | 77.1% | 75.0% | 75.7% | 72.7% | 73.4% | 74.6% | 73.4% | 73.1% | 73.0% | 75.8% | 73.8% | 70.5% | 73.5% | 73.8% | 73.2% | 71.1% | 70.0% | 68.0% | 70.7% | 66.8% | 70.8% | 65.6% | 67.3% | 60.4% | 66.9% | 64.4% | 61.2% | 63.4% | 59.1% | 54.5% | 51.6% | 49.6% |
| Confidence vs. accuracy gap | 2.1 pp | 2.3 pp | 2.4 pp | 3.4 pp | 3.2 pp | 2.1 pp | 3.0 pp | 1.9 pp | 1.4 pp | 2.3 pp | 3.4 pp | 3.7 pp | 10.7 pp | 8.6 pp | 2.3 pp | 8.7 pp | 9.3 pp | 18.1 pp | 5.9 pp | 5.3 pp | 4.7 pp | 8.0 pp | 3.8 pp | 8.4 pp | 4.5 pp | 9.0 pp | 3.7 pp | 13.1 pp | 9.9 pp | 3.7 pp | 12.9 pp | 11.0 pp | 10.2 pp | 5.7 pp | 9.7 pp |
| Brier scoreHow far the whole probability table is from the answer. Lower is better | 0.235 | 0.249 | 0.242 | 0.251 | 0.241 | 0.237 | 0.258 | 0.268 | 0.249 | 0.277 | 0.289 | 0.299 | 0.320 | 0.319 | 0.295 | 0.335 | 0.328 | 0.430 | 0.330 | 0.324 | 0.342 | 0.335 | 0.336 | 0.344 | 0.367 | 0.384 | 0.381 | 0.429 | 0.422 | 0.406 | 0.454 | 0.487 | 0.507 | 0.517 | 0.547 |
| Level errorOn scale items, the average gap between the model's expected level and the correct level | 0.51 | 0.54 | 0.50 | 0.52 | 0.61 | 0.60 | 0.60 | 0.67 | 0.57 | 0.63 | 0.58 | 0.61 | 0.59 | 0.56 | 0.60 | 0.61 | 0.66 | 0.61 | 0.69 | 0.65 | 0.65 | 0.71 | 0.64 | 0.62 | 0.59 | 0.81 | 0.62 | 0.71 | 0.75 | 0.68 | 0.79 | 0.90 | 0.76 | 0.84 | 0.81 |
| Distance from human ratingsCompared with the rater distributions in RTP-LX and PolyGuard. 0 = same, 1 = opposite | 0.274 | 0.285 | 0.271 | 0.277 | 0.305 | 0.297 | 0.299 | 0.320 | 0.305 | 0.291 | 0.283 | 0.293 | 0.304 | 0.286 | 0.312 | 0.311 | 0.314 | 0.349 | 0.324 | 0.308 | 0.325 | 0.319 | 0.331 | 0.299 | 0.343 | 0.352 | 0.365 | 0.335 | 0.405 | 0.379 | 0.363 | 0.510 | 0.371 | 0.411 | 0.421 |
| Option-order flips | 4.2% | 6.3% | 7.2% | 6.9% | 6.1% | 7.2% | 9.6% | 11.0% | 5.6% | 6.4% | 6.0% | 6.5% | 12.5% | 5.6% | 14.9% | 14.3% | 17.2% | 19.5% | 13.7% | 9.5% | 8.7% | 5.4% | 7.5% | 7.0% | 18.5% | 7.1% | 32.5% | 14.5% | 4.0% | 19.3% | 25.9% | 18.8% | 13.3% | 12.4% | 40.2% |
| Chosen answer ≠ top probabilityOn multiple-choice items, how often the model's chosen option was not its highest-probability option. Grading uses the top probability | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 901 | 0 / 915 | 2 / 915 | 1 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 1 / 913 | 0 / 915 | 1 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 0 / 915 | 18 / 915 | 2 / 909 | 0 / 915 | 0 / 915 | 8 / 915 |
| Accuracy by question form | |||||||||||||||||||||||||||||||||||
| Yes/no | 91.1 (660) | 90.0 (660) | 88.9 (660) | 89.4 (660) | 93.8 (660) | 91.8 (660) | 91.7 (660) | 93.3 (660) | 86.8 (660) | 83.9 (660) | 86.8 (660) | 85.9 (660) | 84.2 (660) | 86.4 (660) | 84.4 (660) | 88.0 (660) | 87.9 (660) | 86.4 (660) | 82.7 (660) | 80.3 (660) | 79.1 (660) | 84.8 (660) | 91.4 (660) | 78.3 (660) | 74.2 (660) | 83.0 (660) | 82.6 (660) | 72.0 (660) | 72.7 (660) | 72.0 (660) | 80.9 (660) | 63.6 (660) | 63.0 (660) | 77.7 (660) | 56.8 (660) |
| Multiple choice | 89.3 (915) | 88.6 (915) | 89.1 (915) | 88.0 (915) | 88.2 (915) | 89.1 (915) | 86.9 (915) | 86.1 (915) | 88.0 (913) | 89.2 (915) | 85.8 (915) | 86.3 (915) | 85.6 (915) | 82.2 (915) | 85.5 (915) | 81.7 (915) | 83.0 (915) | 78.9 (915) | 84.3 (915) | 84.8 (915) | 80.4 (915) | 81.6 (915) | 74.1 (915) | 82.0 (915) | 83.9 (915) | 77.8 (915) | 73.6 (915) | 79.6 (915) | 79.9 (915) | 76.1 (915) | 68.6 (915) | 73.0 (915) | 65.9 (915) | 55.5 (915) | 58.9 (915) |
| Scale score | 53.9 (425) | 56.5 (425) | 56.7 (425) | 56.0 (425) | 48.2 (425) | 48.9 (425) | 49.6 (425) | 45.6 (425) | 51.3 (425) | 48.9 (425) | 50.8 (425) | 50.8 (425) | 50.6 (425) | 53.6 (425) | 48.2 (425) | 50.6 (425) | 46.6 (425) | 50.4 (425) | 43.5 (425) | 44.5 (425) | 52.9 (425) | 40.0 (425) | 46.1 (425) | 49.2 (425) | 45.9 (425) | 39.8 (425) | 45.2 (425) | 44.2 (425) | 41.4 (425) | 44.2 (425) | 41.9 (425) | 40.0 (425) | 40.9 (425) | 38.1 (425) | 42.1 (425) |
| By language | |||||||||||||||||||||||||||||||||||
| Korean accuracy | 84.6 (1,250) | 84.3 (1,250) | 83.7 (1,250) | 83.4 (1,250) | 84.0 (1,250) | 83.0 (1,250) | 81.5 (1,250) | 81.6 (1,250) | 82.2 (1,250) | 78.1 (1,250) | 79.4 (1,250) | 78.4 (1,250) | 78.5 (1,250) | 78.9 (1,250) | 78.2 (1,250) | 81.0 (1,250) | 78.2 (1,250) | 77.6 (1,250) | 74.6 (1,250) | 73.0 (1,250) | 73.6 (1,250) | 73.9 (1,250) | 77.4 (1,250) | 74.2 (1,250) | 72.5 (1,250) | 70.9 (1,250) | 72.4 (1,250) | 69.0 (1,250) | 68.2 (1,250) | 67.8 (1,250) | 68.1 (1,250) | 64.1 (1,250) | 56.3 (1,250) | 60.6 (1,250) | 52.0 (1,250) |
| Korean confidence vs. accuracy gap | 1.4 pp | 2.7 pp | 1.6 pp | 3.0 pp | 4.3 pp | 2.8 pp | 4.0 pp | 3.4 pp | 2.4 pp | 3.0 pp | 3.0 pp | 3.1 pp | 10.0 pp | 6.9 pp | 4.3 pp | 4.8 pp | 8.5 pp | 16.0 pp | 6.6 pp | 7.1 pp | 5.3 pp | 8.5 pp | 4.9 pp | 8.3 pp | 4.2 pp | 11.1 pp | 5.5 pp | 14.7 pp | 12.0 pp | 3.8 pp | 12.9 pp | 10.4 pp | 13.1 pp | 4.8 pp | 13.2 pp |
| Korean option-order flips | 3.6% | 6.2% | 7.7% | 7.7% | 5.6% | 8.3% | 10.2% | 10.8% | 6.4% | 5.9% | 7.4% | 8.3% | 14.0% | 5.7% | 15.4% | 15.7% | 16.6% | 20.0% | 14.7% | 10.9% | 9.8% | 5.5% | 6.9% | 8.1% | 19.6% | 7.0% | 31.8% | 17.2% | 4.8% | 20.8% | 25.9% | 17.6% | 12.6% | 12.6% | 42.5% |
| Japanese accuracy | 78.7 (750) | 78.8 (750) | 79.6 (750) | 78.7 (750) | 77.5 (750) | 78.9 (750) | 78.9 (750) | 77.1 (750) | 75.7 (748) | 80.3 (750) | 77.6 (750) | 79.1 (750) | 76.4 (750) | 75.2 (750) | 75.6 (750) | 70.8 (750) | 74.5 (750) | 71.5 (750) | 76.0 (750) | 77.6 (750) | 75.1 (750) | 73.7 (750) | 67.9 (750) | 73.2 (750) | 72.9 (750) | 72.4 (750) | 67.3 (750) | 70.4 (750) | 71.3 (750) | 68.1 (750) | 65.2 (750) | 60.9 (750) | 65.2 (750) | 56.8 (750) | 59.1 (750) |
| Japanese confidence vs. accuracy gap | 4.4 pp | 4.3 pp | 4.3 pp | 6.0 pp | 4.3 pp | 2.0 pp | 2.1 pp | 3.1 pp | 3.6 pp | 1.9 pp | 6.1 pp | 5.6 pp | 11.8 pp | 11.4 pp | 1.9 pp | 15.4 pp | 11.0 pp | 21.6 pp | 6.0 pp | 3.4 pp | 4.4 pp | 7.2 pp | 4.4 pp | 8.6 pp | 7.0 pp | 6.4 pp | 3.6 pp | 10.6 pp | 6.4 pp | 6.9 pp | 13.0 pp | 12.6 pp | 6.2 pp | 7.8 pp | 4.0 pp |
| Japanese option-order flips | 4.8% | 6.4% | 6.6% | 5.9% | 6.7% | 6.1% | 8.8% | 11.2% | 4.6% | 7.0% | 4.3% | 4.5% | 10.7% | 5.4% | 14.4% | 12.7% | 17.8% | 18.9% | 12.5% | 7.8% | 7.5% | 5.3% | 8.2% | 5.8% | 17.3% | 7.2% | 33.3% | 11.5% | 3.2% | 17.4% | 25.9% | 20.2% | 14.1% | 12.2% | 37.6% |
| Latency | |||||||||||||||||||||||||||||||||||
| Median | 383ms | 530ms | 470ms | 340ms | 472ms | 514ms | 469ms | 466ms | 303ms | 455ms | 376ms | 211ms | 276ms | 613ms | 295ms | 247ms | 669ms | 432ms | 362ms | 433ms | 454ms | 507ms | 343ms | 284ms | 490ms | 351ms | 554ms | 483ms | 253ms | 691ms | 568ms | 397ms | 425ms | 607ms | 596ms |
| Mean | 483ms | 647ms | 546ms | 412ms | 485ms | 581ms | 480ms | 501ms | 325ms | 652ms | 389ms | 225ms | 292ms | 626ms | 314ms | 319ms | 950ms | 4.3s | 400ms | 446ms | 479ms | 577ms | 376ms | 305ms | 563ms | 371ms | 611ms | 566ms | 276ms | 737ms | 567ms | 401ms | 438ms | 629ms | 637ms |
| Within 95% | 816ms | 1.0s | 879ms | 433ms | 527ms | 1.0s | 522ms | 632ms | 476ms | 1.3s | 462ms | 306ms | 399ms | 688ms | 408ms | 623ms | 2.5s | 13.6s | 611ms | 508ms | 611ms | 1.1s | 558ms | 375ms | 930ms | 476ms | 944ms | 939ms | 550ms | 921ms | 800ms | 439ms | 499ms | 791ms | 834ms |
| Within 99% | 1.5s | 3.1s | 1.2s | 900ms | 746ms | 1.6s | 642ms | 1.1s | 593ms | 2.9s | 593ms | 467ms | 559ms | 748ms | 729ms | 1.3s | 2.7s | 16.2s | 993ms | 756ms | 673ms | 1.5s | 616ms | 534ms | 1.3s | 723ms | 1.4s | 1.3s | 848ms | 1.5s | 1.0s | 513ms | 756ms | 897ms | 1.5s |
| Longest response | 6.5s | 9.1s | 2.0s | 19.7s | 2.1s | 3.3s | 1.2s | 4.9s | 6.4s | 5.3s | 1.4s | 586ms | 1.5s | 1.9s | 1.1s | 3.8s | 6.1s | 59.5s | 3.6s | 1.5s | 1.5s | 2.8s | 1.4s | 4.8s | 6.5s | 1.6s | 5.9s | 10.4s | 1.3s | 5.8s | 5.8s | 949ms | 1.6s | 2.2s | 2.4s |
| Cost | |||||||||||||||||||||||||||||||||||
| Per 1,000 | $0.0043 | $0.0045 | $0.014 | $0.0046 | $0.0079 | $0.0079 | $0.0038 | $0.0038 | $0.023 | $0.024 | $0.022 | $0.022 | Free for now | $0.0070 | $0.0092 | $0.031 | $0.0024 | $0.024 | $0.0075 | $0.0086 | $0.0023 | $0.069 | No cost | $0.0021 | $0.0094 | $0.048 | $0.016 | $0.0063 | $0.026 | $0.0066 | $0.024 | Free for now | $0.0024 | $0.0077 | $0.0085 |
| Total measurement cost | $0.014 | $0.015 | $0.046 | $0.015 | $0.026 | $0.026 | $0.013 | $0.013 | $0.078 | $0.082 | $0.073 | $0.073 | $0 | $0.023 | $0.031 | $0.104 | $0.0081 | $0.080 | $0.025 | $0.029 | $0.0076 | $0.229 | $0 | $0.0071 | $0.031 | $0.159 | $0.054 | $0.021 | $0.086 | $0.022 | $0.080 | $0 | $0.0082 | $0.026 | $0.028 |
| Input tokens | 714,686 | 758,106 | 768,130 | 768,126 | 627,785 | 627,785 | 627,785 | 627,785 | 780,833 | 815,916 | 1,727,266 | 1,727,266 | 568,400 | 556,102 | 768,126 | 1,036,252 | 813,472 | 1,601,606 | 627,785 | 815,916 | 758,106 | 956,111 | 722,955 | 714,686 | 522,753 | 1,057,752 | 1,351,012 | 522,753 | 956,111 | 522,753 | 1,601,606 | 627,158 | 815,916 | 647,871 | 706,963 |
Datasets and call routesLanguage, item count and license per set; call route and served version per model
Each version fixes its items, and scores are only comparable within the same version. Adding a model keeps the version; adding a dataset starts the next version.
| Set | Language | Items | License |
|---|---|---|---|
| KoBEST BoolQ | Korean | 100 | CC BY-SA 4.0 |
| KoBEST COPA | Korean | 90 | CC BY-SA 4.0 |
| KoBEST SentiNeg | Korean | 60 | CC BY-SA 4.0 |
| KLUE NLI | Korean | 150 | CC BY-SA 4.0 |
| KLUE STS | Korean | 100 | CC BY-SA 4.0 |
| JGLUE JCommonsenseQA | Japanese | 75 | CC BY-SA 4.0 |
| JGLUE JNLI | Japanese | 100 | CC BY-SA 4.0 |
| JGLUE JSTS | Japanese | 75 | CC BY-SA 4.0 |
| KMMLU | Korean | 125 | CC BY-ND 4.0 |
| JMMLU | Japanese | 125 | CC BY-SA 4.0 / CC BY-NC-ND 4.0 |
| M-RewardBench (한국어) | Korean | 125 | ODC-By 1.0 |
| M-RewardBench (日本語) | Japanese | 125 | ODC-By 1.0 |
| PolyGuardPrompts (한국어) | Korean | 125 | CC BY 4.0 |
| PolyGuardPrompts (日本語) | Japanese | 125 | CC BY 4.0 |
| K-MHaS | Korean | 250 | CC BY-SA 4.0 |
| RTP-LX (한국어) | Korean | 125 | ODC-By 1.0 |
| RTP-LX (日本語) | Japanese | 125 | ODC-By 1.0 |
KMMLU: Son et al., "KMMLU: Measuring Massive Multitask Language Understanding in Korean" (2024, arXiv:2402.11548), HAERAE. JMMLU: Waseda University NLP Lab — Japanese and world history questions © Step, civics, geography and idiom questions © VIST, MMLU translation supported by RIKEN.
| Model | Called via | Version that answered |
|---|---|---|
| Perplexity | pplx-decider-v1.3 | |
| OpenRouter | perplexity/pplx-decider-v1.2-20261016 | |
| OpenRouter | nace-ai/drex-max-20261016 | |
| Perplexity | pplx-decider-v1.1-27b | |
| OpenRouter | liquid/d1.2-ultra-20261116 | |
| OpenRouter | liquid/d1-ultra-20261015 | |
| OpenRouter | liquid/d1.2-20261116 | |
| OpenRouter | liquid/d1.1-20261027 | |
| Gemini API | — | |
| OpenRouter | baseten/mammoth-1-20261021 | |
| OpenRouter | typesafe/jev-1.14-20261014 | |
| OpenRouter | typesafe/jev-1.13-20260917 | |
| OpenRouter | inception/mercury-decide-20260930 | |
| OpenRouter | microsoft/microsoft-decision-1-20261009 | |
| Perplexity | pplx-decider-v1-27b | |
| OpenAI | gpt-6-luna | |
| OpenRouter | respan/span-02-20261015 | |
| OpenRouter | upstage/solar-decide-20260928 | |
| OpenRouter | liquid/d1-20260930 | |
| OpenRouter | baseten/elephant-1-20261021 | |
| OpenRouter | perplexity/pplx-decider-flash-v1.2-20261016 | |
| Cloudflare Workers AI | clef | |
| Self-hosted GPU | kakaocorp/kanana-2-v-decide-3b-2611 | |
| Perplexity | pplx-decider-flash-v1.3 | |
| OpenRouter | nace-ai/drex-v2.0-20261009 | |
| OpenRouter | cloudflare/clef-omni-20261009 | |
| OpenRouter | scaledown/classify-1-20260928 | |
| OpenRouter | nace-ai/drex-v1.5-20261009 | |
| Cloudflare Workers AI | clef-flash | |
| JJared PalmerKev 4B | OpenRouter | jaredpalmer/kev-4b-20260924 |
| OpenRouter | upstage/solar-decide-flash-20261008 | |
| Alibaba Cloud Model Studio | decision-model-preview | |
| OpenRouter | baseten/joey-1-20261029 | |
| meraGPT | decider-1 | |
| meraGPT | state-decider-1 |
ReproducibilitySeed, item fingerprints (SHA-256) and source revisions
Items were stratified-sampled from each dataset with seed 20261003. Each set's item fingerprint (SHA-256) lets you check that the items are the same; the source revision is listed only for sets where it was recorded. Contact us if you need the list of source rows for each set. Contact
| Set | Item fingerprint | Source revision |
|---|---|---|
| kobest-boolq | 323717fbfaaa2fc575f3adb5e8fb0983f2b80ee9b45402b70fdf2afefcbbeaa2 | — |
| kobest-copa | afa227de3bce46f5013ffa5c4887d74b929c68a97c2d7da732f3819c6a69b75c | — |
| kobest-sentineg | 4af55bf4bbca356dd5f996b367d133a82173aad2fa87ab77cebb4a586df20ebb | — |
| klue-nli | 78a9c69705d487232845ec18f4c050f7833035b16fd07b7e755aba3ce19ad5cd | — |
| klue-sts | c979a7d5258d8d25bf7605bf1f895366e2a1934f7a5b1fb6350921fadd8e0a26 | — |
| jglue-jcommonsenseqa | cb87b85c4b6c94be5e0ae5081bd8ae99603769c7f8ea7a367697a37a5f489ab4 | — |
| jglue-jnli | e812223e4ac8b7e3de15a2298da954a1e78915f9e46f6b7b2e54d335d4c2eef6 | — |
| jglue-jsts | 8161627fe3f72e5cd5b60a41c24a3dd5e2e7c76abbaa66d509fff54bf2f4d5a9 | — |
| kmmlu | 575b8ade0493fa4fefdf3795db56d8dbf58e746336e1726f196a4a9e491d6c2e | d61b3f19e552c576bf5960dd24289763edc36a88 |
| jmmlu | 0f4f97a840d486d563e55872720874da8bebe9b8144bd1139e014cbb53cbd46e | 762cbf192c9e4588b574d718f95afd64c96fcbf4 |
| mrewardbench-ko | a3582c37dfeaa40b5441d7211ca99f24078ed3919d4771c9994d1da8d8f91a42 | — |
| mrewardbench-ja | aad4a04ac56d89705d4df59e7b7a6d891beed79b9c40aba08cbe478b5a22a6e7 | — |
| polyguard-ko | 5817f4bf61aa1fd9a377af67f4d6a5f68dc5eaff0c407c8c49931c8c0e979c7e | — |
| polyguard-ja | 47af613e0d45cee908937aecb151279602568a9b5e635b8bbd5dae871ead04f4 | — |
| kmhas | 4cbb45e028936cd7db9099bdb5cfb46d2dc88f18397b876a6839e48977c56266 | — |
| rtplx-ko | a95ac77ce6df6df3b57d922e9e471597111db8be8bd2ac726bd663ed6d9d86bb | — |
| rtplx-ja | cef1b894731fe06343005940188578dfe98d3a75513d0138cff32757aa4ed0d2 | — |
NotesExcluded models, models still being measured, contamination
- Left out of the ranking: TTev1 4B Experimental. The provider's terms restrict publishing benchmark results.
- Items from public datasets may already be in models' training data. Some providers' terms also allow them to use submitted inputs to improve their models.
Frequently asked questions
Q.What is a decision model, and how is it different from an LLM like ChatGPT?
A.An AI model that returns probabilities for a question instead of writing text. It answers in three forms — yes/no, picking an option, or a scale score — and each answer comes with a number for how confident it is. Because it is far faster and cheaper than a text-generating LLM, it is used for judgments repeated at volume, such as sorting customer inquiries, filtering harmful content, or checking AI answers.
Q.Which decision model is most accurate in Korean and Japanese?
A.As of November 20, 2026, Decider V1.3 ranks first overall with 82.4 points. On Korean items the leader is Decider V1.3 (82.6 points); on Japanese items it is Mammoth 1 (80.5 points). The score is the equal-weight average of accuracy across 8 datasets, out of 100.
Q.Are decision models as accurate as general-purpose LLMs like GPT or Claude?
A.In the November 20, 2026 run, on the same 2,000 items the general-purpose LLMs scored GPT-6.1 Sol 83.9 pts and Claude Sonnet 5.5 80.9 pts, and the top decision model, Decider V1.3, scored 82.4. The most accurate LLM leads by 1.5 points. In exchange, the top decision model answers about 4× faster on average than GPT-6.1 Sol and costs about 1/192 as much per 1,000 items. On Japanese items alone, the decision model Mammoth 1 scored 80.5, ahead of the general-purpose LLMs (GPT-6.1 Sol 78.6 pts and Claude Sonnet 5.5 73.9 pts).
Q.What tasks are decision models good for?
A.On par with general-purpose LLMs (top decision model ahead or within 2 points): Checking AI answers, Checking claims against a source, Blocking harmful requests, Catching hate speech, Grading toxicity, Finding similar sentences, Commonsense and cause-effect, Yes/no answers from a document, and Classifying sentiment. Still well behind, where an LLM is more accurate: Checking expert knowledge.
Q.How fast and how cheap are decision models?
A.Among fully measured models, median latency ranged from 211ms to 691ms, and judging 1,000 items cost $0.0021 to $0.069. None of the measured models charges for output tokens, so input length sets the cost.
Q.Can you trust the probabilities (confidence) a decision model returns?
A.It depends on the model. Gemini Decision 1 has the smallest gap between confidence and actual accuracy, 1.4 points on average, so an answer given at 80% confidence is right close to 80% of the time — you can use the probability as a threshold to split automatic handling from human review. Solar Decide has the largest gap, 18.1 points on average, and is overconfident, so calibrate your own threshold before relying on it.
Q.How is this benchmark measured?
A.We drew 2,000 items across 8 datasets from Korean and Japanese public datasets and asked every model the same items. Items with options are asked again with the options reversed, to see whether the order sways the answer. The score is the equal-weight average of per-dataset accuracy, out of 100. As a reference, the general-purpose LLMs GPT-6.1 Sol and Claude Sonnet 5.5 answered the same items too (lowest reasoning setting, not ranked).
Q.Why are the items not published?
A.Publishing them would let future models train on them and inflate their scores. Instead we publish each set's item fingerprint (SHA-256), the sampling seed and the source revision so anyone can check the items are the same, and we share the list of source rows on request.
Curious how other models perform?
If there's a decision model you'd like added to the benchmark, let us know. We'll review it and include it in the next measurement.