Fireworks Evaluates Kimi K3.1 Pro for Agents
- •Fireworks AI’s evaluation found Kimi K3.1 Pro beat Kimi K3 on 14 of 17 shared benchmarks, with regressions in two areas.
- •K3.1 Pro cut web-research cost 72% versus Opus, but banking resolutions cost 89% more.
- •Routing simulations found model pairings could improve scores; the Astra pairing’s advantage held in 98% of resamples.
Fireworks AI’s November 17, 2026 evaluation found that Kimi K3.1 Pro outscored Kimi K3 on 14 of 17 shared benchmarks, but regressed on memory creation and contract review. The study compared quality, model cost and execution time using 21 benchmark leaderboards and closer task-level analysis of banking support, corporate research, financial research and legacy-code maintenance. In routing simulations across 461 tasks, pairing K3.1 Pro with Claude Opus 5.5 cut model spending by more than half; pairings with GPT-6 Astra and GPT-6.1 Sol scored higher than either model alone.
On Factory’s Legacy-Bench, K3.1 Pro completed 62.5% of 97 tasks, versus K3’s 54.3%, a 15% increase in completed tasks, while costing 48% less. In Sierra’s banking-support benchmark, unrequested fields in account-changing tool calls fell from 8.5% to 2.6%, about 69% lower. In one example involving three bank cards, K3 closed all three, while K3.1 Pro froze them, checked transactions and closed only the card that required closure. Memory creation was 22% lower than K3: K3.1 Pro assigned about 50–65% of available records to workers, while K3 assigned the full set in two of three runs. On the separate LAB: Contracts benchmark, K3.1 Pro’s score fell 64%, from 8.3 to 3.0, despite its overall Legal Agent Benchmark score rising 18%, from 13.1 to 15.4.
For web research, K3.1 Pro scored 74.2 against Claude Opus 5.5’s 75.4, at 72% lower cost ($0.27 versus $0.95 per question). On financial research, it matched Opus at 48.2% for 81% less ($0.17 versus $0.91 per question), while GPT-6.1 Sol scored 52.0% at about $0.05 per question, roughly 69% less than K3.1 Pro. In corporate research, K3.1 Pro scored 78.4 versus Opus’s 75.4 at 83% lower cost ($0.47 versus $2.80 per task); GPT-6.1 Sol scored 82.2 at $0.41 per task. Yet on 16 employee-history tasks, K3.1 Pro averaged 78 out of 100, below GPT-6 Astra’s 96 and Sol’s 93.
In banking support, K3.1 Pro resolved 63.3% of attempts, versus Opus’s 56.0%, including an average of five of 12 especially difficult conversations that at least 15 of 20 models failed to resolve in any run. It made 22 knowledge-base lookups per conversation on average, versus 10–17 for other analyzed models; the logs do not establish that the extra lookups caused its advantage. K3.1 Pro used roughly three times Opus’s input tokens per conversation and cost $1.51 per successful resolution versus $0.80, about 89% more; execution time per attempt was more than twice as long. In 256 software tasks attempted three times each, Surge AI found that in 64% of attempts K3.1 Pro identified a risk but neither addressed nor reported it. Its sudo L7 pass rate was 16.7%, versus 45.6% for Claude Opus 5.5 and 50.6% for Claude Sonnet 5.5; its rate was 37% higher than K3’s. On incident diagnosis, it found the exact cause in 32.9% of cases, compared with 44.3% for Claude Opus 5 and 50.2% for GPT-6 Astra.
The routing simulation paired K3.1 Pro separately with Opus 5.5, GPT-6 Astra and GPT-6.1 Sol, using a router to assign each of 461 tasks to one model. It balanced scores and spending using a hypothetical $20 value per score point. The Astra and Sol pairings scored higher than either model alone; that advantage held in 98% and 89% of 2,000 resamples, respectively. The Opus pairing beat Opus alone in 99% of resamples and cost less in all of them, but beat K3.1 Pro alone in only 55%. On CorporateBench, the Opus pairing scored about 100 out of 128, 11 points above Opus alone, while reducing spending from $358 to $100. On Legacy-Bench, the Astra pairing completed about five more tasks while cutting spending about 82%: 61 tasks for $43 versus 56 for $235 using Astra alone. The authors note that simulations used recorded results and known task categories; a production router would need to obtain or predict categories, and the study did not test adding K3.1 Pro to a larger multi-model ensemble.
The fastest, most reliable AI news, straight to your inbox