SWE-Bench Pro
Citation frequency 3/3
- What does it measure?
- Tests whether an AI can fix long software tasks taken from real open-source code repositories. Made by Scale AI.
- Who evaluates it, and how?
- A task passes when the AI's code changes pass hidden tests. Per the paper there are 1,865 tasks in total, of which the public set has 731.
- Reading the score
- Scores run from 0 to 100%. Higher is better. Makers use different task sets and tools, so read each score together with its setting.
This ranking is built only from scores the model makers published themselves. Test settings such as tool use and reasoning effort differ by maker, so check each score's setting before comparing models directly.
Top score97.7%
Models tested53
Scoring basisHigher is better
Last updated2026.09.22
Model release date
Can AI carry out long development tasks autonomously by operating a computer itself?Terminal-Bench 4.0
Kimi K3.1 Pro🥇
Kimi K3.1 Pro🥇Terminal-Bench 4.0
Kimi K3.1 Pro🥇Which AI do people prefer most for coding answers?Arena Coding
Gemini 4 Carbon🥇
Gemini 4 Carbon🥇Arena Coding
Gemini 4 Carbon🥇Can AI test and fix code itself until it completes the given task?LiveBench Agentic Coding v1
DeepSeek V4.1 Flash🥇
DeepSeek V4.1 Flash🥇LiveBench Agentic Coding v1
DeepSeek V4.1 Flash🥇
Can it fix real issues in real software projects?
SWE-Bench Pro
- 🥇InklingInkling 2 Small97.7%
- 🥈AnthropicClaude Opus 5.5Reasoning effort: Max89.9%
- 🥉AnthropicClaude Fable 5.1Reasoning effort: Max81.2%
- #4AnthropicClaude Fable 5Reasoning effort: Max80.0%
- #5AnthropicClaude Opus 5Reasoning effort: Max79.2%
- #6AnthropicClaude Opus 4.8Reasoning effort: Max69.2%
- #7TencentHy4Reasoning effort: Max65.7%
- #8SpaceXSIGrok 4.5Reasoning effort: High64.7%
- #9OpenAIGPT-5.6 Sol64.6%
- #10AnthropicClaude Opus 4.7Reasoning effort: Max64.3%
- #11OpenAIGPT-5.6 Terra63.4%
- #12AnthropicClaude Sonnet 5Reasoning effort: Max63.2%
- #13OpenAIGPT-5.6 Luna62.7%
- #14AlibabaQwen3.8 Flash62.5%
- #15Z.aiGLM 5.262.1%
- #16MetaMuse Spark 1.1Reasoning effort: Extra High61.5%
- #17AlibabaQwen3.7 Max60.6%
- #18MeituanLongCat 2.059.5%
- #19MiniMaxMiniMax M359.0%
- #20GoogleGemini 3.6 Flash58.7%
- #21OpenAIGPT-5.5Reasoning effort: Extra High58.6%
- #21Moonshot AIKimi K2.6Reasoning effort: Thinking58.6%
- #23Z.aiGLM-5.158.4%
- #24AnthropicClaude Sonnet 4.658.1%
- #25OpenAIGPT-5.4Reasoning effort: Extra High57.7%
- #26AlibabaQwen3.7 Plus57.6%
- #27AlibabaQwen3.6 Max57.3%
- #28XiaomiMiMo V2.5 Pro57.2%
- #29AlibabaQwen3.6 Plus56.6%
- #30StepfunStep 3.7 Flash56.3%
- #31MiniMaxMiniMax M2.756.2%
- #32XiaomiMiMo V2.556.1%
- #33MiniMaxMiniMax M2.555.4%
- #33DeepSeekDeepSeek V4 ProReasoning effort: Max55.4%
- #35GoogleGemini 3.5 Flash55.1%
- #35Z.aiGLM-555.1%
- #37XiaomiMiMo V2 Pro55.0%
- #37MetaMuse Spark55.0%
- #39OpenAIGPT-5.4 MiniReasoning effort: Extra High54.4%
- #40GoogleGemini 3.5 Flash-LiteReasoning effort: High54.2%
- #40GoogleGemini 3.1 ProReasoning effort: High54.2%
- #42AnthropicClaude Opus 4.653.4%
- #43DeepSeekDeepSeek V4 FlashReasoning effort: Max52.6%
- #44OpenAIGPT-5.4 NanoReasoning effort: Extra High52.4%
- #45AnthropicClaude Opus 4.5Reasoning effort: None52.0%
- #46Moonshot AIKimi K2.550.7%
- #47GoogleGemini 3 Flash49.6%
- #48AlibabaQwen3.6 Flash49.5%
- #48AlibabaQwen3.6 35B A3B49.5%
- #50PoolsideLaguna M.149.2%
- #51ByteDanceDola Seed 2.0 Pro46.9%
- #52CohereNorth Mini Code40.2%
- #53GoogleGemini 3.1 Flash Lite38.3%
Half of models ≤ 57.3%Measured by: Maker-reported
The "harness" is the agent program the AI used to carry out the task. The same model can score very differently depending on its harness and reasoning effort.