DeepSWE
Citation frequency 3/3
- What does it measure?
- A test by Datacurve of whether an AI coding agent can finish long, complex software tasks written from scratch.
- Who evaluates it, and how?
- 113 tasks written for 91 active open-source repositories in five languages, solved in a sandbox without internet access. A hand-written verifier checks each task's behavior, and the whole suite is run four times and averaged.
- Reading the score
- Scores run from 0 to 100%. Higher is better.
Top score77.9%
Models tested28
Scoring basisHigher is better
Last updated2026.09.30
Model release date
Can AI carry out long development tasks autonomously by operating a computer itself?Terminal-Bench 4.0
Kimi K3.1 Pro🥇
Kimi K3.1 Pro🥇Terminal-Bench 4.0
Kimi K3.1 Pro🥇Which AI do people prefer most for coding answers?Arena Coding
Gemini 4 Carbon🥇
Gemini 4 Carbon🥇Arena Coding
Gemini 4 Carbon🥇Can AI test and fix code itself until it completes the given task?LiveBench Agentic Coding v1
DeepSeek V4.1 Flash🥇
DeepSeek V4.1 Flash🥇LiveBench Agentic Coding v1
DeepSeek V4.1 Flash🥇
Can it finish long development tasks on its own?
DeepSWE
- 🥇GoogleGemini 4 ArgonReasoning effort: Max77.9%
- 🥈OpenAIGPT-6 AstraReasoning effort: Extra High74.1%
- 🥉GoogleGemini 3.8 FlashReasoning effort: High73.8%
- #4MiniMaxMiniMax M3-1 Flash73.7%
- #5AnthropicClaude Opus 5Reasoning effort: Max73.6%
- #6OpenAIGPT-5.6 SolReasoning effort: Max72.7%
- #7InklingInkling 2 Small72.1%
- #8AnthropicClaude Fable 5Reasoning effort: Extra High69.9%
- #9OpenAIGPT-5.6 TerraReasoning effort: Max69.6%
- #10Z.aiGLM 5.3Reasoning effort: Max69.0%
- #11Moonshot AIKimi K3Reasoning effort: Max68.5%
- #12SpaceXSIGrok 4.6Reasoning effort: Medium67.5%
- #13OpenAIGPT-5.6 LunaReasoning effort: Max67.2%
- #14OpenAIGPT-5.5Reasoning effort: Extra High67.0%
- #15GoogleGemini 3.7 FlashReasoning effort: Medium65.5%
- #16Z.aiGLM 5.3 FlashReasoning effort: Max63.4%
- #17AnthropicClaude Opus 4.8Reasoning effort: Max59.0%
- #18MetaMuse Spark 1.2Reasoning effort: Extra High54.9%
- #19AnthropicClaude Sonnet 5Reasoning effort: Max53.8%
- #20SpaceXSIGrok 4.5Reasoning effort: High53.8%
- #21MetaMuse Spark 1.1Reasoning effort: Extra High53.3%
- #22OpenAIGPT-5.4Reasoning effort: Extra High51.8%
- #23GoogleGemini 3.6 FlashReasoning effort: High46.7%
- #24Z.aiGLM 5.2Reasoning effort: Max43.8%
- #25GoogleGemini 3.5 FlashReasoning effort: Medium37.4%
- #26Moonshot AIKimi K2.7 Code30.5%
- #27AnthropicClaude Sonnet 4.6Reasoning effort: High29.9%
- #28GoogleGemini 3.1 ProReasoning effort: High11.7%
Half of models ≤ 66.3%
The "harness" is the agent program the AI used to carry out the task. The same model can score very differently depending on its harness and reasoning effort.