DeepSWE

Citation frequency 3/3
What does it measure?
A test by Datacurve of whether an AI coding agent can finish long, complex software tasks written from scratch.
Who evaluates it, and how?
113 tasks written for 91 active open-source repositories in five languages, solved in a sandbox without internet access. A hand-written verifier checks each task's behavior, and the whole suite is run four times and averaged.
Reading the score
Scores run from 0 to 100%. Higher is better.
Top score77.9%
Models tested28
Scoring basisHigher is better
Last updated2026.09.30
Model release date

Can it finish long development tasks on its own?

DeepSWE

  1. 🥇GoogleGemini 4 ArgonReasoning effort: Max77.9%
  2. 🥈OpenAIGPT-6 AstraReasoning effort: Extra High74.1%
  3. 🥉GoogleGemini 3.8 FlashReasoning effort: High73.8%
  4. #4MiniMaxMiniMax M3-1 Flash73.7%
  5. #5AnthropicClaude Opus 5Reasoning effort: Max73.6%
  6. #6OpenAIGPT-5.6 SolReasoning effort: Max72.7%
  7. #7InklingInkling 2 Small72.1%
  8. #8AnthropicClaude Fable 5Reasoning effort: Extra High69.9%
  9. #9OpenAIGPT-5.6 TerraReasoning effort: Max69.6%
  10. #10Z.aiGLM 5.3Reasoning effort: Max69.0%
  11. #11Moonshot AIKimi K3Reasoning effort: Max68.5%
  12. #12SpaceXSIGrok 4.6Reasoning effort: Medium67.5%
  13. #13OpenAIGPT-5.6 LunaReasoning effort: Max67.2%
  14. #14OpenAIGPT-5.5Reasoning effort: Extra High67.0%
  15. #15GoogleGemini 3.7 FlashReasoning effort: Medium65.5%
  16. #16Z.aiGLM 5.3 FlashReasoning effort: Max63.4%
  17. #17AnthropicClaude Opus 4.8Reasoning effort: Max59.0%
  18. #18MetaMuse Spark 1.2Reasoning effort: Extra High54.9%
  19. #19AnthropicClaude Sonnet 5Reasoning effort: Max53.8%
  20. #20SpaceXSIGrok 4.5Reasoning effort: High53.8%
  21. #21MetaMuse Spark 1.1Reasoning effort: Extra High53.3%
  22. #22OpenAIGPT-5.4Reasoning effort: Extra High51.8%
  23. #23GoogleGemini 3.6 FlashReasoning effort: High46.7%
  24. #24Z.aiGLM 5.2Reasoning effort: Max43.8%
  25. #25GoogleGemini 3.5 FlashReasoning effort: Medium37.4%
  26. #26Moonshot AIKimi K2.7 Code30.5%
  27. #27AnthropicClaude Sonnet 4.6Reasoning effort: High29.9%
  28. #28GoogleGemini 3.1 ProReasoning effort: High11.7%
Half of models ≤ 66.3%

The "harness" is the agent program the AI used to carry out the task. The same model can score very differently depending on its harness and reasoning effort.