SWE-Bench Pro

Citation frequency 3/3
What does it measure?
Tests whether an AI can fix long software tasks taken from real open-source code repositories. Made by Scale AI.
Who evaluates it, and how?
A task passes when the AI's code changes pass hidden tests. Per the paper there are 1,865 tasks in total, of which the public set has 731.
Reading the score
Scores run from 0 to 100%. Higher is better. Makers use different task sets and tools, so read each score together with its setting.

This ranking is built only from scores the model makers published themselves. Test settings such as tool use and reasoning effort differ by maker, so check each score's setting before comparing models directly.

Top score97.7%
Models tested53
Scoring basisHigher is better
Last updated2026.09.22
Model release date

Can it fix real issues in real software projects?

SWE-Bench Pro

  1. 🥇InklingInkling 2 Small97.7%
  2. 🥈AnthropicClaude Opus 5.5Reasoning effort: Max89.9%
  3. 🥉AnthropicClaude Fable 5.1Reasoning effort: Max81.2%
  4. #4AnthropicClaude Fable 5Reasoning effort: Max80.0%
  5. #5AnthropicClaude Opus 5Reasoning effort: Max79.2%
  6. #6AnthropicClaude Opus 4.8Reasoning effort: Max69.2%
  7. #7TencentHy4Reasoning effort: Max65.7%
  8. #8SpaceXSIGrok 4.5Reasoning effort: High64.7%
  9. #9OpenAIGPT-5.6 Sol64.6%
  10. #10AnthropicClaude Opus 4.7Reasoning effort: Max64.3%
  11. #11OpenAIGPT-5.6 Terra63.4%
  12. #12AnthropicClaude Sonnet 5Reasoning effort: Max63.2%
  13. #13OpenAIGPT-5.6 Luna62.7%
  14. #14AlibabaQwen3.8 Flash62.5%
  15. #15Z.aiGLM 5.262.1%
  16. #16MetaMuse Spark 1.1Reasoning effort: Extra High61.5%
  17. #17AlibabaQwen3.7 Max60.6%
  18. #18MeituanLongCat 2.059.5%
  19. #19MiniMaxMiniMax M359.0%
  20. #20GoogleGemini 3.6 Flash58.7%
  21. #21OpenAIGPT-5.5Reasoning effort: Extra High58.6%
  22. #21Moonshot AIKimi K2.6Reasoning effort: Thinking58.6%
  23. #23Z.aiGLM-5.158.4%
  24. #24AnthropicClaude Sonnet 4.658.1%
  25. #25OpenAIGPT-5.4Reasoning effort: Extra High57.7%
  26. #26AlibabaQwen3.7 Plus57.6%
  27. #27AlibabaQwen3.6 Max57.3%
  28. #28XiaomiMiMo V2.5 Pro57.2%
  29. #29AlibabaQwen3.6 Plus56.6%
  30. #30StepfunStep 3.7 Flash56.3%
  31. #31MiniMaxMiniMax M2.756.2%
  32. #32XiaomiMiMo V2.556.1%
  33. #33MiniMaxMiniMax M2.555.4%
  34. #33DeepSeekDeepSeek V4 ProReasoning effort: Max55.4%
  35. #35GoogleGemini 3.5 Flash55.1%
  36. #35Z.aiGLM-555.1%
  37. #37XiaomiMiMo V2 Pro55.0%
  38. #37MetaMuse Spark55.0%
  39. #39OpenAIGPT-5.4 MiniReasoning effort: Extra High54.4%
  40. #40GoogleGemini 3.5 Flash-LiteReasoning effort: High54.2%
  41. #40GoogleGemini 3.1 ProReasoning effort: High54.2%
  42. #42AnthropicClaude Opus 4.653.4%
  43. #43DeepSeekDeepSeek V4 FlashReasoning effort: Max52.6%
  44. #44OpenAIGPT-5.4 NanoReasoning effort: Extra High52.4%
  45. #45AnthropicClaude Opus 4.5Reasoning effort: None52.0%
  46. #46Moonshot AIKimi K2.550.7%
  47. #47GoogleGemini 3 Flash49.6%
  48. #48AlibabaQwen3.6 Flash49.5%
  49. #48AlibabaQwen3.6 35B A3B49.5%
  50. #50PoolsideLaguna M.149.2%
  51. #51ByteDanceDola Seed 2.0 Pro46.9%
  52. #52CohereNorth Mini Code40.2%
  53. #53GoogleGemini 3.1 Flash Lite38.3%
Half of models ≤ 57.3%Measured by: Maker-reported

The "harness" is the agent program the AI used to carry out the task. The same model can score very differently depending on its harness and reasoning effort.

  • Self-reported via Hugging Face model reposEvaluation results that model makers posted on their own Hugging Face model pages (only those accepted and merged). Conditions differ by maker.
  • Self-reported by each model maker — see the source link on each scoreFigures that model makers announced themselves, checked against the original post or system card and copied over. Conditions differ by maker, and each score links to its original source.