Skip to content

SWE-bench Pro

A long-horizon repository benchmark built to test realistic software engineering work. Its scores need a task-quality and setup check before they support a coding-agent decision.

69 models

Higher is better.

  1. 1Claude Opus 5.5Anthropic89.9
  2. 2Claude Sonnet 5.5Anthropic81.3
  3. 3Claude Fable 5.1Anthropic81.2
  4. 4Claude Mythos 5Anthropic80.3
  5. 5Claude Fable 5Anthropic80.0
  6. 6Claude Opus 5Anthropic79.2
  7. 7Claude Opus 4.8Anthropic69.2
  8. 8Ling 3.1 FlashInclusionAI68.7
  9. 9Qwen3.8 MaxAlibaba67.7
  10. 10Beam 1.0 501B-A23BReflection AI66.1
  11. 11Hy4 previewTencent65.7
  12. 12Ornith-1.5-397BOrnith AI65.1
  13. 13Grok 4.5xAI64.7
  14. 14GPT-5.6 SolOpenAI64.6
  15. 15Claude Opus 4.7 (Adaptive)Anthropic64.3
  16. 16GPT-5.6 TerraOpenAI63.4
  17. 17Qwen3.8-Omni-FlashAlibaba63.3
  18. 18Claude Sonnet 5Anthropic63.2
  19. 19GPT-5.6 LunaOpenAI62.7
  20. 20Qwen3.8-Flash-NextAlibaba62.5
  21. 21Ornith-1.0-397BDeepReinforce AI62.2
  22. 22GLM-5.2Z.AI62.1
  23. 23Qwen3.8-27BAlibaba61.7
  24. 24Muse Spark 1.1Meta61.5
  25. 25dots3-note PreviewDots Studio61.0
  26. 26Qwen3.7 MaxAlibaba60.6
  27. 27Ornith-1.5-35B-A3BOrnith AI59.6
  28. 28Atria Dawn PreviewShanghai Artificial Intelligence Laboratory59.6
  29. 29Laguna S 2.1Poolside59.4
  30. 30MiniMax M3MiniMax59.0
  31. 31GPT-5.5OpenAI58.6
  32. 32Kimi K2.6Moonshot AI58.6
  33. 33GLM-5.1Z.AI58.4
  34. 34GPT-5.4OpenAI57.7
  35. 35Qwen3.7 PlusAlibaba57.6
  36. 36Qwen 3.6 Max (preview)Alibaba57.3
  37. 37MiMo-V2.5-ProXiaomi57.2
  38. 38Claude Opus 4.5Anthropic57.1
  39. 39GPT-5.3 CodexOpenAI56.8
  40. 40Qwen3.6 PlusAlibaba56.6
Higher is better. Showing the top 40 of 69. Bar colour: blue proprietary, orange open weights, green other.

About this benchmark

Metric
Repository task completion
Tasks
1,865 repository problems
Authors
Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, Brad Kenstler (2025)

Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.

All results

1Claude Opus 5.5 Anthropic89.9
2Claude Sonnet 5.5 Anthropic81.3
3Claude Fable 5.1 Anthropic81.2
4Claude Mythos 5 Anthropic80.3
5Claude Fable 5 Anthropic80.0
6Claude Opus 5 Anthropic79.2
7Claude Opus 4.8 Anthropic69.2
8Ling 3.1 Flash InclusionAI68.7
9Qwen3.8 Max Alibaba67.7
10Beam 1.0 501B-A23B Reflection AI66.1
11Hy4 preview Tencent65.7
12Ornith-1.5-397B Ornith AI65.1
13Grok 4.5 xAI64.7
14GPT-5.6 Sol OpenAI64.6
15Claude Opus 4.7 (Adaptive) Anthropic64.3
16GPT-5.6 Terra OpenAI63.4
17Qwen3.8-Omni-Flash Alibaba63.3
18Claude Sonnet 5 Anthropic63.2
19GPT-5.6 Luna OpenAI62.7
20Qwen3.8-Flash-Next Alibaba62.5
21Ornith-1.0-397B DeepReinforce AI62.2
22GLM-5.2 Z.AI62.1
23Qwen3.8-27B Alibaba61.7
24Muse Spark 1.1 Meta61.5
25dots3-note Preview Dots Studio61.0
26Qwen3.7 Max Alibaba60.6
27Ornith-1.5-35B-A3B Ornith AI59.6
28Atria Dawn Preview Shanghai Artificial Intelligence Laboratory59.6
29Laguna S 2.1 Poolside59.4
30MiniMax M3 MiniMax59.0
31GPT-5.5 OpenAI58.6
32Kimi K2.6 Moonshot AI58.6
33GLM-5.1 Z.AI58.4
34GPT-5.4 OpenAI57.7
35Qwen3.7 Plus Alibaba57.6
36Qwen 3.6 Max (preview) Alibaba57.3
37MiMo-V2.5-Pro Xiaomi57.2
38Claude Opus 4.5 Anthropic57.1
39GPT-5.3 Codex OpenAI56.8
40Qwen3.6 Plus Alibaba56.6
41Ling 3.0 Flash InclusionAI56.6
42Step 3.7 Flash StepFun56.3
43MiniMax M2.7 MiniMax56.2
44MiMo-V2.5 Xiaomi56.1
45Inkling-Small Thinking Machines Lab55.9
46GPT-5.2 OpenAI55.6
47DeepSeek V4 Pro 0813 DeepSeek55.4
48Gemini 3.5 Flash Google55.1
49GLM-5 Z.AI55.1
50Inkling Thinking Machines Lab54.3
51Gemini 3.5 Flash-Lite Google54.2
52Qwen3.6-27B Alibaba53.5
53Claude Opus 4.6 Anthropic53.4
54MAI-Thinking-1 Microsoft52.8
55DeepSeek V4 Flash 0731 DeepSeek52.6
56Muse Spark Meta52.4
57Grok 4.20 xAI51.8
58Nemotron 3.5 Super VL 120B A12B NVIDIA51.8
59Muse Glimmer 30B Meta51.2
60Qwen3.5 397B Alibaba50.9
61Kimi K2.5 Moonshot AI50.7
62Qwen3.6-35B-A3B Alibaba49.5
63Laguna M.1 Poolside49.2
64Ornith-1.5-9B Ornith AI47.5
65LongCat-Flash-Lite-Sparse Meituan40.6
66LLaDA2.2-flash InclusionAI30.1
67Mellum2.1-12B-A2.5B-Thinking JetBrains28.0
68Granite 4.2 8B IBM19.1
69MiniCPM5-2B OpenBMB14.4