Skip to content

Software Engineering Benchmark Verified

SWE-bench Verified tests patches for 500 human-filtered GitHub issues. A score measures how many tasks an evaluated model-and-agent setup resolved; the model name alone does not identify that setup.

98 models

Higher is better.

  1. 1Claude Opus 5Anthropic96.0
  2. 2Claude Mythos 5Anthropic95.5
  3. 3Claude Fable 5Anthropic95.0
  4. 4Ember-1Fireworks92.2
  5. 5Claude Opus 4.8Anthropic88.6
  6. 6Claude Opus 4.7 (Adaptive)Anthropic87.6
  7. 7Ornith-1.5-397BOrnith AI86.0
  8. 8Claude Sonnet 5Anthropic85.2
  9. 9GPT-5.3 CodexOpenAI85.0
  10. 10Ornith-1.0-397BDeepReinforce AI82.4
  11. 11Beam 1.0 501B-A23BReflection AI81.6
  12. 12Claude Opus 4.5Anthropic80.9
  13. 13Claude Opus 4.6Anthropic80.8
  14. 14DeepSeek V4 Pro 0813DeepSeek80.6
  15. 15MiniMax M3MiniMax80.5
  16. 16Qwen3.7 MaxAlibaba80.4
  17. 17Kimi K2.6Moonshot AI80.2
  18. 18Inkling-SmallThinking Machines Lab80.2
  19. 19GPT-5.2OpenAI80.0
  20. 20Claude Sonnet 4.6Anthropic79.6
  21. 21Ornith-1.5-35B-A3BOrnith AI79.0
  22. 22DeepSeek V4 Flash 0731DeepSeek79.0
  23. 23Qwen3.6 PlusAlibaba78.8
  24. 24dots3-note PreviewDots Studio78.4
  25. 25BTL-4Bad Theory Labs78.4
  26. 26MiMo-V2-ProXiaomi78.0
  27. 27GLM-5Z.AI77.8
  28. 28Qwen3.7 PlusAlibaba77.7
  29. 29Apodex 1.1Apodex77.7
  30. 30InklingThinking Machines Lab77.6
  31. 31Mistral Medium 3.5 128BMistral77.6
  32. 32Muse SparkMeta77.4
  33. 33Qwen3.6-27BAlibaba77.2
  34. 34Claude Sonnet 4.5Anthropic77.2
  35. 35Kimi K2.5 (Reasoning)Moonshot AI76.8
  36. 36Kimi K2.5Moonshot AI76.8
  37. 37Grok 4.20xAI76.7
  38. 38Qwen3.5 397BAlibaba76.2
  39. 39Muse Glimmer 30BMeta76.0
  40. 40MiMo-V2-OmniXiaomi74.8
Higher is better. Showing the top 40 of 98. Bar colour: blue proprietary, orange open weights, green other.

About this benchmark

Metric
Code patch generation
Tasks
500 verified issues
Authors
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan (2024)

Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.

All results

1Claude Opus 5 Anthropic96.0
2Claude Mythos 5 Anthropic95.5
3Claude Fable 5 Anthropic95.0
4Ember-1 Fireworks92.2
5Claude Opus 4.8 Anthropic88.6
6Claude Opus 4.7 (Adaptive) Anthropic87.6
7Ornith-1.5-397B Ornith AI86.0
8Claude Sonnet 5 Anthropic85.2
9GPT-5.3 Codex OpenAI85.0
10Ornith-1.0-397B DeepReinforce AI82.4
11Beam 1.0 501B-A23B Reflection AI81.6
12Claude Opus 4.5 Anthropic80.9
13Claude Opus 4.6 Anthropic80.8
14DeepSeek V4 Pro 0813 DeepSeek80.6
15MiniMax M3 MiniMax80.5
16Qwen3.7 Max Alibaba80.4
17Kimi K2.6 Moonshot AI80.2
18Inkling-Small Thinking Machines Lab80.2
19GPT-5.2 OpenAI80.0
20Claude Sonnet 4.6 Anthropic79.6
21Ornith-1.5-35B-A3B Ornith AI79.0
22DeepSeek V4 Flash 0731 DeepSeek79.0
23Qwen3.6 Plus Alibaba78.8
24dots3-note Preview Dots Studio78.4
25BTL-4 Bad Theory Labs78.4
26MiMo-V2-Pro Xiaomi78.0
27GLM-5 Z.AI77.8
28Qwen3.7 Plus Alibaba77.7
29Apodex 1.1 Apodex77.7
30Inkling Thinking Machines Lab77.6
31Mistral Medium 3.5 128B Mistral77.6
32Muse Spark Meta77.4
33Qwen3.6-27B Alibaba77.2
34Claude Sonnet 4.5 Anthropic77.2
35Kimi K2.5 (Reasoning) Moonshot AI76.8
36Kimi K2.5 Moonshot AI76.8
37Grok 4.20 xAI76.7
38Qwen3.5 397B Alibaba76.2
39Muse Glimmer 30B Meta76.0
40MiMo-V2-Omni Xiaomi74.8
41Laguna M.1 Poolside74.6
42Claude 4.1 Opus Anthropic74.5
43Hy3 Preview Tencent74.4
44Step 3.5 Flash StepFun74.4
45GLM-4.7 Z.AI73.8
46MAI-Thinking-1 Microsoft73.5
47Qwen3.6-35B-A3B Alibaba73.4
48MiMo-V2-Flash Xiaomi73.4
49Claude Haiku 4.5 Anthropic73.3
50Jais-3-MoVA-36B-A4B IFM72.8
51Claude 4 Sonnet Anthropic72.7
52MAI-Code-1.1-Flash Microsoft72.6
53Claude 4 Opus Anthropic72.5
54Qwen3.5-27B Alibaba72.4
55Jais-3-32B IFM72.4
56Qwen3.5-122B-A10B Alibaba72.0
57Nemotron 3 Ultra NVIDIA71.9
58Nemotron 3.5 Super VL 120B A12B NVIDIA71.1
59Grok Code Fast 1 xAI70.8
60Ornith-1.5-9B Ornith AI70.6
61Solar Pro 4 Upstage70.6
62Solar Open 2 Upstage70.4
63Jais-3-375B-A23B IFM70.2
64Jais-3-7B IFM70.2
65Qwen3-Coder 480B-A35B Alibaba69.6
66Qwen3.5-35B-A3B Alibaba69.2
67K-EXAONE 2.0 LG AI Research68.2
68LongCat-Flash-Lite-Sparse Meituan68.2
69DeepSeek V3.1 DeepSeek66.0
70Kimi K2 Moonshot AI65.8
71Gemini 2.5 Pro Google63.8
72GPT-OSS 120B OpenAI62.4
73Claude 3.7 Sonnet Anthropic62.3
74Ternary Bonsai 2 27B Prism ML60.8
75GPT-OSS 20B OpenAI60.7
76Nemotron 3 Super 100B NVIDIA60.5
77GLM-4.7-Flash Z.AI59.2
78MiniMax M1 80k MiniMax56.0
79GPT-4.1 OpenAI54.6
80ZAYA1-74B-Preview Zyphra53.2
81Nemotron 3.5 Lightning 30B A3B NVFP4 NVIDIA52.8
82LFM2.5-24B-A2B Liquid AI51.0
83K-Exaone LG AI Research49.4
84o3-mini OpenAI49.3
85LLaDA2.2-flash InclusionAI49.3
86DeepSeek-R1 DeepSeek49.2
87Claude 3.5 Sonnet Anthropic49.0
88Granite 4.2 8B IBM47.7
89Mellum2.1-12B-A2.5B-Thinking JetBrains47.0
90Devstral Small Mistral46.8
91MiniCPM5-2B OpenBMB46.4
92Sarvam 105B Sarvam45.0
93DeepSeek V3 DeepSeek42.0
94Claude 3.5 Haiku Anthropic40.6
95Sarvam 30B Sarvam34.0
96GPT-4.1 mini OpenAI23.6
97Gemini 2.0 Flash Google21.4
98Gemini 1.5 Flash Google9.6