Skip to content

DeepSWE

A long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories.

50 models

Higher is better.

  1. 1Muse Wonder FullMeta79.0
  2. 2Muse WonderMeta78.8
  3. 3Gemini 4 ArgonGoogle77.9
  4. 4DeepSeek V4.1 Flash 1017DeepSeek77.0
  5. 5Muse Spark 1.3Meta75.4
  6. 6Ember-1Fireworks75.2
  7. 7Claude Opus 5.5Anthropic74.2
  8. 8GPT-6 AstraOpenAI74.1
  9. 9Pareto 26.9Unbiased74.0
  10. 10Gemini 3.8 FlashGoogle73.8
  11. 11MiniMax M3.1 FlashMiniMax73.7
  12. 12SWE-2Cognition73.0
  13. 13GPT-5.6 SolOpenAI72.7
  14. 14GPT-6.1 SolOpenAI71.9
  15. 15MiMo-V2.6-ProXiaomi71.9
  16. 16Claude Sonnet 5.5Anthropic71.0
  17. 17Grok 4.7xAI71.0
  18. 18Pareto 26.10 PreviewUnbiased69.9
  19. 19GPT-5.6 TerraOpenAI69.6
  20. 20Claude Opus 5Anthropic68.8
  21. 21GPT-6 SolOpenAI68.8
  22. 22MiMo-V2.6-FlashXiaomi67.9
  23. 23Step 5 PreviewStepFun67.7
  24. 24Kimi K3Moonshot AI67.5
  25. 25Claude Fable 5.1Anthropic67.4
  26. 26GPT-5.6 LunaOpenAI67.2
  27. 27GLM-5.3Z.AI66.9
  28. 28GPT-6 LunaOpenAI66.6
  29. 29Grok 4.6xAI65.9
  30. 30Gemini 3.7 FlashGoogle65.3
  31. 31Hy4 previewTencent64.3
  32. 32GLM-5.3-FlashZ.AI63.4
  33. 33DeepSeek V4 Pro 0813DeepSeek62.7
  34. 34Ultramarine 1 FlashPaper Instruments61.5
  35. 35Muse Spark OpenMeta60.0
  36. 36Ling 3.1 FlashInclusionAI59.7
  37. 37Muse Spark 1.2Meta59.3
  38. 38Qwen3.8-Flash-NextAlibaba58.7
  39. 39Qwen3.8-Omni-FlashAlibaba57.8
  40. 40Qwen3.8 MaxAlibaba56.6
Higher is better. Showing the top 40 of 50. Bar colour: blue proprietary, orange open weights, green other.

About this benchmark

Metric
Pass@1 with confidence interval, cost, time, and token metadata
Tasks
113 software engineering tasks across 91 repositories and 5 languages
Authors
Datacurve AI (2026)

Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.

All results

1Muse Wonder Full Meta79.0
2Muse Wonder Meta78.8
3Gemini 4 Argon Google77.9
4DeepSeek V4.1 Flash 1017 DeepSeek77.0
5Muse Spark 1.3 Meta75.4
6Ember-1 Fireworks75.2
7Claude Opus 5.5 Anthropic74.2
8GPT-6 Astra OpenAI74.1
9Pareto 26.9 Unbiased74.0
10Gemini 3.8 Flash Google73.8
11MiniMax M3.1 Flash MiniMax73.7
12SWE-2 Cognition73.0
13GPT-5.6 Sol OpenAI72.7
14GPT-6.1 Sol OpenAI71.9
15MiMo-V2.6-Pro Xiaomi71.9
16Claude Sonnet 5.5 Anthropic71.0
17Grok 4.7 xAI71.0
18Pareto 26.10 Preview Unbiased69.9
19GPT-5.6 Terra OpenAI69.6
20Claude Opus 5 Anthropic68.8
21GPT-6 Sol OpenAI68.8
22MiMo-V2.6-Flash Xiaomi67.9
23Step 5 Preview StepFun67.7
24Kimi K3 Moonshot AI67.5
25Claude Fable 5.1 Anthropic67.4
26GPT-5.6 Luna OpenAI67.2
27GLM-5.3 Z.AI66.9
28GPT-6 Luna OpenAI66.6
29Grok 4.6 xAI65.9
30Gemini 3.7 Flash Google65.3
31Hy4 preview Tencent64.3
32GLM-5.3-Flash Z.AI63.4
33DeepSeek V4 Pro 0813 DeepSeek62.7
34Ultramarine 1 Flash Paper Instruments61.5
35Muse Spark Open Meta60.0
36Ling 3.1 Flash InclusionAI59.7
37Muse Spark 1.2 Meta59.3
38Qwen3.8-Flash-Next Alibaba58.7
39Qwen3.8-Omni-Flash Alibaba57.8
40Qwen3.8 Max Alibaba56.6
41Ornith-1.5-397B Ornith AI56.0
42DeepSeek V4 Flash 0731 DeepSeek54.4
43Muse Spark 1.1 Meta53.3
44Grok 4.5 xAI53.0
45Gemini 3.6 Flash Google49.0
46Mercury 3 Inception46.3
47Beam 1.0 501B-A23B Reflection AI43.3
48Qwen3.8-27B Alibaba42.2
49Laguna S 2.1 Poolside40.4
50Ornith-1.5-35B-A3B Ornith AI22.0