Skip to content

VulcanBench v3

An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories.

14 models

Higher is better.

  1. 1Grok 4.5xAI89.9
  2. 2Claude Fable 5Anthropic89.5
  3. 3DeepSeek V4 Flash 0731DeepSeek88.4
  4. 4Claude Opus 5Anthropic87.0
  5. 5GPT-5.6 SolOpenAI87.0
  6. 6GPT-5.6 TerraOpenAI87.0
  7. 7Grok 4.6xAI87.0
  8. 8Muse Spark 1.2Meta87.0
  9. 9GPT-5.6 LunaOpenAI85.5
  10. 10Qwen3.8-27BAlibaba82.6
  11. 11Qwen3.8 MaxAlibaba81.2
  12. 12GLM-5.3Z.AI78.3
  13. 13Claude Haiku 4.5Anthropic76.2
  14. 14Kimi K3Moonshot AI73.7
Higher is better. Bar colour: blue proprietary, orange open weights, green other.

About this benchmark

Metric
Pass@1 with low, medium, and high effort
Tasks
23 post-cutoff repository tasks in the v3 report
Authors
VulcanBench contributors (2026)

Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.