Skip to content

NL2Repo

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

31 models

Higher is better.

  1. 1DeepSeek V4.1 Flash 1017DeepSeek68.2
  2. 2DeepSeek V4 Pro 0813DeepSeek61.5
  3. 3MiniMax M3.1 FlashMiniMax61.2
  4. 4Ornith-1.5-397BOrnith AI59.5
  5. 5Hy4 previewTencent58.9
  6. 6GLM-5.3Z.AI58.0
  7. 7GLM-5.3-FlashZ.AI56.3
  8. 8Qwen3.8 MaxAlibaba55.9
  9. 9DeepSeek V4 Flash 0731DeepSeek54.2
  10. 10dots3-note PreviewDots Studio49.8
  11. 11GLM-5.2Z.AI48.9
  12. 12Qwen3.8-Omni-FlashAlibaba48.9
  13. 13Ornith-1.0-397BDeepReinforce AI48.2
  14. 14Qwen3.8-Flash-NextAlibaba48.1
  15. 15Qwen3.7 MaxAlibaba47.2
  16. 16Seed 2.1 ProByteDance47.0
  17. 17Ornith-1.5-35B-A3BOrnith AI46.2
  18. 18Claude Opus 4.5Anthropic43.2
  19. 19Qwen 3.6 Max (preview)Alibaba42.9
  20. 20GLM-5.1Z.AI42.7
  21. 21Qwen3.8-27BAlibaba42.3
  22. 22MiniMax M3MiniMax42.1
  23. 23Qwen3.7 PlusAlibaba41.1
  24. 24MiniMax M2.7MiniMax39.8
  25. 25Qwen3.6 PlusAlibaba37.9
  26. 26Qwen3.6-27BAlibaba36.2
  27. 27GLM-5Z.AI35.9
  28. 28Ornith-1.5-9BOrnith AI32.4
  29. 29Qwen3.5 397BAlibaba32.2
  30. 30Kimi K2.5Moonshot AI32.0
  31. 31Qwen3.6-35B-A3BAlibaba29.4
Higher is better. Bar colour: blue proprietary, orange open weights, green other.

About this benchmark

Metric
Repository understanding benchmark
Tasks
Natural language to repository tasks
Authors
MiniMax (2026)

Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.