Massive Multitask Language Understanding Professional
An enhanced version of MMLU with 10 answer choices instead of 4, featuring more reasoning-focused questions that better differentiate frontier models.
67 models
Higher is better.
- 1Qwen3.7 MaxAlibaba89.6
- 2Claude Opus 4.5Anthropic89.5
- 3Qwen3.6 PlusAlibaba88.5
- 4Qwen3.7 PlusAlibaba88.5
- 5Qwen3.5 397BAlibaba87.8
- 6DeepSeek V4 Pro 0813DeepSeek87.5
- 7Kimi K2.5 (Reasoning)Moonshot AI87.1
- 8Kimi K2.5Moonshot AI87.1
- 9Nemotron 3 UltraNVIDIA86.8
- 10Qwen3.5-122B-A10BAlibaba86.7
- 11Solar Pro 4Upstage86.3
- 12Qwen3.6-27BAlibaba86.2
- 13DeepSeek V4 Flash 0731DeepSeek86.2
- 14Solar Open 2Upstage86.2
- 15Qwen3.5-27BAlibaba86.1
- 16GLM-5Z.AI85.7
- 17Qwen3.5-35B-A3BAlibaba85.3
- 18Qwen3.6-35B-A3BAlibaba85.2
- 19Gemma 4 31BGoogle85.2
- 20MAI-Thinking-1Microsoft85.0
- 21MiMo-V2-FlashXiaomi84.9
- 22DeepSeek V3.1 (Reasoning)DeepSeek84.8
- 23GLM-4.7Z.AI84.3
- 24DeepSeek-R1DeepSeek84.0
- 25Nemotron 3 Super 100BNVIDIA83.7
- 26DeepSeek V3.1DeepSeek83.7
- 27K-EXAONE 2.0LG AI Research83.5
- 28Qwen3 235B 2507Alibaba83.0
- 29Gemma 4 26B A4BGoogle82.6
- 30Claude Opus 4.6Anthropic82.0
- 31Exaone 4.0 32BLG AI Research81.8
- 32Sarvam 105BSarvam81.7
- 33Nemotron 3.5 Lightning 30B A3B NVFP4NVIDIA81.6
- 34Kimi K2Moonshot AI81.1
- 35Sarvam 30BSarvam80.0
- 36Grok 3 [Beta]xAI79.9
- 37LongCat-Flash-Lite-SparseMeituan79.2
- 38Claude Sonnet 4.6Anthropic79.2
- 39Nemotron 3 Nano 30BNVIDIA78.3
- 40Nemotron 3 Nano Omni 30B A3BNVIDIA77.3
About this benchmark
- Metric
- 10-way multiple choice
- Tasks
- Multiple subjects
- Authors
- Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, Wenhu Chen (2024)
Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.