Multilingual Grade School Math

A multilingual benchmark that translates 250 grade school math problems from GSM8K into 10 typologically diverse languages: Bengali, German, Spanish, French, Japanese, Russian, Swahili, Telugu, Thai, and Chinese.

11 models

Higher is better.

1Llama 3.3 70BMeta91.1
2o1-miniOpenAI89.9
3Llama 3.1 70BMeta86.9
4GPT-4.5OpenAI86.9
5Claude 3.5 HaikuAnthropic85.6
6Phi-4Microsoft80.6
7Llama 3.1 8BMeta68.9
8Phi-3.5-miniMicrosoft47.9
9Granite-4.0-H-1BIBM27.5
10Granite-4.0-H-350MIBM16.2
11Granite-4.0-350MIBM14.7

Higher is better. Bar colour: blue proprietary, orange open weights, green other.

About this benchmark

Metric: Math word problems
Tasks: 250 problems × 11 languages
Authors: Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, Jason Wei (2022)
Source: Language Models are Multilingual Chain-of-Thought Reasoners ↗

Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.