BIG-Bench Hard

A suite of 23 challenging tasks from the BIG-Bench collaborative benchmark where prior language models failed to exceed average human performance, even with chain-of-thought prompting.

7 models

Higher is better.

1Claude 3.5 HaikuAnthropic86.6
2Soofi S 30B-A3BSoofi Project78.8
3MiniCPM5-1BOpenBMB71.9
4Phi-3.5-miniMicrosoft69.0
5Granite-4.0-H-1BIBM59.7
6Granite-4.0-H-350MIBM33.3
7Granite-4.0-350MIBM33.1

Higher is better. Bar colour: blue proprietary, orange open weights, green other.

About this benchmark

Metric: Mixed reasoning tasks
Tasks: 23 tasks
Authors: Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, Jason Wei (2022)
Source: Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them ↗

Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.