Skip to content

Testing the Limits of Chain-of-thought with Multistep Soft Reasoning

A dataset for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Tests the ability to perform complex, structured reasoning.

2 models

Higher is better.

  1. 1Ternary Bonsai 8BPrism ML56.2
  2. 21-bit Bonsai 8BPrism ML50.0
Higher is better. Bar colour: blue proprietary, orange open weights, green other.

About this benchmark

Metric
Narrative-based reasoning
Tasks
Multi-step reasoning
Authors
Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, Greg Durrett (2023)

Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.