Skip to content

Measuring Short-Form Factuality in Large Language Models

A benchmark that evaluates the ability of language models to answer short, fact-seeking questions accurately. Focuses on factual correctness rather than reasoning complexity.

10 models

Higher is better.

  1. 1GPT-4.5OpenAI62.5
  2. 2Qwen3 235B 2507Alibaba54.3
  3. 3Grok 3 [Beta]xAI43.6
  4. 4Kimi K2Moonshot AI31.0
  5. 5DeepSeek-R1DeepSeek30.1
  6. 6Gemini 2.0 FlashGoogle29.9
  7. 7DeepSeek V3DeepSeek24.9
  8. 8Mistral Small 3.1Mistral10.4
  9. 9Gemini 1.5 FlashGoogle8.6
  10. 10o1-miniOpenAI7.6
Higher is better. Bar colour: blue proprietary, orange open weights, green other.

About this benchmark

Metric
Short-form Q&A
Tasks
Factual questions
Authors
Jason Wei, Najoung Kim, Hyung Won Chung, Yu-An Chung, Siddhartha Papay, Yifeng Lu, Hannaneh Hajishirzi, Luke Zettlemoyer (2024)

Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.