Benchmarks

SWE-bench Verified tests patches for 500 human-filtered GitHub issues. A score measures how many tasks an evaluated model-and-agent setup resolved; the model name alone does not identify that setup.

A long-horizon repository benchmark built to test realistic software engineering work. Its scores need a task-quality and setup check before they support a coding-agent decision.

Terminal-Bench 2.1 results, mostly provider-reported, stored separately from the Terminal-Bench 2.0 lane.

A long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories.

LiveCodeBench v6 is a named release slice used in provider comparison tables. Keeping it separate prevents v6 results from being mixed into older or rolling LiveCodeBench windows.

A multilingual software-engineering benchmark for real-world code issue resolution across multiple programming languages.

A benchmark for agentic software engineering tasks executed in real terminal environments. DeepSeek reports it in the agentic section, while BenchLM also mirrors it in coding for models that publish it as a developer-task signal.

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

A blind-graded coding-agent benchmark with 105 planted bugs across two production TypeScript repositories.

SciCode evaluates language models on generating code for realistic scientific research problems across 16 subfields of physics, math, chemistry, biology, and material science. Problems decompose into 338 subproblems requiring domain knowledge recall, scientific reasoning, and precise code synthesis. Based on real scripts from published research.

A 34-task expansion of FrontierSWE for ultra-long-horizon engineering and research work that remains far from saturation.

HumanEval tests whether generated Python functions pass unit tests on 164 handwritten problems. These scores describe function-level correctness; they do not measure repository work or a coding agent.

Cognition's 100-task software-engineering benchmark for whether coding agents produce mergeable, production-quality pull requests, scored for correctness, tests, scope, style, and maintainability through maintainer-authored rubrics.

Post-training four base language models across seven weighted benchmarks, with ten hours and one H100 per run.

A cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation, then must architect and implement a complete codebase that reproduces the original program's behavior.

An open benchmark for AI coding agents on real-world React Native implementation tasks, emphasizing working app behavior, recommended architecture choices, and strict constraint adherence.

An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories.

A continuously updated software engineering benchmark by Nebius using fresh GitHub issues to avoid contamination. Models are evaluated 5 times per problem under a fixed ReAct scaffolding; the Resolved Rate (best pass@1) is reported.

An app-level coding benchmark that asks DevEco Code configurations to implement observable behavior in buildable OpenHarmony ArkTS applications.

A continuously updated coding benchmark built from newly collected LeetCode, AtCoder, and Codeforces problems. Fresh problem windows reduce one contamination path, but results still need a release and setup check.

Cognition's 150-task Extended subset of the FrontierCode 1.1 software-engineering benchmark.

A multimodal variant of SWE-bench that adds visual context such as screenshots and design mockups to software engineering issue descriptions.

A harder competitive-programming benchmark family built from Codeforces, ICPC, and IOI problems, with quarter-specific public leaderboards and difficulty-aware reporting.

LiveCodeBench v5 is a named release and date-window slice. BenchLM keeps explicitly labeled v5 rows outside the rolling weighted lane.

A display-only SWE-bench Verified reference from Arcee AI's Trinity-Large-Thinking comparison chart.

Competitive-programming rating reported for DeepSeek-V4 thinking-mode evaluations.

A 30-task subset of MLS-Bench that evaluates whether AI systems can invent generalizable and scalable machine-learning methods.

A code-comprehension benchmark covering production repositories across Go, Python, C, and TypeScript.

An ultra-long-horizon software-engineering benchmark with open-ended implementation, performance, and research tasks designed to challenge frontier coding agents.

A software-engineering benchmark for post-training infrastructure and implementation tasks, evaluated through the official Harbor implementation.

A post-cutoff software-engineering benchmark with hidden functional tests and regression guards, reported for vendor coding-agent harnesses.

DeepSeek's internal full-stack coding-agent benchmark.

DeepSeek's internal hard coding-agent benchmark.

A Moonshot AI internal coding-agent benchmark for realistic software-engineering tasks across mainstream programming languages and production technology stacks.

This lane contains DeepSeek's LiveCodeBench Pass@1-COT results. The explicit metric and prompting label keeps them separate from generic and version-specific LiveCodeBench rows.

A code-generation benchmark reported in DeepSeek-V4 base-model evaluations.

An electrical-engineering agent benchmark maintained by atopile.

A display-only benchmark for difficult GPU kernel implementation and optimization tasks.

A multi-language software-engineering benchmark that measures repository-level bug fixing and implementation across more than one programming ecosystem.

A research-reproduction benchmark that asks agents to recreate the contributions of AI papers from the paper alone.

Program-reconstruction hidden-test pass rate after the first of five sequential long-context episodes.

Qwen's internal benchmark for building and rendering bilingual React projects across seven categories.

A display-only provider benchmark for generating or manipulating SVG outputs from natural-language requirements.

A text-to-SQL benchmark over realistic warehouse-scale schemas, reported by Interfaze for model comparison.

A display-only MiniMax provider benchmark for end-to-end coding-agent and product-building tasks.

A repo-level code generation and full-project delivery benchmark spanning web, mobile, and simulation-style implementation tasks.

Epoch AI's corrected v2 core FrontierMath suite of private advanced mathematics problems. Models can reason iteratively and use Python; scores are pass rates on the private set.

Epoch AI's corrected v2 Tier 4 expansion, a separate set of exceptionally difficult research-level mathematics problems evaluated with Python-enabled iterative reasoning.

The most recent AIME examination, featuring 15 challenging mathematics problems testing olympiad-level mathematical reasoning with integer answers from 000-999.

A 2026 American Invitational Mathematics Examination snapshot used in frontier-model comparison tables for mathematical reasoning.

A February 2026 HMMT slice used in newer frontier-model math comparisons.

A curated subset of 500 problems from the MATH dataset, covering algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus.

The 2024 edition of AIME, maintaining the same format of 15 challenging mathematics problems with integer answers from 000 to 999.

A February 2025 HMMT slice used in exact-value provider tables for advanced contest-math reasoning.

A November 2025 HMMT slice for high-end mathematical reasoning comparisons.

A multimodal mathematical reasoning benchmark that tests whether models can answer visually grounded math questions correctly.

A challenging mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations.

A high-difficulty mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations.

Legacy FrontierMath values retained for historical model pages. This field is not used in current rankings because it can mix prior benchmark versions and slices.

GSM8K tests multi-step arithmetic in grade-school word problems. OpenAI introduced the dataset in 2021; a high score on this fixed test is one piece of evidence about mathematical reasoning.

A display-only AIME25 reference from Arcee AI's Trinity-Large-Thinking launch chart.

A shortlist subset of the Apex mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations.

The premier US mathematical olympiad competition, featuring proof-based problems that require deep mathematical insight and rigorous argumentation at the highest competition level.

Final-answer research mathematics problems drawn from recent arXiv papers, including problems based on resolved or refuted conjectures.

The August 2026 ArXivMath release evaluated with a code-execution sandbox and no internet access.

Proof-based olympiad performance on all six IMO 2026 problems.

Final-answer research mathematics problems drawn from recent arXiv abstracts.

Final-answer research mathematics problems drawn from recent arXiv abstracts with tool access.

The three official theory problems from the 2025 International Physics Olympiad, scored with blinded human evaluation.

Research-level mathematics problems with unique programmatically verified closed-form answers.

Research-level mathematics problems with unique programmatically verified closed-form answers and tool access.

ARC-AGI-2 tests whether a system can infer a transformation from example grids and apply it to a new grid. Compare accuracy with the evaluation split, attempt limit, and cost attached.

ARC Prize fluid-intelligence benchmark using novel visual grid transformations.

An interactive successor to ARC-AGI-2 that evaluates whether an AI agent can learn unfamiliar task mechanics through action and feedback.

A long-context benchmark that measures whether models can actually use extended context windows for reasoning and retrieval.

A long-context benchmark for memory, retrieval, and multi-round coherence over large contexts.

Scores how far a model advances 120 open research tasks across biology, chemistry, physics, astronomy, Earth science and engineering on a 0 to 4 progress scale, crediting supported partial work and honest statements of what remains unresolved.

ARC Prize verified interactive reasoning results using a provider-specific stateful adapter, kept separate from Standard harness results.

A suite of 23 challenging tasks from the BIG-Bench collaborative benchmark where prior language models failed to exceed average human performance, even with chain-of-thought prompting.

A reading-comprehension benchmark requiring discrete reasoning over paragraphs, reported in DeepSeek-V4 base-model evaluations.

A long-context retrieval benchmark that measures whether a model can recover relevant information embedded deep inside very long contexts.

Breadth-first-search graph traversal on 200 problems with context lengths from 256K to 1M tokens.

A million-token MRCR long-context retrieval benchmark reported in DeepSeek-V4 model evaluations.

A million-token CorpusQA long-context question-answering benchmark reported in DeepSeek-V4 model evaluations.

A multistage statistical-reasoning benchmark for genomics and biological-data analysis agents.

Long-context graph traversal benchmark using breadth-first search tasks.

MRCR v2 slice focused on very long contexts at 128K-256K lengths.

MRCR v2 slice focused on retrieval across 256K-512K-token contexts.

MRCR v2 slice focused on retrieval across 512K-1M-token contexts.

MRCR v2 slice focused on long-context retrieval at 64K-128K lengths.

A dataset for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Tests the ability to perform complex, structured reasoning.

Terminal-Bench 2.1 results, mostly provider-reported, stored separately from the Terminal-Bench 2.0 lane.

A game-environment benchmark that evaluates AI models in novel games covering strategic planning, resource management, spatial reasoning, cooperation, and theory of mind.

BrowseComp tests whether a browsing agent can find difficult, fact-seeking answers across the web. Its 1,266 problems use short answers that are easier to grade than an open-ended research report.

A benchmark for tool-calling over Model Context Protocol integrations and external tools.

Tests whether a coding agent can explore a 256-room 3D puzzle world and collect its 100 hidden gems, using box-pushing puzzles, camera rotation, and long-horizon planning.

A benchmark for agentic software engineering tasks executed in real terminal environments. Models must inspect files, run commands, edit code, and recover from errors over multi-step workflows.

A transparent real-world autonomous-agent benchmark with 300 human-verified tasks, 2,159 rubric items, and Pass^3 scoring across general, multi-turn, and native multimodal agent tasks.

OSWorld-Verified is the July 2025 repaired release of OSWorld's real-computer evaluation. It measures whether a model-agent system can finish desktop and web tasks from configured starting states, with success checked by execution-based evaluators.

An occupational agent benchmark for professional workflows that workers say they most want delegated to AI.

Tool-augmented Humanity's Last Exam scores reported in DeepSeek-V4 thinking-mode evaluations.

An agent benchmark for completing automation workflows in reproducible task environments.

A cybersecurity task benchmark for evaluating defensive cyber workflows and vulnerability-oriented agent performance.

OSWorld 2.0 evaluates long-horizon computer-use tasks. Published results use different task releases, metrics, and run settings, so scores appear by comparable protocol and unmatched rows stay unranked.

τ³-bench is the current evolution of Sierra's tool-agent-user framework, adding corrected task releases and newer knowledge and voice evaluation modes alongside airline, retail, and telecom.

A tool-use benchmark focused on selecting, sequencing, and completing tasks with external tools.

A verified tool-use benchmark variant for completing multi-step workflows with external tools.

An agentic browsing benchmark where models search the web, gather evidence, and answer list-style questions using browser tools.

A function-calling benchmark for tool selection, schema adherence, and argument correctness.

An end-to-end autonomous scientific research benchmark with 40 tasks across 10 scientific domains, where agents receive related literature and raw data, then attempt to rediscover the hidden target paper.

Tests whether a computer-use agent can learn a real accounts-payable job on the job, processing 100 vendor bills in a company ERP system with only the handbook, historical records, and mentor feedback a new hire would get.

The current Terminal-Bench release measures difficult computer work after recalibrating task resources, fixing unstable tasks, and removing tasks that no longer separate frontier systems.

An agent benchmark reported in DeepSeek's V4 Flash 0731 launch comparison.

Tests agents on 150 tasks across 25 categories, including Hermes skills, research, diagrams and art, memory, tool use and safety. Each agent starts in a workspace with real files, some tasks add follow-up turns, and the grader checks the files and state left behind.

An index of agentic capability from Nous Research. It averages pass@1 scores on four suites, Hermes Bench, TerminalBench 4, TerminalBench Science and SkillsBench, with every run inside the Hermes Agent harness, and reports the mean cost per task beside the score.

A broad research-agent benchmark for open-ended information gathering, synthesis, and answer construction across wide search spaces.

A controlled benchmark for evaluating whether AI agents can extend vulnerability-triggering inputs into working exploits.

A continuously maintained benchmark for difficult computer work, including coding, deep learning, finance, engineering, math, and science tasks.

Defensive vulnerability patching on 120 held-out tasks spanning 73 weakness types.

An interactive real-world agent benchmark grounded in practical consumer-service tasks such as delivery, in-store consumption, and online travel workflows.

Qwen's internal OpenClaw-style benchmark for measuring broad real-world agent performance across practical productivity and research tasks.

A professional-services agent benchmark covering long-horizon knowledge-work tasks.

A mobile GUI agent benchmark for completing Android app workflows and on-device tasks.

A long-horizon planning benchmark that tests whether agents can optimize under explicit time, budget, and feasibility constraints.

An OpenClaw agent benchmark from Kilo that measures successful task completion across standardized real-world agent workflows.

Agentic data-analysis tasks scored against per-task rubrics at a 980K-token budget.

A function-calling benchmark for tool selection, schema adherence, and argument correctness, covering single-turn, parallel, irrelevance and multi-turn subsets.

A Model Context Protocol task benchmark used in Qwen's launch tables to measure practical execution over MCP-style tools and integrations.

A Qwen benchmark for artifact and webpage generation quality reported as an Elo-style rating.

A benchmark of AI agents completing expert-curated research workflows across the life, physical, Earth, mathematical, and engineering sciences.

Qwen's internal benchmark for long-horizon professional work across computer science, finance, law, medicine, and other productivity domains.

Fraction of Toolathlon Verified tasks solved in at least one of three trials.

Fraction of Toolathlon Verified tasks solved in all three independent trials.

Average assistant turns per Toolathlon Verified trajectory.

WebArena-Verified is an audited release of the WebArena browser-agent benchmark. It rechecks task descriptions, reference answers, and evaluators, and replaces nondeterministic judging with deterministic checks where possible.

τ²-bench Airline tests conversational agents on airline customer-service tasks governed by domain policy and database-changing tools.

A display-only provider benchmark for finance-oriented tool-use and agent workflows.

Harvey AI's strict held-out task success rate requiring every rubric criterion to pass.

Harvey AI's mean criterion-level score on its held-out legal-agent evaluation.

A cybersecurity benchmark of professional Capture the Flag tasks for measuring autonomous cyber agent capability and risk.

Average coverage of required claims in answers produced during real-world MCP tool-use workflows.

A lightweight machine-learning competition benchmark that measures whether models can iteratively train, evaluate, and improve ML systems in low-resource settings.

An OpenClaw-derived agent benchmark covering practical work and life tasks such as office document delivery, research, planning, and code maintenance.

A computer-use benchmark for GUI task completion across the broader OSWorld task suite.

An agent benchmark for completing business workflows across simulated applications and API endpoints.

BrowseComp accuracy from ten collaborating Opus 5 agents on a pre-release model and unreleased effort configuration.

Evaluates research agents against a fixed document corpus, separately from live-web BrowseComp.

Moonshot AI's internal benchmark for presentation and deck-production workflows.

A display-only Elo rating from blinded comparisons of multi-file web applications built by coding agents.

A display-only provider-table GDPval rubric score for economically valuable work tasks.

A Moonshot AI internal long-horizon agent benchmark for persistent professional coworking tasks.

Strict task success requiring every expert-written legal-work rubric criterion to pass.

Mean fraction of expert-written rubric criteria passed across legal-agent tasks.

Tests long-context agents as tool environments grow to a 256K-token environment description.

A human-verified edition of MCPMark for MCP tool use across Notion, GitHub, Filesystem, Postgres, and Playwright server environments.

A mobile-use agent benchmark for completing interactive tasks in smartphone environments.

Tests whether coding agents can complete long-horizon, whole-repository stack migrations while preserving the original program's behavior.

A spreadsheet-focused benchmark for agentic analysis and editing workflows.

Google-run scientific workflow evaluation using a sixfold verifier timeout to address verification timeouts.

WebArena tests whether a browser-agent system can complete 812 long-horizon tasks inside self-hosted replicas of functional websites. It checks the requested end state, so a result reflects the model, agent scaffold, browser interface, action budget, and evaluator together—not the base model alone.

A browser-agent benchmark for completing multi-step workflows on live websites.

A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Designed to be difficult even for skilled non-experts with access to Google.

An expert-authored benchmark designed to probe frontier knowledge and reasoning. BenchLM keeps protocol differences visible because tool-assisted and closed-book HLE runs answer different questions.

An enhanced version of MMLU with 10 answer choices instead of 4, featuring more reasoning-focused questions that better differentiate frontier models.

Tool-free variant of Humanity's Last Exam that isolates a model's raw frontier reasoning.

A comprehensive multiple-choice question answering test covering 57 tasks including elementary mathematics, US history, computer science, law, and more. Tests knowledge across diverse academic subjects from high school to professional level.

An expanded version of GPQA that evaluates graduate-level knowledge and reasoning capabilities across 285 disciplines, providing comprehensive coverage of academic domains.

An open benchmark for clinician-facing model responses across care consult, writing and documentation, and medical research tasks.

A harder refresh of MMLU intended to keep broad knowledge evaluation useful after the original benchmark became too easy for frontier models.

A harder subset of OpenAI's HealthBench for evaluating open-ended medical and health reasoning with rubric-based grading.

A multilingual MMLU-style benchmark reported in provider evaluation tables.

A Chinese-language academic and professional benchmark spanning humanities, social science, STEM, and applied subjects.

A benchmark that evaluates the ability of language models to answer short, fact-seeking questions accurately. Focuses on factual correctness rather than reasoning complexity.

HealthBench score after applying a verbosity penalty to model responses.

Raw score on realistic multi-turn healthcare conversations graded against expert-written rubrics.

Raw score on physician-authored clinical consult, documentation, and research conversations.

A benchmark of realistic biology-research tasks involving literature, figures, tables, databases, and bioinformatics files.

Computational biology challenges with objective answers that remained unsolved by independent human experts.

A verified and revised Humanity's Last Exam set that removes uncertain items and repairs fixable questions before evaluating frontier expert reasoning.

Computational biology challenges that independent human experts were able to solve.

A display-only MMLU-Pro reference from Arcee AI's Trinity-Large-Thinking launch chart.

A medical multiple-choice benchmark spanning many specialties with 10 answer options per question.

Generates novel protein sequences under family, topology, globularity, and structural-motif constraints.

Detects and fixes errors in molecular-biology protocols using document, code, and web-search tools.

Extends online molecular-biology protocols in additional directions using document and web-search tools.

Single-cell RNA sequencing analysis tasks spanning common bioinformatics workflows.

Analysis of spatial transcriptomics data across externally validated biological problems.

A Chinese short-form factuality benchmark reported by DeepSeek for V4 model evaluations.

Designs diverse miniprotein binders against target protein structures in a GPU-equipped research sandbox.

A research-focused FrontierScience evaluation variant for scientific investigation and problem solving.

Selects compounds with the highest or lowest measured ADME-related property from molecular structures and reference values.

Matches multi-channel cellular imaging data to the molecule-dose perturbations that produced it.

Ranks held-out protein designs for experimental testing using wet-lab campaign measurements and a selection objective.

A human-centric exam benchmark for general knowledge and reasoning reported in DeepSeek-V4 base-model evaluations.

A Chinese multitask academic benchmark reported in DeepSeek-V4 base-model evaluations.

A benchmark for research-level scientific reasoning, designed to separate frontier models on difficult science tasks that mix domain knowledge with deep reasoning.

Chemistry tasks covering spectroscopy, synthesis planning, reaction prediction, and chemical structure images.

Predicts mutation effects by ranking mutant protein sequences against wild type and comparing against laboratory measurements.

A display-only Design Arena website-generation Elo score surfaced on OpenRouter model benchmark pages.

A harder multimodal benchmark for frontier models that combines text with images, diagrams, charts, and academic visual reasoning tasks.

A scientific chart reasoning benchmark that tests whether models can understand, interpret, and reason about complex scientific visualizations including plots, diagrams, and data charts.

Scores document parsers and vision-language models on 2,078 human-verified enterprise pages across tables, charts, content faithfulness, semantic formatting and visual grounding, with the listed cost per page.

Scores schema-guided extraction from 370 enterprise documents and 4,869 pages on value F1, record completeness, word and page grounding and cost, split by short, medium and long documents.

A broad multimodal reasoning benchmark spanning charts, diagrams, tables, and academic visual question answering.

A visual mathematics benchmark that tests whether a model can solve math problems grounded in diagrams, equations, figures, and other visual inputs.

A visual question answering benchmark focused on straightforward image-grounded understanding.

A grounded visual reasoning benchmark focused on evidence-based question answering over real images.

A grounded visual QA benchmark focused on answering practical questions about real-world images and scenes.

A GUI-grounding benchmark for 1,581 instructions in full-screen, high-resolution professional interfaces. It tests where a target is, not whether an agent can finish the surrounding workflow.

Tool-free variant of CharXiv that isolates raw visual reasoning ability without code execution or tool augmentation.

A benchmark for grounded reasoning over office-style documents, spreadsheets, charts, and business artifacts.

A vision-centric benchmark for high-level multimodal reasoning and perception quality.

A video extension of MMMU-style multimodal reasoning over expert questions grounded in temporal media.

A document understanding benchmark used in frontier-model comparison tables to measure extraction and grounded reasoning quality on complex documents.

A video understanding benchmark that allows subtitle access when answering multimodal questions about videos.

A multi-task video understanding benchmark averaged across MLVU categories.

Tool-augmented MMMU-Pro variant that allows Python assistance during multimodal reasoning.

A multimodal benchmark for fine-grained visual perception and grounded reasoning tasks.

A visual counting benchmark that tests whether a model can count objects and entities reliably in complex scenes.

A multimodal medical multiple-choice benchmark covering clinical images such as X-rays, histology, and dermatology.

A referring-expression grounding benchmark averaged across RefCOCO variants to test whether a model can localize described objects correctly.

A multi-step visual reasoning benchmark with pass@5 reporting and optional tool use.

A diagram understanding benchmark focused on scientific and educational visual question answering.

An OCR-focused benchmark for reading and extracting text from visually complex documents and images.

A benchmark for evaluating multimodal models on video understanding tasks across multiple disciplines, emphasizing temporal reasoning and comprehension over video content.

A multimodal variant of SWE-bench that adds visual context (screenshots, design mockups) to software engineering issue descriptions, testing whether models can leverage visual information for code generation.

A stricter Video-MME setting that removes subtitle help and tests video understanding from visual and audio context alone.

A multimodal benchmark for dynamic mathematical reasoning over visual and structured inputs.

A long-video understanding benchmark for retrieving and reasoning over information distributed across extended video inputs.

A Python-assisted BabyVision evaluation for fine-grained visual perception and grounded reasoning.

Professional chart understanding across 100 specialized chart types with expert-set answer tolerances.

Professional chart understanding with a container, standard libraries, and image cropping.

A harder companion to ExtractBench: 100 synthetic documents and 2,622 pages across 86 document types that test document conventions, house vocabulary, counting rules, drawn relations and charts, scored by perfect-document rate.

A long-document multimodal benchmark for grounded reasoning over extended document contexts.

A tool-augmented MathVision variant that permits Python during visual mathematics reasoning.

A native OCR benchmark for reading text from images across multilingual scripts, low-quality scans, handwriting, structured layouts, charts, and screenshots.

A multimodal math benchmark for visually grounded mathematical reasoning and answer generation.

Generates CadQuery code from multi-view renders with image inspection and code-execution tools.

A visual detection and grounding benchmark slice used to compare zero-shot object understanding across diverse domains.

A benchmark for converting visual references into functional web implementations.

Generates CadQuery code from multi-view renders and scores geometric similarity by voxel intersection-over-union.

Grounded numerical reasoning over a corpus of historical U.S. Treasury Bulletin documents.

Moonshot AI's internal benchmark for atomic visual perception capabilities.

A Python-assisted ZeroBench_main evaluation reported as pass@5.

Quantitative biomedical image analysis across segmentation, detection, measurement, classification, registration, reconstruction, and tracking tasks.

An agentic spatial reasoning benchmark reported as a normalized score.

Professional document understanding over 100 real-world PDFs from ten domains.

A comprehensive benchmark for multimodal large language models on video understanding, covering temporal reasoning, perception, and question answering over videos.

A visual question answering benchmark whose questions cannot be answered from the image alone and require commonsense or world knowledge.

A scientific chart understanding benchmark scored across both the descriptive and reasoning splits rather than the reasoning split alone.

Professional document understanding with a container, standard libraries, and image cropping.

A display-only Liquid AI extraction metric measuring field-name agreement between requested schema fields and extracted JSON fields.

A display-only Liquid AI extraction metric measuring the share of image-to-JSON outputs that parse as strict JSON.

A display-only Liquid AI extraction metric measuring judged agreement between extracted values and the source image.

A harder MMSearch variant for multimodal retrieval and grounded tool-use workflows.

A general visual question-answering benchmark used in provider tables for real-image reasoning quality.

A document-understanding benchmark for parsing and reasoning over complex document layouts.

A document parsing benchmark whose overall score composites edit distance, CDM and TEDS sub-scores across diverse PDF layouts.

A forced-answer WorldVQA variant for atomic visual world knowledge.

An end-to-end document understanding benchmark over long, layout-rich PDFs with tables, equations, headers, footnotes, and multi-column flows.

A multilingual extension of SWE-bench covering 300 problems across 9 programming languages, testing code generation and bug fixing beyond Python.

A multilingual extension of professional-level academic evaluation across many languages.

A multilingual benchmark that translates 250 grade school math problems from GSM8K into 10 typologically diverse languages: Bengali, German, Spanish, French, Japanese, Russian, Swahili, Telugu, Thai, and Chinese.

A multilingual benchmark used in provider tables to measure inclusive language coverage and cross-lingual understanding beyond common high-resource languages.

A broad multilingual benchmark row from Qwen's launch comparisons intended to measure cross-lingual capability beyond a single language family.

A multilingual mathematical reasoning benchmark that tests whether math performance transfers across languages rather than only in English.

A multilingual instruction-following and understanding benchmark row published in Qwen's launch comparisons.

MMLU-style knowledge evaluation across 42 high- and low-resource languages.

Culturally grounded knowledge comprehension across ten Indic languages and English.

Cohere's long-document translation evaluation: two book chapters translated in a single call, scored per paragraph with xCOMET-XL.

A display-only multilingual QA retrieval benchmark reported by Liquid AI for LFM2.5 retriever models, using Recall@20 across 11 languages.

A display-only multilingual retrieval benchmark reported by Liquid AI for LFM2.5 retriever models, using NDCG@10 across 11 languages.

Machine-translation quality on the WMT26 general translation task, averaged across all evaluated languages and scored 0-100 by an LLM judge.

A benchmark of 541 prompts built from 25 verifiable instruction types. It tests whether a model follows checkable constraints such as keyword, length, casing, and response-format requirements.

IFBench evaluates precise instruction-following generalization on 58 challenging, verifiable out-of-domain constraints. Unlike IFEval which tests familiar constraint types, IFBench specifically measures how well models follow novel instructions they haven't been optimized for, exposing overfitting to common instruction patterns.

Measures how often indirect prompt-injection attacks succeed within fifteen attempts. Lower scores mean greater resistance to the tested attacks.

Tests instruction retention, memory of user information, versioned editing, and coherence over multi-turn conversations.

A structured-output benchmark from Interfaze measuring whether extracted JSON leaf values exactly match verified ground truth.