AutomationBench, Zapier private evaluation set release 1.0.6

An agent benchmark for completing business workflows across simulated applications and API endpoints.

1 models

Higher is better.

Higher is better. Bar colour: blue proprietary, orange open weights, green other.

Metric: Task pass rate
Tasks: 600+ held-out workflows across 47 simulated applications and six business domains
Authors: Daniel Shepard and Robin Salimans (2026)
Source: AutomationBench ↗

Scores are as reported by model developers or evaluators and may use different settings (for example reasoning effort or tools). Compare with care.