WebTailBench

WebTailBench: A Comprehensive Benchmark for Computer-Using Agents Paper: Fara-7B: An Efficient Agentic Model for Computer Use Universal Verifier: The Art of Building Verifiers for Computer Use Agents Refresh notes **2026-05-12 — WebTailBench V2**: refreshed task strings and precomputed rubrics for the full 609-task suite (the original V1 tasks were calendar-bound through Nov 2025). Available as the `test_v2` split (alongside the original `test` split, which is preserved unchanged). The `test_v2` split is listed first and is the dataset-viewer default. A side-by-side per-task V1↔V2 diff (task summaries and rubric JSON) is hosted on the Fara repo: WebTailBench V1↔V2 diff. Dataset Summary WebTailBench is a comprehensive evaluation benchmark designed to assess Computer-Using Agent (CUA) models' performance across diverse, realistic web-based tasks. This dataset was first released as part of our **Fara** tech report and is formally presented in the paper Fara-7B: An Efficient Agentic Model for Computer Use. The benchmark consists of: **WebTailBench (Main)**: 609 hand-verified tasks across 11 categories designed to test both breadth of skills and depth through complex, multi-step operations **WebTailBench-Refusals**: 111 curated harmful tasks across 7 categories to evaluate agents' ability to appropriately refuse unsafe requests WebTailBench addresses critical gaps in existing benchmarks by providing: 1. **Expanded task diversity and coverage** - Including underrepresented task types like booking restaurant, hotel, and flight reservations, event tickets, real estate, and job searches 2. **Increased task complexity** - Multi-step and cross-site tasks that chain information across websites 3. **Realistic scenarios** - Tasks drawn from high-traffic webpages reflecting actual human information needs 4. **Objective evaluation** - Goal-oriented tasks with clear success criteria verified as specific and achievable by human annotators Key Features **Realism**: Tasks taken from hi