CUAVerifierBench

CUAVerifierBench: A Human-Annotated Benchmark for Computer-Using-Agent Verifiers Universal Verifier paper: *The Art of Building Verifiers for Computer Use Agents* Dataset Summary CUAVerifierBench** is an evaluation benchmark for **verifiers** of computer-using agents (CUAs) — i.e. judges that read an agent's trajectory (screenshots + actions + final answer) and decide whether the task was completed correctly. Where benchmarks like WebTailBench measure *agents*, CUAVerifierBench measures the *judges that score those agents*. Each row pairs a Fara-7B agent trajectory with one human reviewer's verdict, plus the verdicts produced by the **Universal Verifier (MMRubricAgent)** and several legacy verifiers. Researchers can use the dataset to: Compute verifier–human agreement (Cohen's κ, accuracy, F1) on a fixed corpus of trajectories Study disagreement between judges and how it changes when reviewers see the verifier's output (the "UV-informed" stage) Iterate on new verifier prompts/architectures against a frozen ground-truth set Splits Both configs (`trajectories`, `annotations`) carry the same two splits: | Split | Source | Trajectories | Annotation rows | Annotation stages | |---|---|---|---|---| | `fara7b_om2w_browserbase` | Fara-7B trajectories on the Online-Mind2Web tasks executed via the Browserbase remote browser | 106 | 215 (≈2 reviewers/task) | UV-blind **and** UV-informed | | `internal` | Microsoft-internal task suite — heldout aurora-v2 task definitions evaluated with the same WebSurfer + verifier stack | 154 | 154 (1 reviewer/task) | UV-blind only | The two splits share the same column schema. The `internal` split was annotated in a single UV-blind stage, so its `informed_*` fields and `human_process_score` are empty / null. Dataset Structure The dataset is exposed as **two HuggingFace configs** that are joinable on `task_id`: | Config | Granularity | Contents | |---|---|---| | `trajectories` | one row per task | The agent run — instruction, screenshots, web_s