AgentWorldBench

AgentWorldBench AgentWorldBench is a comprehensive evaluation benchmark for language world models, constructed from real-world observations of frontier model trajectories on established benchmarks such as Tool Decathlon, Terminal-Bench 1.0 & 2.0, and OSWorld-Verified. Every evaluation sample is paired with a ground-truth observation obtained from real environment execution, enabling reference-grounded scoring. AgentWorldBench evaluates world modeling quality by scoring each predicted environment observation on five dimensions — **Format**, **Factuality**, **Consistency**, **Realism**, and **Quality** — probing the reasoning, knowledge, and long-context capabilities required for faithful environment simulation. For more details, please refer to the technical report and the blog post. Benchmark Statistics | Domain | Samples | Avg. Turns | Description | |--------|--------:|-----------:|-------------| | MCP | 286 | 23.1 | API server responses: tool call results, database state, service protocols | | Search | 458 | 15.5 | Search engine results: URLs, snippets, rankings, page content | | Terminal | 354 | 26.7 | Command-line environment: shell output, file system state, process behavior | | SWE | 472 | 28.1 | IDE / code editing environment: git diff, test results, compilation errors | | Android | 200 | 37.8 | Android UI hierarchy changes after touch/gesture actions | | Web | 200 | 14.2 | Browser DOM state changes after user interactions | | OS | 200 | 12.7 | Desktop OS state: file system, window management, application behavior | | **Total** | **2,170** | **22.8** | | Data Format Each file is a per-domain JSONL (`{domain}_test.jsonl`). Each record is a single evaluation turn from a multi-turn environment trajectory. `prompt` and `response` are **parallel lists of length `turn_idx`**, representing the full conversation history up to and including the evaluated turn. The ground-truth observation for the current turn is always the **last element** `response[-1]`, while earlie