Long-Horizon-Terminal-Bench

Long-Horizon Terminal-Bench (LHTB) LHTB** is a **46-task** benchmark for measuring how well LLM agents sustain useful work in a containerized terminal over **hundreds of steps**. Unlike short-horizon coding benchmarks where an agent writes one artifact and stops, LHTB drops the agent into a stateful environment and grades it with **hidden, rebuild-from-artifact verifiers** — self-reported progress does not count. 📝 Blog: https://zli12321.github.io/LHTB/ 🏆 Leaderboard: https://zli12321.github.io/LHTB/leaderboard.html 📄 Paper: https://arxiv.org/abs/2607.08964 💻 Code: https://github.com/zli12321/LHTB ### ⚠️ Harness update — continue-until-timeout LHTB adds one behavior on top of Harbor: for long-horizon tasks the agent **keeps working until the task timeout** instead of ending the moment it declares the task complete. After the agent stops, the harness runs the hidden verifier and — if it hasn't fully passed — resumes the same session with the verifier feedback, repeating until the timeout elapses or the verifier passes. This is controlled by `continue_until_timeout = true` in the `[agent]` block of a task's `task.toml`, set on **30 of the 46** tasks. **Stock upstream Harbor ignores this flag**, so those tasks run single-shot there and score lower. To reproduce the official LHTB numbers, use the modified Harbor bundled in the GitHub repo (`harbor/`) — see `harbor/README.md` for the exact diff and `harbor/skills/apply-lhtb-patches/PATCH.md` to apply the same patch to any other Harbor version. What's in this dataset Each of the 46 tasks ships as a Harbor / Terminal-Bench 2.0 task folder under `tasks/`: **Held out on purpose.** The `tests/` (hidden verifiers) and `solution/` (oracle solutions) directories are **intentionally not published** here to keep the benchmark contamination-resistant. Request access or run the official harness for scoring; see the GitHub repo. Dataset viewer The **`tasks`** config (`data/tasks.jsonl`, one row per task) powers the dataset viewer. Co