AbstentionBench

AbstentionBench: A Holistic Benchmark for LLM Abstention Paper | GitHub For reliable LLM deployment, knowing when not to answer is just as important as answering correctly. Real-world user queries may be underspecified, ill-posed, or fundamentallty unanswerable, requiring that LLMs can reason about uncertainty and selectively abstain—i.e., refuse to answer definitively. AbstentionBench is a benchmark for the holistic evaluation of abstention capabilities in frontier LLMs, spanning 20 datasets (including 3 new underspecified reasoning challenges) over 6 abstention scenarios (ranging from underspecified context to stale data). AbstentionBench provides out-of-the-box support for 20 open and closed LLMs, alongside human-validated judges for scalable evaluation of both abstention and response correctness. Getting Started To use the AbstentionBench dataset, first install: NOTE: This dataset only supports datasets versions <= 3.6.0 as it relies on a dataset script.** Then, make sure to enable `trust_remote_code` to allow AbstentionBench to pull in the required data sources: Each sample contains: Example: For the full AbstentionBench pipeline, visit https://github.com/facebookresearch/AbstentionBench. Please note: Third party content pulled from other locations are subject to its own licenses and you may have other legal obligations or restrictions that govern your use of that content. Citation