frames-benchmark

FRAMES: Factuality, Retrieval, And reasoning MEasurement Set FRAMES is a comprehensive evaluation dataset designed to test the capabilities of Retrieval-Augmented Generation (RAG) systems across factuality, retrieval accuracy, and reasoning. Our paper with details and experiments is available on arXiv: https://arxiv.org/abs/2409.12941. Dataset Overview 824 challenging multi-hop questions requiring information from 2-15 Wikipedia articles Questions span diverse topics including history, sports, science, animals, health, etc. Each question is labeled with reasoning types: numerical, tabular, multiple constraints, temporal, and post-processing Gold answers and relevant Wikipedia articles provided for each question Key Features Tests end-to-end RAG capabilities in a unified framework Requires integration of information from multiple sources Incorporates complex reasoning and temporal disambiguation Designed to be challenging for state-of-the-art language models Usage This dataset can be used to: Evaluate RAG system performance Benchmark language model factuality and reasoning Develop and test multi-hop retrieval strategies Baseline Results We provide baseline results using state-of-the-art models like Gemini-Pro-1.5-0514: Naive prompting: 40.8% accuracy BM25 retrieval (4 docs): 47.4% accuracy Oracle retrieval: 72.9% accuracy Multi-step retrieval & reasoning: 66% accuracy Citation If you use this dataset in your research, please cite our paper: We hope FRAMES will be useful for advancing RAG systems and language model capabilities. For more details, please refer to our full paper.