airs-bench

AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents The **AI Research Science Benchmark** (AIRS-Bench) quantifies the autonomous research abilities of LLM agents in the area of machine learning. AIRS-Bench comprises **20 tasks** from state-of-the-art machine learning papers spanning diverse domains: NLP, Code, Math, biochemical modelling, and time series forecasting. Each task is specified by a ⟨problem, dataset, metric⟩ triplet and a SOTA value. The agent receives the full task specification and is expected to develop a solution that generates predictions on a test set, which are then evaluated and compared against the state-of-the-art (SOTA) score from a published paper. For full details see the paper and the GitHub repository. Dataset Description This dataset contains the **task specification files** for the 20 AIRS-Bench tasks, formatted for use with the aira-dojo agentic harness. Categories | Category | # Tasks | |---|---| | Text Classification | 2 | | Question Answering | 4 | | Text Extraction and Matching | 3 | | Molecules and Proteins ML | 5 | | Time Series | 3 | | Code | 2 | | Math | 1 | Data Fields | Column | Type | Description | |---|---|---| | `task` | `string` | Task identifier (directory name, e.g. `SentimentAnalysisYelpReviewFullAccuracy`) | | `category` | `string` | High-level domain category (e.g. `Text Classification`, `Code`) | | `research_problem` | `string` | The specific research problem the task addresses | | `dataset` | `string` | HuggingFace dataset identifier used for the task | | `metric` | `string` | Evaluation metric (e.g. `Accuracy`, `MeanAbsoluteError`, `Rouge1`) | | `metadata.yaml` | `string` | Full content of the task metadata file (dataset config, SOTA info, requirements) | | `project_description.md` | `string` | The task prompt provided to the agent | | `prepare.py` | `string` | Dataset preparation script (creates train/test splits, hides test labels) | | `evaluate_prepare.py` | `string` | Evaluation data preparati