crv

This repository contains the datasets used in the paper **"Verifying Chain-of-Thought Reasoning via its Computational Graph"** (CRV). The data consists of reasoning and math problems, the Chain-of-Thought (CoT) traces generated by **Llama 3.1 8B Instruct**, and step-level correctness annotations used to train and evaluate the CRV verifier. Paper:** arXiv:2510.09312 Codebase:** GitHub Repository Dataset Overview We provide data for three domains. For the arithmetic and boolean domain, the data is organized into files based on complexity (number of operators). Domains 1. **Synthetic Arithmetic:** Nested integer expressions involving addition, multiplication, and unary minus operators. 2. **Synthetic Boolean:** Nested logical expressions over truth values 'True' and 'False' involving boolean operators (`and`, `or`, `not`). 3. **GSM8K:** The test split of the GSM8K benchmark, with generated CoT traces. Data Types For each domain, we provide two types of files: 1. **Raw CoT Generations:** Contains the original expression/question and the CoT response generated by Llama 3.1 8B Instruct. The CoTs are pre-segmented into individual reasoning steps. 2. **Annotated Data:** Contains the same CoTs but with **step-level correctness labels** (Correct/Incorrect). These labels were generated using the consensus pipeline described in the paper (LLM-as-a-Judge + Programmatic Verification). Note on Dataset Size:** You may notice that the annotated datasets (e.g., arith.nt7.annotated.json) for synthetic tasks contain fewer samples than the raw datasets (e.g., arith.nd10000.nt7.json). This is intentional. To ensure high-quality, reliable labels, we employed a dual-verification strategy using both an LLM-as-a-Judge and a Programmatic Verifier. We applied a strict intersection policy: only reasoning steps where both methods agreed on the correctness label were retained. Refer to **Appendix A.2** of the paper for details. File Structure and Naming Convention The files are located in their r