OpenMementos

OpenMementos-228K A dataset of **228,557** reasoning traces annotated with block segmentation and compressed summaries (mementos), derived from OpenThoughts-v3. Memento is a framework for teaching language models to **manage their own context** during long-form reasoning. Instead of generating one long, unstructured chain-of-thought, memento-trained models segment their reasoning into **blocks**, compress each block into a dense **summary** (a *memento*), and continue reasoning from mementos alone. In the released training data, the paper reports **~6× trace-level compression**, from ~10,900 block tokens to ~1,850 memento tokens per trace. Code: microsoft/memento Quick Start Available Subsets `default` — Training-ready data Contains pre-assembled memento-formatted responses ready for SFT training. | Column | Type | Description | |--------|------|-------------| | `problem` | string | Problem statement | | `response` | string | Memento-formatted response with block/summary tokens (see format below) | | `domain` | string | `code`, `math`, or `science` | | `source` | string | Original dataset source | | `difficulty` | int | Difficulty rating (code domain only, 6–10) | `full` — With pipeline components Everything in `default`, plus the intermediate pipeline outputs for researchers who want to re-segment, analyze, or reconstruct: | Column | Type | Description | |--------|------|-------------| | `sentences` | list[string] | Individual sentences from sentence splitting | | `blocks` | list[list[int]] | Block boundaries as `[start_idx, end_idx]` sentence index pairs | | `block_summaries` | list[string] | Iteratively refined summary for each block | Response Format The `response` column contains reasoning traces with Memento special tokens: Special Tokens | Token | Purpose | |-------|---------| | ` ` / ` ` | Reasoning wrapper | | ` ` / ` ` | Reasoning block delimiters | | ` ` / ` ` | Summary (memento) delimiters | During inference with block masking, completed blocks are evict