UltraX-Preview

UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing 📜 Paper | 💻 Code | 🤖 Models | 📦 UltraData Collection English | 中文 📚 Introduction UltraX** is a function-calling refinement framework for large-scale pre-training data that adaptively generates and executes editing functions for efficient instance-wise refinement. Unlike rule-based or end-to-end LLM rewriting methods, UltraX trains a lightweight refinement model to predict structured editing operations — including insertion, deletion, and modification — which are then deterministically executed on the original text. This dataset collection contains **five English pre-training corpora refined by UltraX**, each with ~20B tokens: | Dataset | Source Corpus | Description | |---------|--------------|-------------| | UltraX-FineWeb | FineWeb | Large-scale Common Crawl web corpus | | UltraX-RedPajama-V2 | RedPajama-v2 | Multi-source web corpus | | UltraX-AICC | AICC | HTML-parsed high-fidelity web corpus | | UltraX-Ultra-FineWeb | Ultra-FineWeb | Quality-filtered FineWeb corpus | | UltraX-FineWeb-ProX-Doc | FineWeb-ProX-Doc | ProX document-level refined corpus | 📢 News **[2026.07.17]** **UltraX** tops the Hugging Face Datasets Trending list, reaching the #1 spot! ⭐️⭐️⭐️ **[2026.07.13]** **UltraX** codebase, refinement model, and refined datasets are now available on GitHub and Hugging Face. 🚀🚀🚀 **[2026.07.10]** **UltraX** technical report is available on arXiv. 🔥🔥🔥 💡 Highlights **Function-Calling Refinement:** Instead of end-to-end text rewriting, UltraX predicts structured editing operations (`keep_all`, `remove_all`, `remove_lines`, `replace_str`, `add_line`), enabling fine-grained instance-level editing with deterministic execution. **LAM + DCR Pipeline:** Line Alignment and Mapping (LAM) aligns original and refined text at line level, while Dynamic Context Replacement (DCR) converts character-level edits into reliable `replace_str` operations with unique context anchoring. **Robust Large-Scale