fineweb-tokenized

FineWeb Tokenized 4 trillion tokens of the pre-tokenized data the 🌐 web has to offer Table of Contents What is it? How to download and use Stream a shard with PyArrow Using Hugging Face datasets Dataset Structure Data Format Tokenizer Details Credits & Attribution Citation What is it? This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the **AnisoleAI BPE tokenizer** (52,022 vocabulary size) and packed into compact `uint16` Parquet shards. By distributing the pre-tokenized corpus, we eliminate the CPU and tokenization bottlenecks in pre-training pipelines. You can stream these shards directly into your training pipeline without any on-the-fly tokenization or preprocessing overhead. How to download and use You can load the tokenizer directly from this repository: Stream a shard with PyArrow The most efficient way to use this dataset is to stream the Parquet files and read the token IDs directly as NumPy arrays: Using Hugging Face datasets To load a shard using the standard Hugging Face `datasets` library: Dataset Structure Data Format Each Parquet file contains a single column: | Column | dtype | Description | |---|---|---| | `token_ids` | `uint16` | Flat list of token IDs for all documents concatenated sequentially. An ` ` token (ID `52002`) marks the end of each document and acts as the boundary. | The data is **not padded**. Each shard contains approximately **2 billion tokens** (~4 GB raw, ~2.5 GB Snappy-compressed). Shards are organized by worker directories: Tokenizer Details The tokenizer configuration is stored in `tokenizer.json` at the root of the repository. **Vocabulary Size:** 52,022 **Type:** Byte-Pair Encoding (BPE) **Special Tokens:** ` ` (ID `52002`) Credits & Attribution This dataset is a tokenized derivative of FineWeb, created by Hugging Face: Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., M