finetranslations-edu

💬 FineTranslations The world's knowledge in 1+1T tokens of parallel text Table of Contents 💬 FineTranslations What is it? What is it for? Languages and available subsets How to download and use 💬 FineTranslations + Using 🏭 `datatrove` + Using `huggingface_hub` + Using `datasets` Dataset processing steps + 1. Sourcing the data + 2. Running translation at scale + 3. Post-processing + 4. Edu-filtering Dataset card for 💬 FineTranslations Dataset Description + Dataset Summary Dataset Structure + Data Instances + Data Fields + Data Splits Dataset Creation + Curation Rationale + Source Data + Data processing steps + Annotations + Personal and Sensitive Information and opt-out Considerations for Using the Data + Social Impact of Dataset + Discussion of Biases + Other Known Limitations Additional Information + Licensing Information Citation Information NOTE:** this is the Edu version of the dataset, containing only the top 10% scoring data based on an educational classifier applied to the English translations. It has no splits. For the base dataset, see here. What is it? This dataset contains over 1 trillion tokens of parallel text in English and 500+ languages. It was obtained by translating data from **🥂 FineWeb2** into English using Gemma3 27B. We relied on datatrove's inference runner to deploy a **synthetic data pipeline at scale**. Its **checkpointing** and **VLLM lifecycle management** features allowed us to use leftover compute from the HF cluster without fear of preemption. The **async implementation** ensures strong GPU utilization at all times. The **💬 FineTranslations** dataset is fully reproducible and available under the permissive **ODC-By 1.0 license**. This is the _edu_ version. For the base version, see here. What is it for? The main motivation behind the creation of this dataset was improving **translation capabilities**. While models are generally strong at translating from other languages into English (X->English), the opposite is often not true, particu