fineweb-edu-fortified

Fineweb-Edu-Fortified The composition of fineweb-edu-fortified, produced by automatically clustering a 500k row sample in Airtrain What is it? Fineweb-Edu-Fortified is a dataset derived from Fineweb-Edu by applying exact-match deduplication across the whole dataset and producing an embedding for each row. The number of times the text from each row appears is also included as a `count` column. The embeddings were produced using TaylorAI/bge-micro Fineweb and Fineweb-Edu were obtained by processing data from 95 crawls of Common Crawl, covering a time period from 2013 to 2024. More information about the original datasets can be found by consulting: Fineweb-edu dataset card Fineweb dataset card Fineweb release blog post Fineweb paper The contents of a randomly selected 500k rows from this dataset can be interactively explored in this Airtrain dashboard. Deduplication Deduplication in original Fineweb and Fineweb-Edu During creation of the original Fineweb dataset, a variety of deduplication strategies were explored. The evaluation criteria used to assess deduplication strategies was to train ablation models on randomly selected subsets of the data, using a subset of up to ~350 billion tokens. Using this mechanism, the Fineweb authors selected a MinHash algorithm, using parameters considering documents with approximately 75% similarity or higher to be duplicates. This deduplication was performed *within* each Common Crawl crawl. For example, it would have removed all approximate duplicates from the 20th crawl from 2013, but would have retained an identical record that showed up in both the 2013-20 crawl and the 2013-48 crawl. The authors note that applying the deduplication *across crawls* reduced the evaluation performance of the ablation models used for assessment. The proposed reason for this performance degredation is that data duplicated across crawls is more likely to be high-quality compared to data that is not, so leaving in the duplicates effectively upsamples t