The Stack v2

The Stack v2 Dataset Description Homepage: https://www.bigcode-project.org/ Repository: https://github.com/bigcode-project Paper: Link Point of Contact: contact@bigcode-project.org The dataset consists of 4 versions: bigcode/the-stack-v2: the full "The Stack v2" dataset <-- you are here bigcode/the-stack-v2-dedup: based on the bigcode/the-stack-v2 but further near-deduplicated bigcode/the-stack-v2-train-full-ids: based on the bigcode/the-stack-v2-dedup dataset but further filtered with heuristics and spanning 600+ programming languages. The data is grouped into repositories. bigcode/the-stack-v2-train-smol-ids: based on the bigcode/the-stack-v2-dedup dataset but further filtered with heuristics and spanning 17 programming languages. The data is grouped into repositories. These datasets only contain the SWHIDs to download the code files and not the content of the files itself. See examples below to see how to download content. We are working on making the training datasets available in the coming weeks. The Stack v2 is significantly larger than v1: The Stack v1 The Stack v2 full 6.4TB 67.5TB dedup 2.9TB 32.1TB train (full) ~200B tokens ~900B tokens Changelog Release Description v2.1.0 Removed repositories that opted out before 2024-04-09. Removed unreachable/private repositories (according to SWH) v2.0.1 Removed repositories that opted out before 2023-10-20. StarCoder2 was trained on this version v2.0 Initial release of the Stack v2 Dataset Summary The Stack v2 contains over 3B files in 600+ programming and markup languages. The dataset was created as part of the BigCode Project, an open scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs). The Stack serves as a pre-training dataset for Code LLMs, i.e., code-generating AI systems which enable the synthesis of programs from natural language descriptions as well as other from code snippets. This dataset is derived from the Software Heritage archive, the largest p