CCI4.0-M2-Extra-v1

CCI4.0-M2 v1 Dataset Documentation Tech Report👁 Overview CCI4.0-M2 v1 is a comprehensive dataset collection consisting of two specialized subsets designed for language model training. CCI4.0-M2-Base v1 CCI4.0-M2-CoT v1 Download Link BAAI_datahub / modelscope / hf BAAI_datahub / modelscope / hf Notes 5.2TB Chinese webpage, 22TB English webpage, some data released in CCI4.0-M2-Extra(BAAI_datahub / modelscope / hf) due to the license concern. 430 million CoT sample covers math, code, arxiv, wiki and webpage The disk storage for different subdomain datasets is shown in the table below: DataSets Lines Volume (G) Web-EN 7,175,101,435 22,498.641 Web-ZH 1,643,503,909 5,161.0895 Code 215,521,589.3 896.57865 Math 49,685,043.87 269.06715 Books 255,369,254.6 858.55473 Wiki 44,086,649.31 96.667659 Arxiv 1,536,117.66 87.12 ForumQA 28,664,137.6 78.00555 pes2o 4,354,668.22 117.14234 CoT_synthesis 392,470,068.8 4,121.978 Subset Specifications CCI4.0-M2-Base v1 Purpose: Core pretraining data for general language understanding Data Composition: Chinese: 15% (Including data from cooperation projects and open-source projects) English: 85% (primarily sourced from Nemotron-CC and various specific domains like math, code, books etc) Total Volume: 3000GB Processing: Document-level and phrase-level deduplication Rigorous quality filtering through the integration of three quality scores Knowledge enhancement via LLM rewriting and generation Filtering based on LLM Loss grouped by domain PII and toxic filtering License: Due to the license concern, we split CCI4.0-M2-Base v1 into 2 datasets. CCI4.0-M2-Base-v1 For open-source datasets, we selected those with an Apache-2.0 license. For datasets contributed by various institutions, we conducted additional license verification. Nemotron-CC is subject to the Common Crawl License, so we will only release its metadata along with our processed scores. CCI4.0-M2-Extra-v1 For data that is open-source but requires independent licensing or involves mixed/co