wikipedia

Dataset Card for Wikimedia Wikipedia Table of Contents Table of Contents Dataset Description Dataset Summary Supported Tasks and Leaderboards Languages Dataset Structure Data Instances Data Fields Data Splits Dataset Creation Curation Rationale Source Data Annotations Personal and Sensitive Information Considerations for Using the Data Social Impact of Dataset Discussion of Biases Other Known Limitations Additional Information Dataset Curators Licensing Information Citation Information Contributions Dataset Description **Homepage:** https://dumps.wikimedia.org **Repository:** **Paper:** **Point of Contact:** Dataset Summary Wikipedia dataset containing cleaned articles of all languages. The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/) with one subset per language, each containing a single train split. Each example contains the content of one full Wikipedia article with cleaning to strip markdown and unwanted sections (references, etc.). All language subsets have already been processed for recent dump, and you can load them per date and language this way: Data Visualization Click the Nomic Atlas map below to visualize the 6.4 million samples in the `20231101.en` split. Supported Tasks and Leaderboards The dataset is generally used for Language Modeling. Languages You can find the list of languages here: https://meta.wikimedia.org/wiki/List_of_Wikipedias Dataset Structure Data Instances An example looks as follows: Data Fields The data fields are the same among all configurations: `id` (`str`): ID of the article. `url` (`str`): URL of the article. `title` (`str`): Title of the article. `text` (`str`): Text content of the article. Data Splits All configurations contain a single `train` split. Dataset Creation Curation Rationale [More Information Needed] Source Data Initial Data Collection and Normalization The dataset is built from the Wikipedia dumps: https://dumps.wikimedia.org You can find the full list of languages and dates here: https: