github-code

GitHub Code Dataset Dataset Description The GitHub Code dataset consists of 115M code files from GitHub in 32 programming languages with 60 extensions totaling in 1TB of data. The dataset was created from the public GitHub dataset on Google BiqQuery. How to use it The GitHub Code dataset is a very large dataset so for most use cases it is recommended to make use of the streaming API of `datasets`. You can load and iterate through the dataset with the following two lines of code: You can see that besides the code, repo name, and path also the programming language, license, and the size of the file are part of the dataset. You can also filter the dataset for any subset of the 30 included languages (see the full list below) in the dataset. Just pass the list of languages as a list. E.g. if your dream is to build a Codex model for Dockerfiles use the following configuration: We also have access to the license of the origin repo of a file so we can filter for licenses in the same way we filtered for languages: Naturally, you can also download the full dataset. Note that this will download ~300GB compressed text data and the uncompressed dataset will take up ~1TB of storage: Data Structure Data Instances Data Fields |Field|Type|Description| |---|---|---| |code|string|content of source file| |repo_name|string|name of the GitHub repository| |path|string|path of file in GitHub repository| |language|string|programming language as inferred by extension| |license|string|license of GitHub repository| |size|int|size of source file in bytes| Data Splits The dataset only contains a train split. Languages The dataset contains 30 programming languages with over 60 extensions: Licenses Each example is also annotated with the license of the associated repository. There are in total 15 licenses: Dataset Statistics The dataset contains 115M files and the sum of all the source code file sizes is 873 GB (note that the size of the dataset is larger due to the extra fields). A breakdown per