mbpp

Dataset Card for Mostly Basic Python Problems (mbpp) Table of Contents Dataset Card for Mostly Basic Python Problems (mbpp)) Table of Contents Dataset Description - Dataset Summary - Supported Tasks and Leaderboards - Languages Dataset Structure - Data Instances - Data Fields - Data Splits Dataset Creation - Curation Rationale - Source Data - Initial Data Collection and Normalization - Who are the source language producers? - Annotations - Annotation process - Who are the annotators? - Personal and Sensitive Information Considerations for Using the Data - Social Impact of Dataset - Discussion of Biases - Other Known Limitations Additional Information - Dataset Curators - Licensing Information - Citation Information - Contributions Dataset Description **Repository:** https://github.com/google-research/google-research/tree/master/mbpp **Paper:** Program Synthesis with Large Language Models Dataset Summary The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. As described in the paper, a subset of the data has been hand-verified by us. Released here as part of Program Synthesis with Large Language Models, Austin et. al., 2021. Supported Tasks and Leaderboards This dataset is used to evaluate code generations. Languages English - Python code Dataset Structure Data Instances mbpp - full mbpp - sanitized Data Fields `source_file`: unknown `text`/`prompt`: description of programming task `code`: solution for programming task `test_setup_code`/`test_imports`: necessary code imports to execute tests `test_list`: list of tests to verify solution `challenge_test_list`: list of more challenging test to further probe solution Data Splits There are two version of the dataset (full and sanitized), each with four splits: train evaluation tes