autocodearena-v0

BigCodeArena Dataset BigCodeArena** is an open human evaluation platform for code generation, built on top of Chatbot Arena with a comprehensive and on-the-fly execution environment. Unlike traditional evaluation platforms, BigCodeArena enables the execution of LLM-generated code and allows humans to interact with execution processes and outcomes, addressing the challenge of manually examining code quality. This dataset repository, `bigcode/bigcodereward`, contains a subset of the data collected through the BigCodeArena platform. It includes over 14,000 raw code-centric conversation sessions, from which more than 4,700 multi-turn samples with pairwise human preferences were identified. These high-quality human preferences are crucial for evaluating the consistency between reward models and human judgments in code generation scenarios, forming the basis for the **BigCodeReward** benchmark. **Paper**: BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution **Project Page (Hugging Face Space)**: BigCodeArena Space **Code (GitHub Repository)**: bigcode-project/bigcodearena Sample Usage This dataset is used by the **BigCodeReward** component of the BigCodeArena project to evaluate reward model alignment with human preferences on code generation tasks. The following steps, adapted from the GitHub repository, demonstrate how to use the data for evaluation. First, clone the `bigcodearena` repository and install the necessary dependencies for `BigCodeReward`: Next, set your API key for the judge models (e.g., OpenAI) and run the evaluation scripts: For more details on `BigCodeReward`, `AutoCodeArena`, and configuration, please refer to the comprehensive GitHub README. Citation If you find this dataset or the BigCodeArena project useful for your research, please cite the following paper: License This dataset is licensed under the BigCode OpenRAIL-M License.