TestExplora

TestExplora TestExplora is a systematic, repository-level benchmark designed to evaluate the capability of Large Language Models to proactively discover latent software defects by generating tests. It was developed to evaluate the proactive defect discovery capabilities of LLMs at the repository level. Dataset Details Our dataset is constructed from real-world GitHub pull requests, containing 2,389 test-generation tasks sourced from 1,552 PRs across 482 repositories. Each task is designed such that the model must write test cases capable of triggering a Fail-to-Pass transition between buggy and repaired versions – reflecting true defect detection rather than passive confirmation. The benchmark further includes automatically generated documentation for test entry points to enable scalable evaluation. Uses TestExplora is a benchmark designed to evaluate LLMs as proactive testers within full-scale, realistic repository environments: TestExplora is best suited for evaluating large language models’ capabilities in proactive, repository-level software testing and defect discovery. TestExplora is being shared with the research community to facilitate reproduction of our results and foster further research in this area. TestExplora is intended to be used by domain experts who are independently capable of evaluating the quality of outputs before acting on them. Direct Use Use scripts from TestExplora repo to get info for task instances and organize them into prompt, which can be used to LLMs' inference. Also, you can get info or use agents. Out-of-Scope Use This dataset is not aimed at training for LLMs. You should not take the FEA-Bench as the training dataset to avoid contamination. Dataset Structure An example: