GitBug-Java: A Reproducible Benchmark of Recent Java Bugs

GitBug-Java: A Reproducible Benchmark of Recent Java Bugs

April 15-16, 2024 | André Silva, Nuno Saavedra, Martin Monperrus
GitBug-Java is a reproducible benchmark of recent Java bugs, featuring 199 bugs extracted from the 2023 commit history of 55 notable open-source repositories. The benchmark ensures the preservation of bug-fixes in fully-reproducible environments, addressing the lack of recent and reproducible bug-fixes in existing benchmarks. GitBug-Java is built using the GitBug-Actions framework to identify and select high-quality bug-fixes. The methodology involves finding locally executable repositories, selecting bug-fixes based on specific criteria, exporting reproduction environments, checking for flakiness, and manually curating the bug-fixes. The benchmark includes reproducible environments that can be executed offline, ensuring long-term reproducibility. GitBug-Java addresses the crucial problem of benchmark leakage in LLM-based research and provides a publicly available repository with proper documentation and a visualization companion website. The benchmark contains a diverse range of relevant open-source repositories with high-quality test suites. It includes 199 reproducible Java bug-fixes, with descriptive statistics on the size of the bug-fix patches, test failures, and distribution over the collection period. GitBug-Java is a high-quality, reproducible benchmark of recent Java bugs, providing a valuable resource for future research in program repair, fault localization, and related fields.GitBug-Java is a reproducible benchmark of recent Java bugs, featuring 199 bugs extracted from the 2023 commit history of 55 notable open-source repositories. The benchmark ensures the preservation of bug-fixes in fully-reproducible environments, addressing the lack of recent and reproducible bug-fixes in existing benchmarks. GitBug-Java is built using the GitBug-Actions framework to identify and select high-quality bug-fixes. The methodology involves finding locally executable repositories, selecting bug-fixes based on specific criteria, exporting reproduction environments, checking for flakiness, and manually curating the bug-fixes. The benchmark includes reproducible environments that can be executed offline, ensuring long-term reproducibility. GitBug-Java addresses the crucial problem of benchmark leakage in LLM-based research and provides a publicly available repository with proper documentation and a visualization companion website. The benchmark contains a diverse range of relevant open-source repositories with high-quality test suites. It includes 199 reproducible Java bug-fixes, with descriptive statistics on the size of the bug-fix patches, test failures, and distribution over the collection period. GitBug-Java is a high-quality, reproducible benchmark of recent Java bugs, providing a valuable resource for future research in program repair, fault localization, and related fields.
Reach us at info@study.space