409 research outputs found
ConDefects: A New Dataset to Address the Data Leakage Concern for LLM-based Fault Localization and Program Repair
With the growing interest on Large Language Models (LLMs) for fault
localization and program repair, ensuring the integrity and generalizability of
the LLM-based methods becomes paramount. The code in existing widely-adopted
benchmarks for these tasks was written before the the bloom of LLMs and may be
included in the training data of existing popular LLMs, thereby suffering from
the threat of data leakage, leading to misleadingly optimistic performance
metrics. To address this issue, we introduce "ConDefects", a novel dataset of
real faults meticulously curated to eliminate such overlap. ConDefects contains
1,254 Java faulty programs and 1,625 Python faulty programs. All these programs
are sourced from the online competition platform AtCoder and were produced
between October 2021 and September 2023. We pair each fault with fault
locations and the corresponding repaired code versions, making it tailored for
in fault localization and program repair related research. We also provide
interfaces for selecting subsets based on different time windows and coding
task difficulties. While inspired by LLM-based tasks, ConDefects can be adopted
for benchmarking ALL types of fault localization and program repair methods.
The dataset is publicly available, and a demo video can be found at
https://www.youtube.com/watch?v=22j15Hj5ONk.Comment: 5pages, 3 figure
- …