BugsInPy: A database of existing bugs in Python programs to enable controlled testing and debugging studies

Lee Kuan Yew Fellowship, Singapore Management Universit

Back to the Future! Studying Data Cleanness in Defects4J and its Impact on Fault Localization

For software testing research, Defects4J stands out as the primary benchmark dataset, offering a controlled environment to study real bugs from prominent open-source systems. However, prior research indicates that Defects4J might include tests added post-bug report, embedding developer knowledge and affecting fault localization efficacy. In this paper, we examine Defects4J's fault-triggering tests, emphasizing the implications of developer knowledge of SBFL techniques. We study the timelines of changes made to these tests concerning bug report creation. Then, we study the effectiveness of SBFL techniques without developer knowledge in the tests. We found that 1) 55% of the fault-triggering tests were newly added to replicate the bug or to test for regression; 2) 22% of the fault-triggering tests were modified after the bug reports were created, containing developer knowledge of the bug; 3) developers often modify the tests to include new assertions or change the test code to reflect the changes in the source code; and 4) the performance of SBFL techniques degrades significantly (up to --415% for Mean First Rank) when evaluated on the bugs without developer knowledge. We provide a dataset of bugs without developer insights, aiding future SBFL evaluations in Defects4J and informing considerations for future bug benchmarks

A Survey on Automated Program Repair Techniques

With the rapid development and large-scale popularity of program software, modern society increasingly relies on software systems. However, the problems exposed by software have also come to the fore. Software defect has become an important factor troubling developers. In this context, Automated Program Repair (APR) techniques have emerged, aiming to automatically fix software defect problems and reduce manual debugging work. In particular, benefiting from the advances in deep learning, numerous learning-based APR techniques have emerged in recent years, which also bring new opportunities for APR research. To give researchers a quick overview of APR techniques' complete development and future opportunities, we revisit the evolution of APR techniques and discuss in depth the latest advances in APR research. In this paper, the development of APR techniques is introduced in terms of four different patch generation schemes: search-based, constraint-based, template-based, and learning-based. Moreover, we propose a uniform set of criteria to review and compare each APR tool, summarize the advantages and disadvantages of APR techniques, and discuss the current state of APR development. Furthermore, we introduce the research on the related technical areas of APR that have also provided a strong motivation to advance APR development. Finally, we analyze current challenges and future directions, especially highlighting the critical opportunities that large language models bring to APR research.Comment: This paper's earlier version was submitted to CSUR in August 202

Extending the Reach of Fault Localization to Assist in Automated Debugging

Software debugging is one of the most time-consuming tasks in modern software maintenance. To assist developers with debugging, researchers have proposed fault localization techniques. These techniques aim to automate the process of locating faults in software, which can greatly reduce debugging time and assist developers in understanding the faults. Effective fault localization is also crucial for automated program repair techniques, as it helps identify potential faulty locations for patching. Despite recent efforts to advance fault localization techniques, their effectiveness is still limited. With the increasing complexity of modern software, fault localization may not always provide direct identification of the root causes of faults. Further, there is a lack of studies on their application in modern software development. Most prior studies have evaluated these techniques in traditional software development settings, where only a single snapshot of the system is considered. However, modern software development often involves continuous and fine-grained changes to the system. This dissertation proposes a series of approaches to explore new automated debugging solutions that can enhance software quality assurance and reliability practices, with a specific focus on extending the reach of fault localization in modern software development. The dissertation begins with an empirical study on user-reported logs in bug reports, revealing that re-constructed execution paths from these logs provide valuable debugging hints. To further assist developers in debugging, we propose using static analysis techniques for information-retrieval and path-guided fault localization. By leveraging execution paths from logs in bug reports, we can improve the effectiveness of fault localization techniques. Second, we investigate the characteristics of operational data in continuous integration that can help capture faults early in the testing phase. As there is currently no available continuous integration benchmark that incorporates continuous test execution and failure, we present T-Evos, a dataset that comprises various operational data in continuous integration settings. We propose automated fault localization techniques that integrate change information from continuous integration settings, and demonstrate that leveraging such fine-grained change information can significantly improve their effectiveness. Finally, the dissertation investigates the data cleanness in fault localization by examining developers' knowledge in fault-triggering tests. The study reveals a significant degradation in the performance of fault localization techniques when evaluated on faults without developer knowledge. Through case studies and experiments, the proposed techniques in this dissertation significantly improve the effectiveness of fault localization and facilitate their adoption in modern software development. Additionally, this dissertation provides valuable insights into new debugging solutions for future research