3 research outputs found
Attention Please: Consider Mockito when Evaluating Newly Proposed Automated Program Repair Techniques
Automated program repair (APR) has attracted widespread attention in recent
years with substantial techniques being proposed. Meanwhile, a number of
benchmarks have been established for evaluating the performances of APR
techniques, among which Defects4J is one of the most wildly used benchmark.
However, bugs in Mockito, a project augmented in a later-version of Defects4J,
do not receive much attention by recent researches. In this paper, we aim at
investigating the necessity of considering Mockito bugs when evaluating APR
techniques. Our findings show that: 1) Mockito bugs are not more complex for
repairing compared with bugs from other projects; 2) the bugs repaired by the
state-of-the-art tools share the same repair patterns compared with those
patterns required to repair Mockito bugs; however, 3) the state-of-the-art
tools perform poorly on Mockito bugs (Nopol can only correctly fix one bug
while SimFix and CapGen cannot fix any bug in Mockito even if all the buggy
locations have been exposed). We conclude from these results that existing APR
techniques may be overfitting to their evaluated subjects and we should
consider Mockito, or even more bugs from other projects, when evaluating newly
proposed APR techniques. We further find out a unique repair action required to
repair Mockito bugs named external package addition. Importing the external
packages from the test code associated with the source code is feasible for
enlarging the search space and this action can be augmented with existing
repair actions to advance existing techniques.Comment: 10 pages, 4 figures and 7 table
How Different Is It Between Machine-Generated and Developer-Provided Patches? An Empirical Study on The Correct Patches Generated by Automated Program Repair Techniques
Background: Over the years, Automated Program Repair (APR) has attracted much
attention from both academia and industry since it can reduce the costs in
fixing bugs. However, how to assess the patch correctness remains to be an open
challenge. Two widely adopted ways to approach this challenge, including
manually checking and validating using automated generated tests, are biased
(i.e., suffering from subjectivity and low precision respectively). Aim: To
address this concern, we propose to conduct an empirical study towards
understanding the correct patches that are generated by existing
state-of-the-art APR techniques, aiming at providing guidelines for future
assessment of patches. Method: To this end, we first present a Literature
Review (LR) on the reported correct patches generated by recent techniques on
the Defects4J benchmark and collect 177 correct patches after a process of
sanity check. We investigate how these machine-generated correct patches
achieve semantic equivalence, but syntactic difference compared with
developer-provided ones, how these patches distribute in different projects and
APR techniques, and how the characteristics of a bug affect the patches
generated for it. Results: Our main findings include 1) we do not need to fix
bugs exactly like how developers do since we observe that 25.4% (45/177) of the
correct patches generated by APR techniques are syntactically different from
developer-provided ones; 2) the distribution of machine-generated correct
patches diverges for the aspects of Defects4J projects and APR techniques; and
3) APR techniques tend to generate patches that are different from those by
developers for bugs with large patch sizes. Conclusion: Our study not only
verifies the conclusions from previous studies but also highlights implications
for future study towards assessing patch correctness.Comment: 10+2 pages; accepted by The 13th ACM/IEEE International Symposium on
Empirical Software Engineering and Measurement (ESEM), Proto de Galinhas,
Brazil, 2019. arXiv admin note: text overlap with arXiv:1805.05983,
arXiv:1602.05643 by other author
Investigating and Recommending Co-Changed Entities for JavaScript Programs
JavaScript (JS) is one of the most popular programming languages due to its
flexibility and versatility, but maintaining JS code is tedious and
error-prone. In our research, we conducted an empirical study to characterize
the relationship between co-changed software entities (e.g., functions and
variables), and built a machine learning (ML)-based approach to recommend
additional entity to edit given developers' code changes. Specifically, we
first crawled 14,747 commits in 10 open-source projects; for each commit, we
created one or more change dependency graphs (CDGs) to model the
referencer-referencee relationship between co-changed entities. Next, we
extracted the common subgraphs between CDGs to locate recurring co-change
patterns between entities. Finally, based on those patterns, we extracted code
features from co-changed entities and trained an ML model that recommends
entities-to-change given a program commit. According to our empirical
investigation, (1) three recurring patterns commonly exist in all projects; (2)
80%--90% of co-changed function pairs either invoke the same function(s),
access the same variable(s), or contain similar statement(s); (3) our ML-based
approach CoRec recommended entity changes with high accuracy (73%--78%). CoRec
complements prior work because it suggests changes based on program syntax,
textual similarity, as well as software history; it achieved higher accuracy
than two existing tools in our evaluation