Professional software developers spend a significant amount of
time fixing builds, but this has received little attention as a problem in automatic program repair. We present a new deep learning
architecture, called Graph2Diff, for automatically localizing and
fixing build errors. We represent source code, build configuration
files, and compiler diagnostic messages as a graph, and then use a
Graph Neural Network model to predict a diff. A diff specifies how
to modify the code’s abstract syntax tree, represented in the neural
network as a sequence of tokens and of pointers to code locations.
Our network is an instance of a more general abstraction which we
call Graph2Tocopo, which is potentially useful in any development
tool for predicting source code changes. We evaluate the model on
a dataset of over 500k real build errors and their resolutions from
professional developers. Compared to the approach of DeepDelta
[23], our approach tackles the harder task of predicting a more
precise diff but still achieves over double the accuracy

Aftandilian, E

Chen, Z

Manzagol, PA

Moitra, S

Rice, Andrew

Sutton, C

Tarlow, D

English

Rice, A

Apollo (Cambridge)

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116Learning to Fix Build Errors with Graph2Diff Neural NetworksACM Reference Format:. 2019. Learning to Fix Build Errors with Graph2Diff Neural Networks. In .ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/1122445.11224561 INTRODUCTIONProfessional software developers spend a significant amount oftime fixing builds; for example, one large-scale study found thatdevelopers build their code 7–10 times per day [12], with a signif-icant number of builds being unsuccessful. Build errors includesimple errors such as syntax errors, but for professional developersthese are a small minority of errors; instead, the majority are link-ing errors such as unresolved symbols, type errors, and incorrectbuild dependencies [12]. A recent paper by Google reports roughly10 developer-months of effort are spent every month fixing smallbuild errors [10]. Therefore, automatically repairing build errors isa research problem that has potential to ease a frequent pain pointin the developer workflow. Happily, there are good reasons to thinkthat automatic build repair is feasible: fixes are often short, and wecan test a proposed fix before showing it to a developer simply byrebuilding the project. Build repair is thus a potential “sweet spot”for automatic program repair, hard enough to require new researchideas, but still just within reach.However, there is only a small literature on repairing build errors.Previous work has successfully repaired syntax errors like missingdelimiters and parentheses [4, 5], but in our corpus of professionalbuild errors (Table 1), fixes are more subtle, often requiring detailedinformation about the project APIs and dependencies. Recently, theDeepDelta system [10] aimed to repair build errors by applyingneural machine translation (NMT), translating the text of the diag-nostic message to a description of the repair. Although this work ispromising, the use of an off-the-shelf NMT system severely limitsthe types of build errors that it can handle.We introduce a new deep learning architecture, calledGraph2Diffnetworks, specifically for the problem of predicting edits to sourcecode, as a replacement for the celebrated sequence-to-sequencemodel used for machine translation. Graph2Diff networks map agraph representation of the broken code to a diff 1 in a domain-specific language that describes the repair. The diff can contain notonly tokens, but also pointers into the input graph (such as “inserttoken HERE”) and copy instructions (i.e. “copy a token from HEREin the input graph”). Thus, Graph2Diff networks combine, extend,1We slightly abuse terminology here and use “diff” to mean a sequence of edit opera-tions that can be applied to the broken AST to obtain the fixed AST.Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.ICSE ’20, May 23-29, 2020, Seoul, South Korea© 2019 Association for Computing Machinery.ACM ISBN 0000. . . $15.00https://doi.org/10.1145/1122445.1122456and generalize a number of recent ideas from neural network mod-els for source code [1, 2, 14]. We have obtained promising resultson applying Graph2Diff networks for automatically repairing builderrors, and more broadly, there is exciting potential that a largenumber of tasks across software engineering can be profitably castas Graph2Diff networks.2 METHODWe formulate the problem of resolving a set of build diagnostics asa supervised machine learning problem. The input is the state ofthe source tree at the time of a broken build and a list of diagnosticsreturned by the compiler. The target output is a diff that can beapplied to the code to resolve all of the diagnostics. For our purposes,a diff is a sequence of transformations to apply to the original sourcecode to generate the repaired version. Compiler diagnostics do notalways identify the source of the fault that needs to be repaired, sowe require the models to predict the locations that need changingin addition to the repairs. This combines the well-studied problemsof automated fault localization and automated program repair.We collected a large dataset of 500k fixes by taking advantage of ahighly-instrumented development process similar to that at Google,extending previous work by [10].2 Examples from the dataset ap-pear in Figure 1. An example edit script that Graph2Diff networksare trained to output appears in Figure 2.At a high level, Graph2Diff networks learn to map the brokenversion of the code, represented as a graph, to an edit script thatrepresents the fix. Graph2Diff is based on three key architecturalideas from deep learning: First, graph neural networks [8, 11] can ex-plicitly encode syntactic structure, semantic information, and eveninformation from program analysis in a form that neural networkscan understand, allowing the network to learn to change one partof the code based on its relationship to another part of the code. Forus, the graph represents the input to the build repair task, includingthe broken abstract syntax tree (AST) and the build diagnostics.Second, pointer models [13] can generate locations in the initial ASTto be edited, which leads to a compact way of generating changesto large files (as diffs). Much work on program repair divides theproblem into two separate steps of fault localization and generatingthe repair; unfortunately, fault localization is a difficult problem[9]. Using pointers, the machine learning component can predictboth where and how to fix. Finally, the copy mechanism addressesthe well-known out-of-vocabulary problem of source code [3, 6, 7]:source code projects often include project-specific identifiers thatdo not occur in the training set of a model. A copy mechanism canlearn to copy any needed identifiers from the broken source code,even if they do not occur in the training set.3 RESULTSWe follow standard machine learning experimental details, usingtrain, validation and test splits and grid search for choosing hy-perparameters. Our main metric is sequence-level accuracy, which2Further details omitted for anonymity.1117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174ICSE ’20, May 23-29, 2020, Seoul, South Korea175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254Row Diagnostics FixA cannot find symbol ‘widgetSet()’ - widgetsForX.widgetSet ().stream ().forEach(+ widgetCounts.widgetSet ().stream ().forEach(B cannot find symbol ‘of’ - Framework.createWidget(new FooModule.of(this)).add(this);+ Framework.createWidget(FooModule.of(this)).add(this);C (line 10) cannot find symbol ‘longName’(line 15) cannot find symbol ‘longName’- String longname = "reallylongstringabcdefghijklmnopqrstuvw ..."+ String longName = "reallylongstringabcdefghijklmnopqrstuvw ..."x = f(longName) // (line 10) Diagnostic pointed here// ...y = g(a, b, c, longName) // (line 15) Diagnostic pointed hereD incompatible types: ParamsBuilder cannot beconverted to Optional<Params>- return new ParamsBuilder(args);+ return new ParamsBuilder(args).build();E incompatible types: RpcFuture<LongNameResponse>cannot be converted to LongNameResponse- LongNameResponse produceLongNameResponseFromX(+ ListenableFuture <LongNameResponse > produceLongNameResponseFromX(F incompatible types: WidgetUnit cannot beconverted to Long- public Widget setDefaultWidgetUnit(WidgetUnit defaultUnit) {+ public Widget setDefaultWidgetUnit(Long defaultUnit) {this.defaultUnit = defaultUnit; // Diagnostic pointed hereG cannot find symbol ‘of(Widget,Widget)’- import com.google.common.collect.ImmutableCollection;+ import com.google.common.collect.ImmutableSet;// ...- ImmutableCollection.of(+ ImmutableSet.of(Figure 1: Example diagnostics and fixes from the dataset.- LongNameResponse produceLongNameResponseFromX(+ ListenableFuture <LongNameResponse > produceLongNameResponseFromX(0: MethodDef1: Type2: LongNameResponse3: Id4: produceLongNameResponseFormX5: Args0:INSERT 1:INPUT_POINTER(1) 2:INPUT_POINTER(2) 3:PARAMETERIZED_TYPE 4:TYPEAPPLY5:INSERT 6:OUTPUT_POINTER(0) 7:FIRST_CHILD 8:IDENTIFIER 9:ListenableFuture10:INSERT 11:OUTPUT_POINTER(0) 12:OUTPUT_POINTER(5) 13:IDENTIFIER 14:LongNameResponse15:DELETE 16:INPUT_POINTER(2)17:DONEFigure 2: An example edit script that makes the change specified in Figure 1E. (Top) a textual diff of the change, (Left) A subsetof the original AST annotated with input ids. (Right) An edit script implementing the change, annotated with output ids.is how often the model predicts the full sequence correctly. Thisis a strict metric that only gives credit for exactly matching thedeveloper’s change.We compare to the most closely related work, which is Deep-Delta [10]. Even though DeepDelta can only be evaluated on aless-stringent task than exact developer fix match (because it hasno pointer mechanism to precisely specify locations), we find thatGraph2Diff networks achieve 26% accuracy versus DeepDelta achiev-ing 10% accuracy when predicting less precise diffs. We interpretthis result as showing promise about the potential of Graph2Diffnetworks for repairing broken builds, and also for code-editingtasks including other forms of program repair and beyond.REFERENCES[1] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learningto represent programs with graphs. In ICLR.[2] Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional at-tention network for extreme summarization of source code. In InternationalConference on Machine Learning. 2091–2100.[3] Miltiadis Allamanis and Charles Sutton. 2013. Mining source code repositoriesat massive scale using language modeling. In Proceedings of the 10th WorkingConference on Mining Software Repositories. IEEE Press, 207–216.[4] Joshua Charles Campbell, Abram Hindle, and José Nelson Amaral. 2014. Syntaxerrors just aren’t natural: improving error reporting with language models. InWorking Conference on Mining Software Repositories (MSR). 252–261.[5] Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. Deepfix: Fix-ing common C language errors by deep learning. In Thirty-First AAAI Conferenceon Artificial Intelligence.[6] Vincent J. Hellendoorn and Premkumar Devanbu. 2017. Are Deep Neural Net-works the Best Choice for Modeling Source Code?. In Proceedings of the 2017 11thJoint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, NewYork, NY, USA, 763–773. https://doi.org/10.1145/3106237.3106290[7] Rafael-Michael Karampatsis and Charles Sutton. 2019. Maybe Deep NeuralNetworks are the Best Choice for Modeling Source Code. CoRR abs/1903.05734(2019). arXiv:1903.05734 http://arxiv.org/abs/1903.05734[8] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2016. Gatedgraph sequence neural networks. In ICLR.[9] Kui Liu, Anil Koyuncu, Tegawendé F Bissyandé, Dongsun Kim, Jacques Klein,and Yves Le Traon. 2019. You cannot fix what you cannot find! an investigationof fault localization bias in benchmarking automated program repair systems. In2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST).IEEE, 102–113.[10] Ali Mesbah, Andrew Rice, Emily Johnston, Nick Glorioso, and Edward Aftandilian.2019. DeepDelta: Learning To Repair Compilation Errors. In Proceedings of the2019 27th ACM Joint Meeting on European Software Engineering Conference andSymposium on the Foundations of Software Engineering (ESEC/FSE 2019). ACM,New York, NY, USA.[11] Franco Scarselli, MarcoGori, AhChung Tsoi, MarkusHagenbuchner, andGabrieleMonfardini. 2009. The graph neural network model. IEEE Transactions on NeuralNetworks (2009).[12] Hyunmin Seo, Caitlin Sadowski, Sebastian Elbaum, Edward Aftandilian, andRobert Bowdidge. 2014. Programmers’ build errors: A case study (at Google). InInternational Conference on Software Engineering. ACM, 724–734.[13] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer Networks. InAdvances in Neural Information Processing Systems 28, C. Cortes, N. D. Lawrence,D. D. Lee, M. Sugiyama, and R. Garnett (Eds.). Curran Associates, Inc., 2692–2700.http://papers.nips.cc/paper/5866-pointer-networks.pdf[14] Rui Zhao, David Bieber, Kevin Swersky, andDaniel Tarlow. 2019. Neural Networksfor Modeling Source Code Edits. arXiv preprint arXiv:1904.02818 (2019).2

Learning to Fix Build Errors with Graph2Diff Neural Networks

https://www.repository.cam.ac.uk/bitstream/1810/312420/2/apr20-paper63.pdf

Learning to Fix Build Errors with Graph2Diff Neural Networks

Abstract

Similar works

Full text

Available Versions

Apollo (Cambridge)