Beszédes, Árpád

Di Penta, Massimiliano

Gyimóthy, Tibor

Horváth, Ferenc

Repository of the Academy's Library

Leveraging Contextual Information from FunctionCall Chains to Improve Fault LocalizationA´rpa´d Besze´desSoftware Engineering DepartmentUniversity of SzegedSzeged, Hungarybeszedes@inf.u-szeged.huFerenc Horva´thSoftware Engineering DepartmentUniversity of SzegedSzeged, Hungaryhferenc@inf.u-szeged.huMassimiliano Di PentaDepartment of EngineeringUniversity of SannioBenevento, Italydipenta@unisannio.itTibor Gyimo´thySoftware Engineering DepartmentUniversity of SzegedSzeged, Hungarygyimothy@inf.u-szeged.huAbstract—In Spectrum-Based Fault Localization, program el-ements such as statements or functions are ranked accordingto a suspiciousness score which can guide the programmer infinding the fault more efficiently. However, such a ranking doesnot include any additional information about the element underinvestigation. In this work, we propose to complement function-level spectrum based fault localization with function call chains– i.e., snapshots of the call stack occurring during execution – onwhich the fault localization is first performed, and then narroweddown to functions. Our experiments using defects from fourDefects4J programs show that (i) 84% of the defective functionscan be found in call chains with highest scores, (ii) the proposedapproach improves Ochiai ranking of 1 to 6 positions on average,with a relative improvement of 45%, and (iii) the improvementis substantial when Ochiai produces bad rankings.Index Terms—Spectrum Based Fault Localization, function callchains, call stack trace, testing and debugging.I. PROBLEM ADDRESSEDThis work deals with Spectrum-Based Fault Localization(SBFL) [1], a class of Fault Localization (FL) methods, whosepurpose is to aid debugging by finding the root causes of anobserved failure.There has been a lot of research performed with variousSBFL algorithms, and so far variations to these approachesmay yield only marginal improvements without involvingadditional information to the process. Additional informationcan either be feedback from the user or should go beyond thesimple hit-based spectrum on basic code elements and can beused as a context for the suspicious elements. Early attempts toincorporate control or data flow information, for instance [2],have not been further developed because it soon becameapparent that they are difficult to scale to large programs andreal defects.In this work, we propose to enhance traditional SBFL withfunction call chains on which the FL is performed. Functioncall chains are snapshots of the call stack occurring duringexecution and as such can provide valuable context to thefault being traced. Call chains (and call stack traces) areartifacts occurring during program execution which are well-known to programmers who perform debugging and can show,A´rpa´d Besze´des was supported by the Ja´nos Bolyai Research Scholarship ofthe Hungarian Academy of Sciences. Ministry of Human Capacities, Hungarygrant 20391-3/2018/FEKUSTRAT is acknowledged.for instance, that a function may fail if called from oneplace and perform successfully when called from another.There is empirical evidence that stack traces help developersfix bugs [3], and locate crash-faults [4], for instance. Callchains provide a context about the possible failures, whichcan complement the basic ranking lists of program elementsor, in some cases, replace them.II. APPROACHFig. 1 provides a high-level overview of our approach. Usinga given set of test cases T , the subject program P is executedwhile collecting the necessary execution trace information.This is used to produce the function call chains, as well asthe test case pass/fail outcomes. Based on that, we computethe call chain level program spectrum information, which isused to calculate the ranking of the chains according to theirsuspiciousness levels. In the next step, two algorithms areapplied to compute the ranking of the functions for FL, whichare then merged together to produce the final ranking.Fig. 1. Call chain based FL overview.Let F be the set of functions in a program P , and T aset of test cases used to test P . Then, a Call Chain c is asequence of functions f1 → f2 → ⋅ ⋅ ⋅ → fn (fi ∈ F ), whichoccur during the execution of some test case t ∈ T , and forwhich: (i) f1 is the entry point called by t, (ii) each fi directlycalls fi+1 (0 < i < n), and (iii) fn returns without callingfurther functions in that sequence.TABLE IFAULT LOCALIZATION EFFECTIVENESS COMPARISON (AVERAGES SHOWN).Program Bugs Ochiai E(E′) Comb. E(E′) Diff. E(E′) Rel. change Ochiai> 10 Enabling impr. Rel. impr.Commons Lang 46 4.7 (0.23%) 3.9 (0.19%) -0.8 (-0.04%) -17% 6 4 ( 9%) -15.1 (-67%)Commons Math 74 8.7 (0.23%) 3.8 (0.11%) -4.9 (-0.12%) -56% 18 15 (20%) -24.3 (-87%)JFreeChart 18 5.3 (0.12%) 3.4 (0.08%) -1.9 (-0.04%) -36% 2 2 (11%) -19.0 (-76%)Joda-Time 23 13.4 (0.38%) 7.8 (0.22%) -5.6 (-0.16%) -42% 7 4 (17%) -43.1 (-88%)Total / Average 161 7.8 (0.24%) 4.3 (0.14%) -3.5 (-0.10%) -45% 33 25 (16%) -25.4 (-84%)FL on the call chains takes as inputs the test case executionoutcomes (pass/fail) and uses a program spectrum representa-tion with the chains as code elements. The output is a rankedlist of call chains with the associated suspiciousness scores.We apply a traditional program spectrum representation basedon binary matrices, and for SBFL, any existing suspicious-ness score could be used. We report results with the Ochiaiscore [5], which outperforms other scores.A trivial approach for the user to locate the defectivefunction (and statement, respectively) is to consider the highestranked call chains and investigate the functions occurring inthem. We also propose an approach to produce a ranked listfor functions as well based on the call chain scores. It consistsof executing two competing function ranking algorithms andthen combining their results, as follows.1) Weighted chain counts: In this strategy, we count thenumber of occurrences of each function in the chains weightedby the respective chain scores from the previous phase. Theintuition behind this is that functions frequently occurring inhighly ranked chains will be more suspicious.2) Reapplied spectrum: Here, we re-apply the spectrum-based approach, but this time on the functions using the callchains in place of the test cases. For this purpose, we treat acall chain as a “proxy” to a test case in the following manner.If its score is greater than a threshold z ∈ [0,1) it is treatedas “failing” otherwise as “passing.” The final scores, in thiscase, will be computed by re-applying the Ochiai formula tothis function-level spectrum.3) Merging the ranks: In the final step, we merge the tworanked lists by alternatively selecting the next element fromeach of the two lists.III. RESULTS SO FAROur research question is the following: How much improve-ment can the call chain-based approach achieve compared tobasic function-level fault localization?We performed the experiments on real defects from fourprograms of the Defects4J suite (v1.0.0) [6]. For computingthe effectiveness of an SBFL approach, we follow the strategyto look at “elements that need to be investigated” using the“expected case” in the case of ties and express this in a set ofmeasures called Expense. We use two variants of the measure:an absolute one expressed in the number of code elements (E)and a relative version compared to the length of the rank list(E′).Apart from the general average change, we define thenotion of enabling improvement, an improvement in which thetraditional SBFL algorithm ranks the faulty element beyondthe 10th position but the proposed approach reaches it inat most 10 steps. In a “hopeless” localization scenario, ourapproach enables the user to localize the fault by inspectingonly the top elements in the list.Table I reports the results for FL effectiveness. Columns“Ochiai” and “Comb” show the absolute and relative Expensevalues for function-level Ochiai and for the proposed approach,respectively. Column “Diff” reports the difference between theaverage rankings, while column “Rel. change” expresses thesame as percentage increase/decrease with respect to Ochiai.Column “Ochiai > 10” reports the number of defects in theprograms for which the ranking position is more than 10. “En-abling impr.” indicates how many defects were successfullymoved to the 10th or below position (relative to bug number),and the last column shows the average ranking absolute andrelative difference for such cases.For each program, the improvement in terms of the Expensemetric ranges from one to about six ranking positions on aver-age. The relative change ranges from 17% to 56%, and 45% onthe overall evaluation set. Noticeably, the proposed approach isable to achieve a very good improvement when Ochiai scoresa bad ranking position of the correct recommendation, i.e.,> 10th. More specifically, about 76% of such defects obtainedan enabling improvement in terms of ranking (25 out of the33), and in this case, the relative Expense improvement waseven higher than the overall average, 84%.REFERENCES[1] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey onsoftware fault localization,” IEEE Transactions on Software Engineering,vol. 42, no. 8, pp. 707–740, 2016.[2] M. J. Harrold, G. Rothermel, K. Sayre, R. Wu, and L. Yi, “An empiricalinvestigation of the relationship between spectra differences and regres-sion faults,” Software Testing, Verification and Reliability, vol. 10, no. 3,pp. 171–194, 2000.[3] A. Schro¨ter, N. Bettenburg, and R. Premraj, “Do stack traces helpdevelopers fix bugs?” in 2010 7th IEEE Working Conference on MiningSoftware Repositories (MSR 2010), May 2010, pp. 118–121.[4] D. Zou, J. Liang, Y. Xiong, M. D. Ernst, and L. Zhang, “An empiricalstudy of fault localization families and their combinations,” IEEE Trans-actions on Software Engineering, 2019.[5] R. Abreu, P. Zoeteweij, R. Golsteijn, and A. J. C. van Gemund, “Apractical evaluation of spectrum-based fault localization,” J. Syst. Softw.,vol. 82, no. 11, pp. 1780–1792, Nov. 2009.[6] R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A database of existingfaults to enable controlled testing studies for java programs,” in Pro-ceedings of the 2014 International Symposium on Software Testing andAnalysis. ACM, 2014, pp. 437–440.

Leveraging Contextual Information from Function Call Chains to Improve Fault Localization

Abstract

Similar works

Full text

Available Versions

Repository of the Academy's Library