As software takes on more responsibility, it gets increasingly complex, requiring an extremely large number of tests for effective validation [1, 6]. Executing these large test suites is expensive, both in terms of time and energy. Cache misses are a significant contributing factor to execution time of software. In this paper, we propose an approach that helps order test executions in a test suite in such a way that instruction cache misses are reduced, and thereby execution time

Rajan, Ajitha

Stratis, Panagiotis

Panagiotis Stratis

Ajitha Rajan

Crossref

Reordering tests for faster test suite execution

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Reordering Tests for Faster Test Suite ExecutionCitation for published version:Stratis, P & Rajan, A 2018, Reordering Tests for Faster Test Suite Execution. in Proceedings of the 40thInternational Conference on Software Engineering: Companion Proceeedings. ACM, pp. 442-443, 40thInternational Conference on Software Engineering , Gothenburg, Sweden, 27-3 June. DOI:10.1145/3183440.3195048Digital Object Identifier (DOI):10.1145/3183440.3195048Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionPublished In:Proceedings of the 40th International Conference on Software Engineering: Companion ProceeedingsGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 30. Jun. 2018Reordering Tests for Faster Test Suite ExecutionPanagiotis Stratis, Ajitha RajanSchool of Informatics, University of EdinburghACM Reference Format:Panagiotis Stratis, Ajitha Rajan. 2018. Reordering Tests for Faster TestSuite Execution. In ICSE ’18 Companion: 40th International Conference onSoftware Engineering Companion, May 27-June 3, 2018, Gothenburg, Swe-den. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/3183440.31950481 INTRODUCTIONAs software takes on more responsibility, it gets increasingly com-plex, requiring an extremely large number of tests for effectivevalidation [1, 6]. Executing these large test suites is expensive, bothin terms of time and energy. Cache misses are a significant contribut-ing factor to execution time of software. In this paper, we proposean approach that helps order test executions in a test suite in such away that instruction cache misses are reduced, and thereby executiontime.We conduct an empirical evaluation with 20 subject programsand test suites from the SIR repository, EEMBC suite and LLVMSymbolizer, comparing execution times and cache misses with testorderings maximising instruction locality versus a traditional order-ing maximising coverage, as well as random permutations.Performance gains were considerable for programs and test suiteswhere the average number of different instructions executed betweentests was high. We achieved an average execution speedup of 6.83%and a maximum execution speedup of 17% over subject programswith differing control flow between test executions.2 APPROACHTo maximise temporal re-use of instructions across test executions,we minimise the distance between tests, where the distance betweentwo tests, Ti and Tj , is defined as:D (Ti ,Tj ) =#basic-blocks different between Ti and TjTotal #basic-blocks visited by all tests(1)We define this distance metric based on the idea that temporal in-struction locality would be enhanced if subsequent tests execute asmany common instructions as possible. For scalability reasons, weuse basic blocks instead of instructions. Using this distance metric,we perform nearest neighbour analysis in order to produce a testpermutation where the distance between subsequent tests is mini-mized. Our approach is illustrated in figure 1: The first step is todynamically analyse test executions and map them to their set of vis-ited basic blocks. We then compute the distance matrix and performnearest neighbour analysis. The starting test is the one with the mostunvisited neighbours.This optimisation algorithm, presented in [7], is quadratic incomplexity with respect to number of tests. The main computationalPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden© 2018 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-5663-3/18/05.https://doi.org/10.1145/3183440.3195048bottleneck is the calculation of the distance matrix which has N22complexity for a test suite with N tests. We found in our evaluationthat the optimisation algorithm was unable to scale beyond 14Ktests.To allow the optimisation to be scalable, we implemented anapproximate nearest neighbour algorithm which builds a multi-probelocality-sensitive hashing (LSH) index [3] instead of calculating thefull distance matrix. LSH is a technique for grouping points in high-dimensional space into buckets based on some distance measure(in our case the hamming distance). Our approximation algorithmoperates on the same input as the original algorithm but insteadof computing the distance matrix, we construct a multi-probe LSHindex. A starting test case is picked at random and the order is createdusing approximate nearest neighbour queried from LSH index untilthe index is empty.2.1 Test Analysis.For mapping each test to the set of its visited basic blocks we usedIntel’s Pin tool [2]. Pin is an instrumentation-based dynamic analysisframework which allows the development of customized dynamicprogram analysis tools (a.k.a Pintools). We developed a Pintoolthat records visited basic blocks for a program execution. Given aC/C++ program and its corresponding tests, our implementation willexecute each test independently and dynamically analyse it with ourPintool.2.2 Approximate Nearest Neighbour.For locality sensitive hashing we used the C++ implementationof FLANN [4], a library for performing fast approximate nearestneighbour in high dimensional spaces. In our configurations, we had12 hash tables, each with a key length of 20.Dynamic AnalysisNearest Neighbour AnalysisTest 1Test 2Test 3Test 4Test 2Test 4Test 1Test 3    DistanceMatrixComputationIntel Pin ToolOur Optimization Tool in C++11Approximate Nearest Neighbour Analysis (LSH)Test 2Test 4Test 3Test 1Our Approximation Tool in C++11/FLANNSet of visited basic blocks for each test caseFigure 1: ImplementationICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden Panagiotis Stratis, Ajitha Rajan3 EXPERIMENT RESULTSWe conduct our experiments over programs from different applica-tion domains to assess the execution performance of our test caseorderings. For each benchmark, we use four different types of testsuites and we measure their execution time:• Opt- Test suite ordered according to our optimisation algo-rithm.• Approx - Test suite ordered using our approximation algo-rithm.• BC - Test suite ordered greedily by the branch coveragemetric.• Random - We randomly permute the tests in the test suite.We generate 2000 random permutations per benchmark.Furthermore, we profile each permutation with Cachegrind that ispart of Valgrind [5] in order to measure instruction cache miss rate.Figure 2 shows the results for one fo the benchmarks, LLVMSymbolizer. The histogram frequencies for the 2000 random per-mutations, and 100 runs of each of Opt, Approx and BC are shown.LLVM Symbolizer showed significant execution speedup with bothOpt (17%) and Approx (13%) relative to BC. Furthermore, Optoutperformed 98% of the Random permutations, while Approx out-performed 88%. Median Opt performance was better than medianApprox by 4%.The premise in locality orderings (Opt and Approx) is that theywill reduce the number of instruction cache misses by increasingcache locality. This in turn will translate to faster, or reduced, ex-ecution time. We checked this premise for both Opt and Approxorderings by measuring instruction cache miss rates. We find thatthe reduction in execution times closely follows reduction in cachemisses, for the optimised orderings, relative to BC, over the subjectprograms.We also measured overhead incurred in ordering tests for Optand Approx. Overhead of Opt is quadratic in size of test suite anddoes not scale beyond 14K tests for EEMBC programs. Overhead ofApprox is considerably smaller and scales well to large test suites(70K tests for EEMBC). We found overhead could be further reducedwith the use of GPUs. It is worth noting that, ordering algorithmscan be performed offline and overhead need not be incurred duringactual test suite execution.Lastly, we find that average test distance is positively correlatedwith Approx execution time improvement (r = 0.76), as illustratedin figure 3. A higher average test distance indicates the differencesin instructions executed by tests in the test suite is higher. Orderingfor instruction cache locality has a higher impact on performancegains for such test suites since it ensures that tests wich execute toomany different instructions between them are not executed in succes-sion, avoiding cache misses that result from the difference. LLVMSymbolizer has the highest average TD among subject programs of42% and also the highest speedup with Approx of 13%.4 CONCLUSIONWe presented an approach for ordering tests to increase cache lo-cality across test executions. We conducted empirical evaluationsto assess execution speedups using the original approach and ap-proximation relative to random orderings and a greedy ordering forbranch coverage. We used programs from SIR, EEMBC benchmarksand an LLVM Symbolizer.Our evaluations revealed that ordering test executions to maximiseinstruction locality reduces cache misses and speeds up execution.Figure 2: Histogram frequencies of execution time for Opt,Approx, BC, Random Test Suites for LLVM SymbolizerFigure 3: Test Distance versus Time Improvement for Approxordering over BCBased on the results over our subject programs, we recommendthe Approx ordering of tests in a test suite since it achieves (1)Comparable execution speedups to Opt, and (2) Scales well to largenumbers of tests, as opposed to Opt. Overhead of Approx is lessthan that of Opt for large test suites and can be further reducedby running the algorithm on GPUs. For subject programs whoseexecutions fit in the cache, we found average test distance serves asa good guide for determining whether Approx ordering will resultin reasonable performance improvements.REFERENCES[1] Gregory Gay, Ajitha Rajan, Matt Staats, Michael Whalen, and Mats PE Heimdahl.2016. The effect of program and model structure on the effectiveness of mc/dc testadequacy coverage. ACM Transactions on Software Engineering and Methodology25, 3 (2016), 25.[2] Chi-Keung Luk et al. 2005. Pin: building customized program analysis tools withdynamic instrumentation. In Acm sigplan notices, Vol. 40. ACM, 190–200.[3] Qin Lv, William Josephson, Zhe Wang, Moses Charikar, and Kai Li. 2007. Multi-probe LSH: efficient indexing for high-dimensional similarity search. In Proceed-ings of the 33rd international conference on Very large data bases. 950–961.[4] Marius Muja and David G Lowe. 2012. Fast matching of binary features. InComputer and Robot Vision (CRV), 2012 Ninth Conference on. IEEE, 404–410.[5] Nicholas Nethercote and Julian Seward. 2007. Valgrind: a framework for heavy-weight dynamic binary instrumentation. In ACM Sigplan notices, Vol. 42. 89–100.[6] Ajitha Rajan. 2006. Coverage metrics to measure adequacy of black-box testsuites. In Automated Software Engineering, 2006. 21st IEEE/ACM InternationalConference on. IEEE, 335–338.[7] Panagiotis Stratis and Ajitha Rajan. 2016. Test case permutation to improve execu-tion time. In Automated Software Engineering, 2016 31st IEEE/ACM InternationalConference on. IEEE, 45–50.

Reordering Tests for Faster Test Suite Execution

     Edinburgh Research Explorer                                      Reordering Tests for Faster Test Suite ExecutionCitation for published version:Stratis, P & Rajan, A 2018, Reordering Tests for Faster Test Suite Execution. in Proceedings of the 40thInternational Conference on Software Engineering: Companion Proceeedings. ACM, pp. 442-443, 40thInternational Conference on Software Engineering , Gothenburg, Sweden, 27/05/18.https://doi.org/10.1145/3183440.3195048Digital Object Identifier (DOI):10.1145/3183440.3195048Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionPublished In:Proceedings of the 40th International Conference on Software Engineering: Companion ProceeedingsGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 28. Nov. 2025Reordering Tests for Faster Test Suite ExecutionPanagiotis Stratis, Ajitha RajanSchool of Informatics, University of EdinburghACM Reference Format:Panagiotis Stratis, Ajitha Rajan. 2018. Reordering Tests for Faster TestSuite Execution. In ICSE ’18 Companion: 40th International Conference onSoftware Engineering Companion, May 27-June 3, 2018, Gothenburg, Swe-den. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/3183440.31950481 INTRODUCTIONAs software takes on more responsibility, it gets increasingly com-plex, requiring an extremely large number of tests for effectivevalidation [1, 6]. Executing these large test suites is expensive, bothin terms of time and energy. Cache misses are a significant contribut-ing factor to execution time of software. In this paper, we proposean approach that helps order test executions in a test suite in such away that instruction cache misses are reduced, and thereby executiontime.We conduct an empirical evaluation with 20 subject programsand test suites from the SIR repository, EEMBC suite and LLVMSymbolizer, comparing execution times and cache misses with testorderings maximising instruction locality versus a traditional order-ing maximising coverage, as well as random permutations.Performance gains were considerable for programs and test suiteswhere the average number of different instructions executed betweentests was high. We achieved an average execution speedup of 6.83%and a maximum execution speedup of 17% over subject programswith differing control flow between test executions.2 APPROACHTo maximise temporal re-use of instructions across test executions,we minimise the distance between tests, where the distance betweentwo tests, Ti and Tj , is defined as:D (Ti ,Tj ) =#basic-blocks different between Ti and TjTotal #basic-blocks visited by all tests(1)We define this distance metric based on the idea that temporal in-struction locality would be enhanced if subsequent tests execute asmany common instructions as possible. For scalability reasons, weuse basic blocks instead of instructions. Using this distance metric,we perform nearest neighbour analysis in order to produce a testpermutation where the distance between subsequent tests is mini-mized. Our approach is illustrated in figure 1: The first step is todynamically analyse test executions and map them to their set of vis-ited basic blocks. We then compute the distance matrix and performnearest neighbour analysis. The starting test is the one with the mostunvisited neighbours.This optimisation algorithm, presented in [7], is quadratic incomplexity with respect to number of tests. The main computationalPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden© 2018 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-5663-3/18/05.https://doi.org/10.1145/3183440.3195048bottleneck is the calculation of the distance matrix which has N22complexity for a test suite with N tests. We found in our evaluationthat the optimisation algorithm was unable to scale beyond 14Ktests.To allow the optimisation to be scalable, we implemented anapproximate nearest neighbour algorithm which builds a multi-probelocality-sensitive hashing (LSH) index [3] instead of calculating thefull distance matrix. LSH is a technique for grouping points in high-dimensional space into buckets based on some distance measure(in our case the hamming distance). Our approximation algorithmoperates on the same input as the original algorithm but insteadof computing the distance matrix, we construct a multi-probe LSHindex. A starting test case is picked at random and the order is createdusing approximate nearest neighbour queried from LSH index untilthe index is empty.2.1 Test Analysis.For mapping each test to the set of its visited basic blocks we usedIntel’s Pin tool [2]. Pin is an instrumentation-based dynamic analysisframework which allows the development of customized dynamicprogram analysis tools (a.k.a Pintools). We developed a Pintoolthat records visited basic blocks for a program execution. Given aC/C++ program and its corresponding tests, our implementation willexecute each test independently and dynamically analyse it with ourPintool.2.2 Approximate Nearest Neighbour.For locality sensitive hashing we used the C++ implementationof FLANN [4], a library for performing fast approximate nearestneighbour in high dimensional spaces. In our configurations, we had12 hash tables, each with a key length of 20.Dynamic AnalysisNearest Neighbour AnalysisTest 1Test 2Test 3Test 4Test 2Test 4Test 1Test 3    DistanceMatrixComputationIntel Pin ToolOur Optimization Tool in C++11Approximate Nearest Neighbour Analysis (LSH)Test 2Test 4Test 3Test 1Our Approximation Tool in C++11/FLANNSet of visited basic blocks for each test caseFigure 1: ImplementationICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden Panagiotis Stratis, Ajitha Rajan3 EXPERIMENT RESULTSWe conduct our experiments over programs from different applica-tion domains to assess the execution performance of our test caseorderings. For each benchmark, we use four different types of testsuites and we measure their execution time:• Opt- Test suite ordered according to our optimisation algo-rithm.• Approx - Test suite ordered using our approximation algo-rithm.• BC - Test suite ordered greedily by the branch coveragemetric.• Random - We randomly permute the tests in the test suite.We generate 2000 random permutations per benchmark.Furthermore, we profile each permutation with Cachegrind that ispart of Valgrind [5] in order to measure instruction cache miss rate.Figure 2 shows the results for one fo the benchmarks, LLVMSymbolizer. The histogram frequencies for the 2000 random per-mutations, and 100 runs of each of Opt, Approx and BC are shown.LLVM Symbolizer showed significant execution speedup with bothOpt (17%) and Approx (13%) relative to BC. Furthermore, Optoutperformed 98% of the Random permutations, while Approx out-performed 88%. Median Opt performance was better than medianApprox by 4%.The premise in locality orderings (Opt and Approx) is that theywill reduce the number of instruction cache misses by increasingcache locality. This in turn will translate to faster, or reduced, ex-ecution time. We checked this premise for both Opt and Approxorderings by measuring instruction cache miss rates. We find thatthe reduction in execution times closely follows reduction in cachemisses, for the optimised orderings, relative to BC, over the subjectprograms.We also measured overhead incurred in ordering tests for Optand Approx. Overhead of Opt is quadratic in size of test suite anddoes not scale beyond 14K tests for EEMBC programs. Overhead ofApprox is considerably smaller and scales well to large test suites(70K tests for EEMBC). We found overhead could be further reducedwith the use of GPUs. It is worth noting that, ordering algorithmscan be performed offline and overhead need not be incurred duringactual test suite execution.Lastly, we find that average test distance is positively correlatedwith Approx execution time improvement (r = 0.76), as illustratedin figure 3. A higher average test distance indicates the differencesin instructions executed by tests in the test suite is higher. Orderingfor instruction cache locality has a higher impact on performancegains for such test suites since it ensures that tests wich execute toomany different instructions between them are not executed in succes-sion, avoiding cache misses that result from the difference. LLVMSymbolizer has the highest average TD among subject programs of42% and also the highest speedup with Approx of 13%.4 CONCLUSIONWe presented an approach for ordering tests to increase cache lo-cality across test executions. We conducted empirical evaluationsto assess execution speedups using the original approach and ap-proximation relative to random orderings and a greedy ordering forbranch coverage. We used programs from SIR, EEMBC benchmarksand an LLVM Symbolizer.Our evaluations revealed that ordering test executions to maximiseinstruction locality reduces cache misses and speeds up execution.Figure 2: Histogram frequencies of execution time for Opt,Approx, BC, Random Test Suites for LLVM SymbolizerFigure 3: Test Distance versus Time Improvement for Approxordering over BCBased on the results over our subject programs, we recommendthe Approx ordering of tests in a test suite since it achieves (1)Comparable execution speedups to Opt, and (2) Scales well to largenumbers of tests, as opposed to Opt. Overhead of Approx is lessthan that of Opt for large test suites and can be further reducedby running the algorithm on GPUs. For subject programs whoseexecutions fit in the cache, we found average test distance serves asa good guide for determining whether Approx ordering will resultin reasonable performance improvements.REFERENCES[1] Gregory Gay, Ajitha Rajan, Matt Staats, Michael Whalen, and Mats PE Heimdahl.2016. The effect of program and model structure on the effectiveness of mc/dc testadequacy coverage. ACM Transactions on Software Engineering and Methodology25, 3 (2016), 25.[2] Chi-Keung Luk et al. 2005. Pin: building customized program analysis tools withdynamic instrumentation. In Acm sigplan notices, Vol. 40. ACM, 190–200.[3] Qin Lv, William Josephson, Zhe Wang, Moses Charikar, and Kai Li. 2007. Multi-probe LSH: efficient indexing for high-dimensional similarity search. In Proceed-ings of the 33rd international conference on Very large data bases. 950–961.[4] Marius Muja and David G Lowe. 2012. Fast matching of binary features. InComputer and Robot Vision (CRV), 2012 Ninth Conference on. IEEE, 404–410.[5] Nicholas Nethercote and Julian Seward. 2007. Valgrind: a framework for heavy-weight dynamic binary instrumentation. In ACM Sigplan notices, Vol. 42. 89–100.[6] Ajitha Rajan. 2006. Coverage metrics to measure adequacy of black-box testsuites. In Automated Software Engineering, 2006. 21st IEEE/ACM InternationalConference on. IEEE, 335–338.[7] Panagiotis Stratis and Ajitha Rajan. 2016. Test case permutation to improve execu-tion time. In Automated Software Engineering, 2016 31st IEEE/ACM InternationalConference on. IEEE, 45–50.

English

https://www.pure.ed.ac.uk/ws/files/64318522/Reordering_tests_for_faster_test_suite_execution_2018.pdf

Reordering Tests for Faster Test Suite Execution

Abstract

Similar works

Full text

Available Versions

Crossref

Edinburgh Research Explorer

Edinburgh Research Explorer