From September 24th to September 29th 2008 the Dagstuhl Seminar
08351 ``Evolutionary Test Generation \u27\u27 was held
in Schloss Dagstuhl~--~Leibniz Center for Informatics.
During the seminar, several participants presented their current
research, and ongoing work and open problems were discussed. Abstracts of
the presentations given during the seminar as well as abstracts of
seminar results and ideas are put together in this paper. The first section
describes the seminar topics and goals in general.
Links to extended abstracts or full papers are provided, if available

Schlingloff, Holger

Vos, Tanja E. J.

Wegener, Joachim

Dagstuhl Research Online Publication Server

08351 Abstracts CollectionEvolutionary Test Generation Dagstuhl Seminar Holger Schlingloﬀ1, Tanja E.J. Vos2 and Joachim Wegener31 Fraunhofer Institut - Berlin, Germanyhs@informatik.hu-berlin.de2 Universidad Politècnica de Valencia, Spaintvos@dsic.upv.es3 Berner & Mattner Systemtechnik - Berlin, Germanyjoachim.wegener@berner-mattner.comAbstract. From September 24th to September 29th 2008 the DagstuhlSeminar 08351 Evolutionary Test Generation  was held in SchlossDagstuhl  Leibniz Center for Informatics. During the seminar, severalparticipants presented their current research, and ongoing work and openproblems were discussed. Abstracts of the presentations given during theseminar as well as abstracts of seminar results and ideas are put togetherin this paper. The ﬁrst section describes the seminar topics and goals ingeneral. Links to extended abstracts or full papers are provided, if avail-able.Keywords. Software-testing, evolutionary algoritms, meta-heuristic search08351 Summary  Evolutionary Test GenerationFrom September 24th to September 29th 2008 the Dagstuhl Seminar 08351 Evo-lutionary Test Generation  was held in Schloss Dagstuhl  Leibniz Center forInformatics. During the seminar, several participants presented their current re-search, and ongoing work and open problems were discussed. This paper containsan executive summary of the seminar and the open problems that were found.Keywords: Software-testing, evolutionary algoritms, meta-heuristic searchJoint work of: Schlingloﬀ, Holger; Vos, Tanja; Wegener, JoachimFull Paper: http://drops.dagstuhl.de/opus/volltexte/2009/2022SAT-based Automatic Test Pattern GenerationDue to the rapidly growing size of integrated circuits, there is a need for newalgorithms for Automatic Test Pattern Generation (ATPG). While classical al-gorithms reach their limit, there have been recent advances in algorithms toDagstuhl Seminar Proceedings 08351Evolutionary Test Generationhttp://drops.dagstuhl.de/opus/volltexte/2009/20232 Holger Schlingloﬀ, Tanja Vos and Joachim Wegenersolve Boolean Satisﬁability (SAT). Because Boolean SAT solvers are workingon Conjunctive Normal Forms (CNF), the problem has to be transformed. Dur-ing transformation, relevant information about the problem might get lost andtherefore is not available in the solving process.In the following we brieﬂy motivate the problem and provide the latest devel-opments in the ﬁeld. The technique was implemented and experimental resultsare presented. The approach was combined with the ATPG framework of NXPSemiconductors. Signiﬁcant improvements in overall performance and robustnessare demonstrated.Keywords: Circuit, ATPG, SAT, Boolean SatisﬁabilityJoint work of: Drechsler, Rolf; Eggersglüß, Stephan; Fey, Görschwin; Tille,DanielFull Paper: http://drops.dagstuhl.de/opus/volltexte/2009/2015See also: 1. Drechsler, R., Eggersglüß, S., Fey, G., Glowatz, A., Hapke, F.,Schlöﬀel, J., Tille, D.: On acceleration of SAT-based ATPG for industrial de-signs. IEEE Transactions on Computer Aided Design of Circuits and Systems 27(2008) 1329-1333 2. Drechsler, R., Eggersglüß, S., Fey, G., Tille, D.: Test PatternGeneration using Boolean Proof Engines. Springer (2009) to appear.Ranking schemes for test cases in worst-case coevolutionJürgen Branke (Universität Karlsruhe, DE)In this paper, we consider the case of worst-case optimization, i.e., the user isinterested in a solution's performance in the worst case only. If the number ofpossible test cases is large, it is an optimization problem by itself to determine asolution's worst case performance. In this paper, we apply coevolutionary algo-rithms to co-evolve the worst case test cases along with the solution candidates.We propose a number of new variants of coevolutionary algorithms, and showthat these techniques outperform previously proposed coevolutionary worst-caseoptimizers on some simple test problems.Keywords: Coevolution, worst-case optimization, evolutionary algorithmFull Paper:http://dx.doi.org/10.1007/978-3-540-87700-4_15Stressing and StretchingJohn Clark (University of York, GB)In this brief presentation I talk about evolving input distributions and also waysof distorting programs to make hard to reach paths/states easier.Evolutionary Test Generation 3The key diﬀerence to previous work is that rather than evolve test data, weevolve strategies for generating test data. In one approach, inspired by sheetmusic, we can evolve a "tune" (comprising notes that are server requets) tostress the system. Another approach seeks to learn an input data distributionto maximise the coverage probability for the least covered element (based onThevenod Fosse's statistical testing criterion). Much random testing assumesindependently generated variables. Our approach does not.I also describe brieﬂy the concept of program stretching, where we alterthe branch predicates to make them easier to satisfy, ﬁnd satisfying test dataand then collapse the distorted program back to the original, attempting todrag the test data with us as we go. Distribution work is with Simon Poulding.Stretching work is with Kamran Ghani and Mark Harman.Keywords: Testability Transformation, Program Stretching, Evolutionary Test-ingSearch-based Approaches to Interaction TestingMyra Cohen (University of Nebraska, US)In software systems, speciﬁc combinations of input parameters, conﬁgurationoptions or event sequences often trigger unexpected outcomes or faults. Testingto uncover these types of faults is called interaction testing. A common samplingtechnique for generating samples of inputs or conﬁgurations for interaction test-ing is to ﬁnd a set of input or conﬁguration combinations that tests all pairs ort-way combinations of the inputs or conﬁgurations.Finding a minimal set is an optimization problem, which has been solvedover the years through the use of greedy algorithms, hill climbing, tabu search,genetic algorithms, linear programming and simulated annealing. In this talk Iwill provide an overview of some of the search-based approaches that have beenused for generating interaction test samples, and will discuss some limitationsof the current approaches, opening some new and interesting research directionsfor the search-based testing community.SAT-based Automatic Test Pattern GenerationRolf Drechsler (Universität Bremen, DE)The postproduction test of integrated circuits is crucial to ensure a high qualityof the ﬁnal product. This test is carried out by checking the correct response ofthe chip under predeﬁned input stimuli - or test patterns.These patterns are calculated by algorithms for Automatic Test Pattern Gen-eration (ATPG). The basic concepts and algorithms for ATPG are brieﬂy re-viewed. Then, an advanced SAT-based ATPG tool is introduced and empricallyevaluated.Keywords: Test, ATPG, SAT4 Holger Schlingloﬀ, Tanja Vos and Joachim WegenerMutation TestingYue Jia (King's College - London, GB)Traditional mutation testing considers only ﬁrst order mutants (FOM), createdby the injection of a single fault. Often these FOMs denote trivial faults thatare easily killed. This talk presents the idea of higher order mutation testing. Inhigher order mutation testing, we apply both FOMs and subsuming higher ordermutants (HOMs) together. A subsuming HOM is one that harder to kill thanthe ﬁrst order mutants from which it is constructed. By deﬁnition, subsumingHOMs denote subtle fault combinations. This talk also proposed a new approachthat applies co-evolutionary idea to generate test case that can kill these subtlefault.Keywords: Mutation testing, higher order mutant, search-based software engi-neeringRuntime of Search Heuristics on Software TestingProblemsPer Kristian Lehre (University of Birmingham, GB)Evolutionary test generation methods are usually evaluated experimentally. Todeepen the understanding of the working principles behind these methods, onecould also consider a mathematical analysis.This talk we will describe recent theoretical results on analysing the expectedruntime and success probabilities of search heuristics on the problems of gener-ating unique input output sequences for ﬁnite state machines, and test data forbranch coverage testing of software.Keywords: Finite state machines, evolutionary algorithms, runtime analysis,unique input output sequencesUsing evolutionary algorithms to select parameters fromequivalence classesFelix Lindlar (TU Berlin, DE)This paper presents some ideas about an approach which aims at extendingexisting methodologies for functional testing. Experience in automotive appli-cations has shown that when selecting parameters for functional testing, manytimes a tester has equivalence classes in mind. Instead of losing valuable infor-mation in the process, support should be given to make them manageable. Theproposed approach suggests evolutionary testing strategies to search for criticalrepresentatives within equivalence classes.Evolutionary Test Generation 5Keywords: Equivalence classes, evolutionary testing, functional testing, auto-motive industryJoint work of: Lindlar, Felix; Marrero Peréz, AbelFull Paper: http://drops.dagstuhl.de/opus/volltexte/2009/2012Co-testability Transformation OraclePhilip McMinn (Sheﬃeld University, GB)This talk introduces the use of a testability transformation as a partial oracle insearch-based testing.The ﬁtness function aims to guide the search to the discovery of diﬀerencesin behaviour between the original program and a transformed counterpart.The transformed program works alongside the original program rather thanreplacing it, and is thus called a 'co-testability transformation oracle'.Co-testability transformation oracles are presented for ﬁnding bugs in codewith ﬂoating point calculations.Keywords: Search-based testing, testability transformation, partial oracleCo-testability TransformationPhilip McMinn (Sheﬃeld University, GB)This paper introduces the notion of co-testability transformation. As opposedto traditional testability trans- formations, which replace the original programin testing, co-testability transformations are designed to be used in conjunctionwith the original program (and any additional co-transformations as well). Un-til now, testability transfor- mations have only been used to improve test datagenera- tion. However, co-testability transformations can function as partial or-acles. This paper demonstrates practical usage of a co-testability transformationfor automatically detect- ing ﬂoating-point errors in program code.Keywords: Search-based testing, testability transformationFull Paper: http://drops.dagstuhl.de/opus/volltexte/2009/2013Full Paper:FastAbstract-11549.pdfSee also: Phil McMinn. Co-Testability Transformation. Testing: Academic &Industrial Conference - Practice And Research Techniques (TAIC PART 2008)Fast Abstract, 2008.6 Holger Schlingloﬀ, Tanja Vos and Joachim WegenerIssues in Model-Based TestingIna Schieferdecker (TU Berlin, DE)Model-Based Testing (MBT) is a variant of testing that relies on explicit be-haviour models that encode the intended behaviour of a system and/or thebehaviour of its environment. MBT aims at automating test case generation. Itis often combined with automated test execution.This presentation gives an overview on concepts and principles of MBT andargues about its prospects and costs. MBT is then contrasted with recent ap-proaches for test speciﬁcation and automated test execution, which reveals gapsin the power of current MBT methods.Open issues in MBT research are identiﬁed and selected ones being discussed:namely, ﬁnding model-based measures for test coverage and test quality, MBTfor non-functional requirements, MBT for timed or continuous systems, andthe generation of practical tests. For these issues, ongoing work together withrelations to evolutionary test (ET) generation are presented.This reveals ideas of e.g. guiding the ET search by the test oracle, to enhancethe ET search with test pattern knowledge, to ﬁnd additional test data that wellspread over the SUT or to optimize traﬃc sets for performance tests with ET.Keywords: Model-Based Testing, Test Automation, Evolutionary Test Gener-ationStructural Software Testing with Active Learning in aGraphMichèle Sebag (LRI CNRS, FR)Motivated by Structural Statistical Software Testing (SSST), this paper is in-terested in sampling the feasible execution paths in the control ﬂow graph ofthe program being tested. For some complex programs, the fraction of feasiblepaths becomes tiny, ranging in [10−10, 10−5]. When relying on the uniform sam-pling of the program paths, SSST is thus hindered by the non-Markovian natureof the feasible path concept, due to the long-range dependencies between theprogram nodes.A divide and generate approach relying on an extended Parikh Map rep-resentation is proposed to address this limitation; experimental validation onreal-world and artiﬁcial problems demonstrates gains of orders of magnitudecompared to the state of the art.Keywords: Structural Statistical Software Testing, Active Learning, ControlFlow Graph, Feaisble Paths, Parikh mapsJoint work of: Baskiotis, Nicolas; Sebag, Michèle; Gaudel, Marie-ClaudeFull Paper: http://drops.dagstuhl.de/opus/volltexte/2009/2014Evolutionary Test Generation 7Fuzzy Logic Based Objective Function Construction forEvolutionary Test GenerationAndrea Tettamanzi (University of Milano, IT)The test case generation problem can be stated as an optimization problemwhereby the closeness of test cases to violating the postcondition of a formalspeciﬁcation is maximized, subject to satisfying its precondition.This is usually implemented by constructing an objective function whichprovides a real-valued estimate of how distant all of the constraints are frombeing violated, and then trying to minimize it.A problem with this approach is that such objective functions may containplateaux, which make their minimization hard. We propose a similar approach,grounded on fuzzy logic, which uses, instead of a "distance from violation" objec-tive function, a fuzzy degree of proximity to postcondition violation and producesplateaux-free objective functions by construction.The approach is illustrated with the help of a case study on the functional(black-box) testing of computer programs.Keywords: Functional testing, fuzzy logic, objective functionFull Paper: http://drops.dagstuhl.de/opus/volltexte/2009/2016Pex - White-Box Testing for .NETNikolai Tillmann (Microsoft Research - Redmond, US)Pex is a white-box test generation tool for .NET. Starting from a hand-writtenparameterized unit test, Pex analyzes the program-under-test to determine rel-evant test inputs fully automatically. To this end, Pex executes the programmultiple times with diﬀerent inputs while monitoring the taken execution paths.For each path, the precise path condition is constructed by monitoring. Pexuses several search strategies to select the next path to explore, with the goalof achieving high statement coverage fast. Pex uses an SMT solver to determineif the next chosen path is feasible, and if so, to determine test inputs that willexercise the path. Pex is integrated into Microsoft Visual Studio.Keywords: Whitebox testing dynamic symbolic execution concolic smt solver.net C# instrumentation monitoringFull Paper:http://research.microsoft.com/Pex8 Holger Schlingloﬀ, Tanja Vos and Joachim WegenerInstrumenting where it hurts: an automatic concurrentdebugging techniqueShmuel Ur (IBM - Haifa, IL)As concurrent and distributive applications are becoming more common anddebugging such applications is very diﬃcult, practical tools for automatic de-bugging of concurrent applications are in demand. In previous work, we appliedautomatic debugging to noise-based testing of concurrent programs. The ideaof noise-based testing is to increase the probability of observing the bugs byadding, using instrumentation, timing "noise" to the execution of the program.The technique of ﬁnding a small subset of points that causes the bug to man-ifest can be used as an automatic debugging technique. Previously, we showedthat Delta Debugging can be used to pinpoint the bug location on some smallprograms.In the work reported in this paper, we create and evaluate two algorithmsfor automatically pinpointing program locations that are in the vicinity of thebugs on a number of industrial programs. We discovered that the Delta Debug-ging algorithms do not scale due to the non-monotonic nature of the concurrentdebugging problem. Instead we decided to try a machine learning feature selec-tion algorithm. The idea is to consider each instrumentation point as a feature,execute the program many times with diﬀerent instrumentations, and correlatethe features (instrumentation points) with the executions in which the bug wasrevealed. This idea works very well when the bug is very hard to reveal usinginstrumentation, correlating to the case when a very speciﬁc timing window isneeded to reveal the bug. However, in the more common case, when the bugs areeasy to ﬁnd using instrumentation points ranked high by the feature selectionalgorithm is not high enough. We show that for these cases, the important valueis not the absolute value of the evaluation of the feature but the derivative ofthat value along the program execution path.As a number of groups expressed interest in this research, we built an openinfrastructure for automatic debugging algorithms for concurrent applications,based on noise injection based concurrent testing using instrumentation. Theinfrastructure is described in this paper.Keywords: Concurrent testing, automatic debuggingFull Paper:http://portal.acm.org/citation.cfm?id=1273469&dl=&coll=Evolutionary Test Generation 9Ideas on Signal Generation for Evolutionary Testing ofContinuous SystemsAndreas Windisch (TU Berlin, DE)Test case generation constitutes a critical activity in software testing that iscost-intensive, time-consuming and error-prone when done manually. Hence, anautomation of this process is required. One automation approach is search-basedtesting for which the task of generating test data is transformed into an optimiza-tion problem which is solved using metaheuristic search techniques. However,only little work has so far been done to apply search-based testing techniquesto systems that depend on continuous input signals rather than single discreteinput values.This paper proposes three novel approaches to generating input signals fromwithin search-based testing techniques for continuous systems.Keywords: Search-Based Testing, Optimization, MetaheuristicFull Paper: http://drops.dagstuhl.de/opus/volltexte/2009/2011Combination of EAs and other techniquesQingfu Zhang (University of Essex, GB)In this talk, I present four combinations of EAs and other techniques:1. Orthogonal GA: GA+experimental design2. Guided mutation: GA+EDA3. RM-MEDA: multiobjective EDA based on a math property4. MOEA/D: a multiobjective population algorithm framework based on ag-gregationKeywords: Estimation of distribution algorithm, multiobjective optimization,experimental design, genetic algorithm

08351 Abstracts Collection -- Evolutionary Test Generation

http://drops.dagstuhl.de/opus/volltexte/2009/2023/pdf/08351.abstracts_collection.2023.pdf

08351 Abstracts Collection -- Evolutionary Test Generation

Abstract

Similar works

Full text

Available Versions

Dagstuhl Research Online Publication Server