Structural testing criteria are mandated in many software development standards and guidelines. The process of generating test data to achieve 100% coverage of a given structural coverage metric is labour-intensive and expensive. This paper presents an approach to automate the generation of such test data. The test-data generation is based on the application of a dynamic optimisation-based search for the required test data. The same approach can be generalised to solve other test-data generation problems. Three such applications are discussed-boundary value analysis, assertion/run-time exception testing, and component re-use testing. A prototype tool-set has been developed to facilitate the automatic generation of test data for these structural testing problems. The results of preliminary experiments using this technique and the prototype tool-set are presented and show the efficiency and effectiveness of this approac

Heslington York

John Clark

John Mcdermid

Keith Mander

Nigel Tracey

English

CiteSeerX

An automated framework for structural test-data generation

Structural testing criteria are mandated in many software development standards and guidelines. The process of generating test-data to achieve 100 % coverage of a given structural coverage metric is labour intensive and expensive. This paper presents an approach to automate the generation of such test-data. The test-data generation is based on the application of a dynamic optimisation-based search for the required test-data. The same approach can be be generalised to solve other test-data generation problems. Three such applications are discussed { boundary value analysis, assertion/run-time exception testing and component re-use testing. Aprototype tool-set has been developed to facilitate the automatic generation of test-data for these structural testing problems. The results of preliminary experiments using this technique and the prototype tool-set are presented and show the e ciency and e ectiveness of this approach. 

Tracey, Nigel J.

Clark, John A.

Mander, Keith C.

McDermid, John A.

Kent Academic Repository

An Automated Framework for Structural Test-Data GenerationNigel TraceyJohn Clark Keith Mander John McDermidDepartment of Computer Science. University of York,Heslington, York. YO1 5DD, England. Tel : +44 1904 432749fnjt, jac, mander, jamg@cs.york.ac.ukAbstractStructural testing criteria are mandated in many soft-ware development standards and guidelines. The pro-cess of generating test-data to achieve 100% cover-age of a given structural coverage metric is labourintensive and expensive. This paper presents an ap-proach to automate the generation of such test-data.The test-data generation is based on the application ofa dynamic optimisation-based search for the requiredtest-data. The same approach can be generalised tosolve other test-data generation problems. Three suchapplications are discussed { boundary value analysis,assertion/run-time exception testing and componentre-use testing. A prototype tool-set has been developedto facilitate the automatic generation of test-data forthese structural testing problems. The results of pre-liminary experiments using this technique and the pro-totype tool-set are presented and show the eciencyand eectiveness of this approach.1 IntroductionSoftware testing is an expensive and time-consumingprocess, especially in safety-critical applications, typi-cally consuming at least 50% of the total costs involvedin developing software [2]. Automation of the testingprocess would allow both reduced development costsand an increase in the quality of (or at least condencein) the software. Ould suggested that automation oftest-data generation is vital to advance the state-of-the-art is software testing [9]. The test-data genera-tion problem is that of identifying program input datawhich satisfy selected testing criteria. Test-data selec-tion methods can be divided into two distinct classes{ functional and structural testing [2].Automated test-data generators for structural test-ing can be divided into three classes { random [2],static [3, 4] and dynamic. This work is based on thedynamic approach to test-data generation. This ap-proach was rst suggested in 1976 [8]. Korel's work [7]built on this, suggesting as a way forward the use ofglobal optimisation techniques. More recently, somework has investigated the use of global-optimisation[13, 5] to overcome the problem of locally optimal so-lutions in the search space.The research presented in this paper builds on theseapproaches. An optimisation-based framework hasbeen developed and applied to a number of testingproblems, such as specication conformance testingand worst-case execution time testing [10, 11]. Thispaper focuses on the application of the framework toa number of structural-testing problems.2 Optimisation-Based Struc-tural Test-Data GenerationIn essence, the test-data generation problem for struc-tural testing is nding a set of program inputs thatachieves the desired coverage. The dynamic approachto this problem involves a search for program inputwhich forces execution of the desired part of the soft-ware under test (SUT).For the search to succeed, it needs to be given someguidance. This guidance is given in the form of a cost-function which relates a program input to a measureof how good it is. The amount of guidance given by thecost-function is one of the key elements in determin-ing the eectiveness of the test-data generation. Theother key factor is, of course, the search technique it-self.The input domain of most programs is likely to bevery large. A cost-surface could be formed by applyingthe cost-function to every possible program input. Itis this surface which is eectively being searched whenattempting to generate test-data. Given the complex-ities of software systems it is extremely unlikely thatthis cost surface would be linear or continuous. Thesize and complexity of the search space therefore limitsthe eectiveness of simple gradient-descent or neigh-bourhood searches as they are likely to get stuck inlocally optimal solutions and hence fail to nd the de-sired test-data.Heuristic-global optimisation techniques are de-signed to nd good approximations to the optimalsolution in large complex search spaces. Simulatedannealing is one such general purpose optimisationtechnique [6]. Simulated annealing allows movementswhich worsen the value of the cost-function based ona control parameter known as the temperature. Earlyin the search inferior solutions are accepted with rel-ative freedom, but as the search progresses (or cools)accepting inferior solutions becomes more and morerestricted. Accepting these inferior solutions is basedon the premise that it is better to accept a short termpenalty in the hope of longer term rewards.It is necessary to dene the cost-function whichwill guide the simulated annealing search. The cost-function needs to indicate whether the desired state-ment (or branch, etc.) has been executed. The cost-function needs to return good values for test-data thatnearly executes the desired statement and bad valuesfor test-data that is a long way from executing the de-sired statement. The branch predicates determine thepath followed and are therefore vital in determiningan eective cost-function.Branch predicates consist of relational expressionsconnected with logical operators. The cost-functionhas been designed such that it will evaluate to zero ifthe branch predicate evaluates to the desired conditionand will be positive otherwise. This property gives anecient stopping criteria for the search process. Thecost-function is calculated as shown in table 1.In order to evaluate the cost-function it is neces-sary to execute an instrumented version of the SUT.There are two types of procedure call added by the in-strumenter { execution path monitor calls and branchevaluation calls. The monitor calls work as follows: Record that node X has been executed. If X is current target node then move to nexttarget node on target path.The branch evaluation calls replace the branchpredicates in the SUT. They are responsible for addingto the overall cost the contribution made by each indi-vidual branch predicate which is executed and return-Element ValueBoolean if TRUE then 0 else Ka = b if abs(a  b) = 0 then 0else abs(a  b) +Ka 6= b if abs(a  b) 6= 0 then 0else Ka < b if a  b < 0 then 0else (a  b) +Ka  b if a  b  0 then 0else (a  b) +Ka > b if b  a < 0 then 0else (b  a) +Ka  b if b  a  0 then 0else (b  a) +Ka _ b min (cost (a), cost (b))a ^ b cost (a) + cost (b):a Negation is moved inwards andpropagated over aTable 1: Cost-Function Calculationing the boolean value of the predicate. They functionas follows: If the target node is only reachable if the branchpredicate is true then add cost of branch predicateto the overall cost for the current test-data. Ifbranch predicate must be false then the cost of :(branch predicate) is used. For loop predicates the desired number of iter-ations determines whether the cost of the looppredicate or : (loop predicate) is used. Within loops, adding the cost of branch predi-cates is deferred until exit from the loop. At thispoint the minimum cost evaluated for that branchpredicate is added to the overall cost. This pre-vents punishment of taking an undesirable branchuntil exit from a loop, as the desirable branch maybe taken on subsequent iterations.The cost-function gives a quantitative measure ofthe suitability of the generated test-data for the pur-pose of executing the specied node in (or paththrough) the SUT. The simulated annealing searchuses this to guide its generation of test-data until ei-ther it has successfully found test-data with a zerocost or until the search freezes and no further progresscan be made. At this point the system moves on toattempt generation of test-data for the next desirednode or path.3 Other Structural TestingProblemsWhile it is useful to generate automatically test-datathat meets structural objectives there are many othertesting problems. For an automatic test-data gener-ator to be generally useful it must be exible. Thissection describes how the optimisation-based test-datageneration framework which has been developed canbe extended to address other structural testing prob-lems. Previous work has illustrated how the frame-work may be extended to address other non-structuraltesting problems, [10, 11].3.1 Boundary Value AnalysisBoundary value analysis considers test-data which liesclose the boundaries of a sub-domain to be of a higheradequacy than test-data which is further away.To address the problems of boundary value analy-sis the cost-function needs to be biased towards lowercosts for test-data at or near a boundary. This canbe achieved with a simple extension to the previouscost function. Consider the example of X < 42, ifthe desired path requires this condition to be false,the test-data generator needs to generate X  42. Tobias this towards the boundary case of X = 42 theprevious cost-function is augmented with the cost ofX = 42. The value of this is reduced by a scalingfactor to allow the cost of X  42 to dominate. Thisremoves the ecient stopping criterion dened earlier.This simply means that the search will continue untilit freezes, at which point it will terminate.For more complex branch predicates, the particu-lar boundary may be more dicult to identify. Thecurrent tool-set will automatically search for test-dataset which is on the boundary of each of the individualrelational expressions in the predicate. This can beoverridden by the user to allow combined boundariesto be targeted.3.2 Assertion/Run-Time ExceptionTestingAssertions and run-time exceptions are a useful toolfor automatic detection of run-time errors. The goalof the test-data generator for assertion and run-timeexception testing is to nd test-data which breaks theassertion or causes the run-time exception. To meetthis goal the cost-function must guide the search intwo ways. First, the search must be guided to test-data that executes the statement associated with theassertion or possible exception. Secondly the searchmust be guided to test-data that causes the assertionto be false, or the exception condition to be true.To allow improved guidance the assertion orexception-condition is converted to disjunctive normalform (DNF). A solution to any one disjunct then rep-resents a solution to the entire condition. Each dis-junct is considered in turn. The search process aimsto nd test-data which executes the desired statementand satises the current disjunct. The cost-functionfor the disjuncts is the same as that used for branchpredicates.3.3 Component Re-use TestingWhen re-using components it is vital that the envi-ronment within which they are re-used meets the as-sumptions that were made when the component wasdeveloped, i.e. for all input-data in the component'sdomain the pre-condition holds true. The goal of re-use testing can be stated as nding test-data whichcauses the precondition of a re-usable component tobe broken. The component is instrumented such thatit evaluates the cost of meeting a given disjunct in thenegated pre-condition. This value is added to the over-all cost. Thus the search is guided to test-data whichexecutes the call of the component and also breaks thepre-condition of the component.4 EvaluationA preliminary evaluation has used a collection of smallprograms written in Ada 95 (20 to 200 lines of code).In all but one case 100% branch coverage was achievedwith search times of 2 to 35 seconds. In the failing case48 of the 50 attempts gave 100% coverage. Tuning thesearch parameters to slow down the cooling overcamethis problem. A number of the programs containedbreakable assertions or could cause run-time excep-tions. The simulated annealing search found test-datain call cases which was able to highlight the error. Thetest-data generation took between 0.5 and 17 seconds.Component re-use testing has been evaluated using asmall number of programs with explicit pre-conditionsspecied as Spark Ada proof-context annotations [1].Harness software was produced which used these com-ponents in an unsafe manner which for some inputs.For each of these simple examples suitable test-datawas generated to highlight the unsafe reuse. As a com-parison random test-data generaton has been used onthe same problems and was found to be up to 4 ordersof magnitude less ecient.5 ConclusionsAs with all testing approaches, we can only show thepresence of faults not their absence. Indeed, the fail-ure of the search to nd exception conditions or re-useproblems does not indicate that the software is correct,only that the search failed. However, given an inten-sive directed search for a specic problem, the failureto nd the problem does allow increased condencein the quality of the software which is after all theaim of testing. We view the tools simply as generat-ing test-data that can be checked by other means, i.e.use of a test-harness to check that the generated test-data does in fact cause the desired exception. This isimportant as the algorithms are stochastic and it isdicult to reason about their ecacy for applicationto arbitrary code. More details and full results canbe found in the full version of this paper (located athttp://www.cs.york.ac.uk/testsig in the publicationssection)5.1 Further WorkThe results presented above show that it is possible touse optimisation techniques to develop a frameworkto automate the generation of test-data for a numberof common software testing problems. This frameworkcan be used to generate test-data eciently. The mostimportant area requiring further work is that of gath-ering empirical evidence as to the eectiveness of thetechnique.There is a variety of other optimisation techniqueswhich could be examined. A detailed comparison ofthe various optimisation techniques to discover theirrelative strengths and weaknesses would be required.Tabu-search and genetic algorithms are two such op-timisation techniques which are suitable for investi-gation. Some preliminary work on methods to applytabu-search and genetic-algorithms has already beencarried out [12].6 AcknowledgementsThis work was funded by grant GR/L42872 from theEngineering and Physical Sciences Research Coun-cil (EPSRC) in the UK as part of the CONVERSEproject.References[1] John Barnes. High Integrity Ada: The SPARKApproach. Addison-Wesley, 1997.[2] B. Beizer. Software System Testing and QualityAssurance. Thomson Computer Press, 1996.[3] L. Clarke. A system to generate test data andsymbolically execute programs. IEEE Transac-tions on Software Engineering, SE-2(3):215{222,September 1976.[4] R. Demillo and A. Outt. Experimental re-sults form an automatic test case generator.ACM Transactions on Software Engineering andMethodology, 2(2):109{127, April 1993.[5] B. Jones, H. Sthamer, and D. Eyres. Automaticstructural testing using genetic algorithms. Soft-ware Engineering Journal, 11(5):299{306, 1996.[6] S. Kirkpatrick, Jr. C. D. Gelatt, and M. P. Vecchi.Optimization by simulated annealing. Science,220(4598):671{680, May 1983.[7] B. Korel. Automated software test data genera-tion. IEEE Transactions on Software Engineer-ing, 16(8):870{879, 1990.[8] W. Miller and D. Spooner. Automatic generationof oating-point test data. IEEE Transactions onSoftware Engineering, SE-2(3):223{226, Septem-ber 1976.[9] M. Ould. Testing - a challenge to method andtool developers. Software Engineering Journal,6(2):59{64, March 1991.[10] Nigel Tracey, John Clark, and Keith Mander.Automated program aw nding using simu-lated annealing. In International Symposiumon Software Testing and Analysis, pages 73{81.ACM/SIGSOFT, 1998.[11] Nigel Tracey, John Clark, and Keith Mander. Theway forward for unifying dynamic test case gen-eration: The optimisation-based approach. In In-ternational Workshop on Dependable Computingand Its Applications, pages 169{180. IFIP, 1998.[12] Nigel J. Tracey. Test-case data generation usingoptimisation techniques { rst year DPhil report.Department of Computer Science, University ofYork, 1997.[13] Alison Lachut Watkins. The automatic genera-tion of test data using genetic algorithms. Pro-ceedings of the 4th Software Quality Conference,2:300{309, 1995.

A system to generate test data and symbolically execute programs.

Automated program   using simulated annealing.

Automated software test data generation.

Automatic generation of  test data.

Automatic structural testing using genetic algorithms.

Experimental results form an automatic test case generator.

High Integrity Ada: The SPARK Approach.

Optimization by simulated annealing.

Software System Testing and Quality Assurance.

Test-case data generation using optimisation techniques {  year DPhil report.

Testing - a challenge to method and tool developers.

The automatic generation of test data using genetic algorithms.

The way forward for unifying dynamic test case generation: The optimisation-based approach.

An Automated Framework for Structural Test-data Generation

Structural testing criteria are mandated in many software development standards and guidelines. The process of generating test-data to achieve 100 % coverage of a given structural coverage metric is labour intensive and expensive. This paper presents an approach to automate the generation of such test-data. The test-data generation is based on the application of a dynamic optimisation-based search for the required test-data. The same approach can be generalised to solve other test-data generation problems. Three such applications are discussed { boundary value analysis, assertion/run-time exception testing and component re-use testing. Aprototype tool-set has been developed to facilitate the automatic generation of test-data for these structural testing problems. The results of preliminary experiments using this technique and the prototype tool-set are presented and show the e ciency and e ectiveness of this approach. 

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.67.1350

An Automated Framework for Structural Test-data Generation

Abstract

Similar works

Full text

Available Versions

CiteSeerX

CiteSeerX

Kent Academic Repository

CiteSeerX