Pengembangan sistem otomatisasi pembangkitan kasus uji dengan algoritma genetika dan test case generation method

Pengujian perangkat lunak merupakan salah satu bagian penting dari pembuatan perangkat lunak. Pada pengujian perangkat lunak terdapat pengujian unit. Pengujian unit merupakan proses pengujian komponen yang berfokus untuk memverifikasi unit terkecil pada perancangan perangkat lunak. Pada tahap pengujian unit terdapat proses pembangkitan kasus uji. Selama ini, pembangkitan kasus uji dari suatu kode program dilakukan secara manual se-hingga membutuhkan waktu yang lama. Hal ini dikarenakan banyaknya kemungkinan jalur pada kode sumber yang akan diuji.  Dalam penelitian ini dibangun suatu sistem otomatis untuk membangkitkan kasus uji. Alur kerja sistem dimulai dari analisa kode sumber dengan Spoon Library, selanjutnya dibentuk CFG (Control Flow Graph) dan DDG (Dynamic Directed Graph). Dari DDG tersebut akan dibangkitkan jalur layak yang terdapat pada DDG, dengan menggunakan algoritma genetika diharapkan dapat mengoptimalkan penentuan jalur independen. Dari masing-masing jalur independen akan dibangkitkan kasus ujinya dengan metode test case generation. Pengujian akurasi sistem pada sistem otomatisasi pembangkit kasus uji dengan jumlah populasi 5, 10 dan 15 serta jumlah maksimum generasi 50, 100, 200 dan 250 dihasilkan jumlah populasi paling optimal yaitu 10 dan maksimum generasi optimal yaitu 200 dengan akurasi 93,33%. Pada jumlah populasi dan maksimum generasi sesudahnya tidak terjadi peningkatan akurasi yang signifikan. Tiap peningkatan jumlah populasi dan maksimum generasi dapat meningkatkan akurasi sistem.  Software testing is one of the most important part of making software. On the software testing there are unit testing. Unit Testing is a process for verifying component, focusing on the smallest unit of software design. In the unit testing phase contained test case generation process. During this time, the generation of test cases of a program code is done manually. In this study, constructed an automated system to generate test cases. The workflow system starts from the analysis of the source code with the library spoon and then create CFG (Control Flow Graph) and DDG (Dynamic Directed graph). From the DDG will be raised feasible path using a genetic algorithm. Furthermore, from fea-sible path sought independenth path which is a path base d on the level of uniqueness of the path to the other path. From each independenth path raised the test case with a test case generation method. Testing accuracy of the system on the automation system generating test cases with populations of 5,10 and 15 as well as the maximum number of generations 50, 100, 200 and 250 produced the most optimal population number is 15 and the most optimal maximum generation is 200 with accuracy 93.33%. Each increase in the number of population and maximum generation can improve the accuracy of the system. Level accuracy with population number over 10 and maximum generation over 200 has no increace accuracy significant

The Effectiveness of <i>t</i>-Way Test Data Generation

Modern society is increasingly dependent on the correct functioning of software and increasingly so in areas that are considered safety related or safety critical. Therefore, there is an increasing need to be able to verify and validate that the software is in fact correct and will perform its intended function. Many approaches to this problem have been proposed; however, none seems likely to supplant the role of testing in the near future. If we accept that there is, and will be, a continuing need to be able to test software then the question becomes one of how can this be done effectively, both in terms of ability to detect errors and in terms of cost. One avenue of research that offers prospects of improving both of these aspects is the automatic generation of test data. There has recently been a large amount of work conducted in this area. One particularly promising direction has been the application of ideas from the field of experimental design and in particular, the field of t-way adequate factorial designs. The area however, is not without issues; there is evidence that the technique is capable of detecting errors but that evidence is not unequivocal. Moreover, as with almost all work in the area of automatic test generation, there has been very little comparative work comparing the technique with other test data generation techniques. Worse, there has been effectively no work done that compares any automatic test data generation technique with the effectiveness of tests generated by humans. Another major issue with the technique is the number of tests that applying the technique can result in. This implies that there is a need for an automated oracle if the technique is to be successfully applied. The flaw with this is of course that in most situations the oracle is the human that is conducting the tests, a point often ignored in testing research. The work presented here addresses both of these points. To do this I have used a code base taken from an industrial engine control system that has an existing set of high quality unit tests developed by hand. To complement this, several other techniques for automatically generating test data have been applied, namely random testing, random experimental designs and a technique for generating single factor experiments. To address the issue of being able to compare the error detection ability of all of the sets of test vectors, rather than the usual effectiveness surrogates of code coverage I have used mutation analysis on the code base to directly measure the ability of each set of test vectors to discover common coding errors. The results presented here show that test data generation techniques based on t-way factorial designs are at least as effective as handgenerated tests and superior to random testing and the factor experimental technique. The oracle problem associated with the factorial design techniques was addressed using a test set minimisation approach. The mutation tool monitored which vectors could “kill” which code mutants. After a subset of the test vectors had been run, the most effective vectors were retained and the rest discarded. Likewise, mutants that were killed were removed from further consideration and the process repeated. Experimental results show that this minimisation procedure is effective at reducing computational overhead and is capable of producing final sets of test vectors that are comparable in size with the sets of hand-generated tests and so amenable to final hand checking

SmartUnit: Empirical Evaluations for Automated Unit Testing of Embedded Software in Industry

In this paper, we aim at the automated unit coverage-based testing for embedded software. To achieve the goal, by analyzing the industrial requirements and our previous work on automated unit testing tool CAUT, we rebuild a new tool, SmartUnit, to solve the engineering requirements that take place in our partner companies. SmartUnit is a dynamic symbolic execution implementation, which supports statement, branch, boundary value and MC/DC coverage. SmartUnit has been used to test more than one million lines of code in real projects. For confidentiality motives, we select three in-house real projects for the empirical evaluations. We also carry out our evaluations on two open source database projects, SQLite and PostgreSQL, to test the scalability of our tool since the scale of the embedded software project is mostly not large, 5K-50K lines of code on average. From our experimental results, in general, more than 90% of functions in commercial embedded software achieve 100% statement, branch, MC/DC coverage, more than 80% of functions in SQLite achieve 100% MC/DC coverage, and more than 60% of functions in PostgreSQL achieve 100% MC/DC coverage. Moreover, SmartUnit is able to find the runtime exceptions at the unit testing level. We also have reported exceptions like array index out of bounds and divided-by-zero in SQLite. Furthermore, we analyze the reasons of low coverage in automated unit testing in our setting and give a survey on the situation of manual unit testing with respect to automated unit testing in industry.Comment: In Proceedings of 40th International Conference on Software Engineering: Software Engineering in Practice Track, Gothenburg, Sweden, May 27-June 3, 2018 (ICSE-SEIP '18), 10 page

JWalk: a tool for lazy, systematic testing of java classes by design introspection and user interaction

Popular software testing tools, such as JUnit, allow frequent retesting of modified code; yet the manually created test scripts are often seriously incomplete. A unit-testing tool called JWalk has therefore been developed to address the need for systematic unit testing within the context of agile methods. The tool operates directly on the compiled code for Java classes and uses a new lazy method for inducing the changing design of a class on the fly. This is achieved partly through introspection, using Java’s reflection capability, and partly through interaction with the user, constructing and saving test oracles on the fly. Predictive rules reduce the number of oracle values that must be confirmed by the tester. Without human intervention, JWalk performs bounded exhaustive exploration of the class’s method protocols and may be directed to explore the space of algebraic constructions, or the intended design state-space of the tested class. With some human interaction, JWalk performs up to the equivalent of fully automated state-based testing, from a specification that was acquired incrementally

Maintenance of Automated Test Suites in Industry: An Empirical study on Visual GUI Testing

Context: Verification and validation (V&V) activities make up 20 to 50 percent of the total development costs of a software system in practice. Test automation is proposed to lower these V&V costs but available research only provides limited empirical data from industrial practice about the maintenance costs of automated tests and what factors affect these costs. In particular, these costs and factors are unknown for automated GUI-based testing. Objective: This paper addresses this lack of knowledge through analysis of the costs and factors associated with the maintenance of automated GUI-based tests in industrial practice. Method: An empirical study at two companies, Siemens and Saab, is reported where interviews about, and empirical work with, Visual GUI Testing is performed to acquire data about the technique's maintenance costs and feasibility. Results: 13 factors are observed that affect maintenance, e.g. tester knowledge/experience and test case complexity. Further, statistical analysis shows that developing new test scripts is costlier than maintenance but also that frequent maintenance is less costly than infrequent, big bang maintenance. In addition a cost model, based on previous work, is presented that estimates the time to positive return on investment (ROI) of test automation compared to manual testing. Conclusions: It is concluded that test automation can lower overall software development costs of a project whilst also having positive effects on software quality. However, maintenance costs can still be considerable and the less time a company currently spends on manual testing, the more time is required before positive, economic, ROI is reached after automation