29 research outputs found
Automated Unit Testing of Evolving Software
As software programs evolve, developers need to ensure that new changes do
not affect the originally intended functionality of the program. To increase their
confidence, developers commonly write unit tests along with the program, and
execute them after a change is made. However, manually writing these unit-tests
is difficult and time-consuming, and as their number increases, so does the cost
of executing and maintaining them.
Automated test generation techniques have been proposed in the literature
to assist developers in the endeavour of writing these tests. However, it remains
an open question how well these tools can help with fault finding in practice,
and maintaining these automatically generated tests may require extra effort
compared to human written ones.
This thesis evaluates the effectiveness of a number of existing automatic
unit test generation techniques at detecting real faults, and explores how these
techniques can be improved. In particular, we present a novel multi-objective
search-based approach for generating tests that reveal changes across two versions
of a program. We then investigate whether these tests can be used such that no
maintenance effort is necessary.
Our results show that overall, state-of-the-art test generation tools can indeed
be effective at detecting real faults: collectively, the tools revealed more than half
of the bugs we studied. We also show that our proposed alternative technique
that is better suited to the problem of revealing changes, can detect more faults,
and does so more frequently. However, we also find that for a majority of
object-oriented programs, even a random search can achieve good results. Finally, we
show that such change-revealing tests can be generated on demand in practice,
without requiring them to be maintained over time
Random or Evolutionary Search for Object-Oriented Test Suite Generation?
An important aim in software testing is constructing a test suite with high structural code coverage
â that is, ensuring that most if not all of the code under test has been executed by the test cases
comprising the test suite. Several search-based techniques have proved successful at automatically
generating tests that achieve high coverage. However, despite the well-established arguments
behind using evolutionary search algorithms (e.g., genetic algorithms) in preference to random
search, it remains an open question whether the benefits can actually be observed in practice
when generating unit test suites for object-oriented classes. In this paper, we report an empirical
study on the effects of using evolutionary algorithms (including a genetic algorithm and chemical
reaction optimization) to generate test suites, compared with generating test suites incrementally
with random search. We apply the EVOSUITE unit test suite generator to 1,000 classes randomly
selected from the SF110 corpus of open source projects. Surprisingly, the results show that the
difference is much smaller than one might expect: While evolutionary search covers more branches
of the type where standard fitness functions provide guidance, we observed that, in practice, the
vast majority of branches do not provide any guidance to the search. These results suggest that,
although evolutionary algorithms are more effective at covering complex branches, a random
search may suffice to achieve high coverage of most object-oriented classes
Automatic generation of smell-free unit tests
Tese de mestrado, Engenharia InformĂĄtica, 2022, Universidade de Lisboa, Faculdade de CiĂȘnciasAutomated test generation tools (such as EvoSuite) typically aim to maximize code
coverage. However, they frequently disregard non-coverage aspects that can be relevant
for testers, such as the quality of the generated tests. Therefore, automatically generated
tests are often affected by a set of test-specific bad programming practices that may hinder
the quality of both test and production code, i.e., test smells. Given that other researchers
have successfully integrated non-coverage quality metrics into EvoSuite, we decided to
extend the EvoSuite tool such that the generated test code is smell-free. To this aim, we
compiled 54 test smells from several sources and selected 16 smells that are relevant to the
context of this work. We then augmented the tool with the respective test smell metrics
and investigated the diffusion of the selected smells and the distribution of the metrics.
Finally, we implemented an approach to optimize the test smell metrics as secondary
criteria. After establishing the optimal configuration to optimize as secondary criteria
(which we used throughout the remainder of the study), we conducted an empirical study
to assess whether the tests became significantly less smelly. Furthermore, we studied
how the proposed metrics affect the fault detection effectiveness, coverage, and size of
the generated tests. Our study revealed that the proposed approach reduces the overall
smelliness of the generated tests; in particular, the diffusion of the âIndirect Testingâ and
âUnrelated Assertionsâ smells improved considerably. Moreover, our approach improved
the smelliness of the tests generated by EvoSuite without compromising the code coverage
or fault detection effectiveness. The size and length of the generated tests were also not
affected by the new secondary criteria
Random or evolutionary search for object-oriented test suite generation?
An important aim in software testing is constructing a test suite with high structural code coverage, that is, ensuring that most if not all of the code under test have been executed by the test cases comprising the test suite. Several searchâbased techniques have proved successful at automatically generating tests that achieve high coverage. However, despite the wellâestablished arguments behind using evolutionary search algorithms (eg, genetic algorithms) in preference to random search, it remains an open question whether the benefits can actually be observed in practice when generating unit test suites for objectâoriented classes. In this paper, we report an empirical study on the effects of using evolutionary algorithms (including a genetic algorithm and chemical reaction optimization) to generate test suites, compared with generating test suites incrementally with random search. We apply the EVOSUITE unit test suite generator to 1000 classes randomly selected from the SF110 corpus of openâsource projects. Surprisingly, the results show that the difference is much smaller than one might expect: While evolutionary search covers more branches of the type where standard fitness functions provide guidance, we observed that, in practice, the vast majority of branches do not provide any guidance to the search. These results suggest that, although evolutionary algorithms are more effective at covering complex branches, a random search may suffice to achieve high coverage of most objectâoriented classes
Achievements, open problems and challenges for search based software testing
Search Based Software Testing (SBST) formulates testing as an optimisation problem, which can be attacked using computational search techniques from the field of Search Based Software Engineering (SBSE). We present an analysis of the SBST research agenda, focusing on the open problems and challenges of testing non-functional properties, in particular a topic we call 'Search Based Energy Testing' (SBET), Multi-objective SBST and SBST for Test Strategy Identification. We conclude with a vision of FIFIVERIFY tools, which would automatically find faults, fix them and verify the fixes. We explain why we think such FIFIVERIFY tools constitute an exciting challenge for the SBSE community that already could be within its reach
Automatic Test Data Generation Using Constraint Programming and Search Based Software Engineering Techniques
RĂSUMĂ
Prouver qu'un logiciel correspond à sa spécification ou exposer des erreurs cachées dans son implémentation est une tùche de test trÚs difficile, fastidieuse et peut coûter plus de 50% de coût total du logiciel. Durant la phase de test du logiciel, la génération des données de test est l'une des tùches les plus coûteuses. Par conséquent, l'automatisation de cette tùche permet de réduire considérablement le coût du logiciel, le temps de développement et les délais de commercialisation.
Plusieurs travaux de recherche ont proposé des approches automatisées pour générer des données de test. Certains de ces travaux ont montré que les techniques de génération des données de test qui sont basées sur des métaheuristiques (SB-STDG) peuvent générer automatiquement des données de test. Cependant, ces techniques sont trÚs sensibles à leur orientation qui peut avoir un impact sur l'ensemble du processus de génération des données de test. Une insuffisance d'informations pertinentes sur le problÚme de génération des données de test peut affaiblir l'orientation et affecter négativement l'efficacité et l'effectivité de SB-STDG.
Dans cette thÚse, notre proposition de recherche est d'analyser statiquement le code source pour identifier et extraire des informations pertinentes afin de les exploiter dans le processus de SB-STDG pourrait offrir davantage d'orientation et ainsi d'améliorer l'efficacité et l'effectivité de SB-STDG.
Pour extraire des informations pertinentes pour l'orientation de SB-STDG, nous analysons de maniÚre statique la structure interne du code source en se concentrant sur six caractéristiques, i.e., les constantes, les instructions conditionnelles, les arguments, les membres de données, les méthodes et les relations.
En mettant l'accent sur ces caractéristiques et en utilisant différentes techniques existantes d'analyse statique, i.e, la programmation par contraintes (CP), la théorie du schéma et certains analyses statiques légÚres,
nous proposons quatre approches:
(1) en mettant l'accent sur les arguments et les instructions conditionnelles, nous définissons une approche hybride qui utilise les techniques de CP pour guider SB-STDG à réduire son espace de recherche;
(2) en mettant l'accent sur les instructions conditionnelles et en utilisant des techniques de CP, nous dĂ©finissons deux nouvelles mĂ©triques qui mesurent la difficultĂ© Ă satisfaire une branche (i.e., condition), d'oË nous tirons deux nouvelles fonctions objectif pour guider SB-STDG;
(3) en mettant l'accent sur les instructions conditionnelles et en utilisant la théorie du schéma, nous adaptons l'algorithme génétique pour mieux répondre au problÚme de la génération de données de test;
(4) en mettant l'accent sur les arguments, les instructions conditionnelles, les constantes, les membres de données, les méthodes et les relations, et en utilisant des analyses statiques légÚres, nous définissons un générateur d'instance qui génÚre des données de test candidates pertinentes
et une nouvelle représentation du problÚme de génération des données de test orienté-objet qui réduit implicitement l'espace de recherche de SB-STDG.
Nous montrons que les analyses statiques aident à améliorer l'efficacité et l'effectivité de SB-STDG. Les résultats obtenus dans cette thÚse montrent des améliorations importantes en termes d'efficacité et d'effectivité. Ils sont prometteurs et nous espérons que d'autres recherches dans le domaine de la génération des données de test pourraient améliorer davantage l'efficacité ou l'effectivité.----------ABSTRACT
Proving that some software system corresponds to its specification or revealing hidden errors in its implementation is a time consuming and tedious testing process, accounting for 50% of the total software. Test-data generation is one of the most expensive parts of the software testing phase. Therefore, automating this task can significantly reduce software cost, development time, and time to market.
Many researchers have proposed automated approaches to generate test data.
Among the proposed approaches, the literature showed that Search-Based Software Test-data Generation (SB-STDG) techniques can automatically generate test data.
However, these techniques are very sensitive to their guidance which impact the whole test-data generation process. The insufficiency of information relevant about the test-data generation problem can weaken the SB-STDG guidance and negatively affect its efficiency and effectiveness.
In this dissertation, our thesis is statically analyzing source code to identify and extract relevant information to exploit them in the SB-STDG process could offer more guidance and thus improve the efficiency and effectiveness of SB-STDG.
To extract information relevant for SB-STDG guidance, we statically analyze the internal structure of the source code focusing on six features, i.e., constants, conditional statements, arguments, data members, methods, and relationships.
Focusing on these features and using different existing techniques of static analysis, i.e., constraints programming (CP), schema theory, and some lightweight static analyses,
we propose four approaches:
(1) focusing on arguments and conditional statements, we define a hybrid approach that uses CP techniques to guide SB-STDG in reducing its search space;
(2) focusing on conditional statements and using CP techniques, we define two new metrics that measure the difficulty to satisfy a branch, hence we derive two new fitness functions to guide SB-STDG;
(3) focusing on conditional statements and using schema theory, we tailor genetic algorithm to better fit the problem of test-data generation;
(4) focusing on arguments, conditional statements, constants, data members, methods, and relationships, and using lightweight static analyses, we define an instance generator that generates relevant test-data candidates
and a new representation of the problem of object-oriented test-data generation that implicitly reduces the SB-STDG search space.
We show that using static analyses improve the SB-STDG efficiency and effectiveness. The achieved results in this dissertation show an important improvements in terms of effectiveness and efficiency. They are promising and we hope that further research in the field of test-data generation might improve efficiency or effectiveness
Exploring means to facilitate software debugging
In this thesis, several aspects of software debugging from automated crash reproduction to bug report analysis and use of contracts have been studied.Algorithms and the Foundations of Software technolog
Search-based Unit Test Generation for Evolving Software
Search-based software testing has been successfully applied to generate unit test cases for object-oriented software. Typically, in search-based test generation approaches, evolutionary search algorithms are guided by code coverage criteria such as branch coverage to generate tests for individual coverage objectives. Although it has been shown that this approach can be effective, there remain fundamental open questions. In particular, which criteria should test generation use in order to produce the best test suites? Which evolutionary algorithms are more effective at generating test cases with high coverage? How to scale up search-based unit test generation to software projects consisting of large numbers of components, evolving and changing frequently over time? As a result, the applicability of search-based test generation techniques in practice is still fundamentally limited. In order to answer these fundamental questions, we investigate the following improvements to search-based testing. First, we propose the simultaneous optimisation of several coverage criteria at the same time using an evolutionary algorithm, rather than optimising for individual criteria. We then perform an empirical evaluation of different evolutionary algorithms to understand the influence of each one on the test optimisation problem. We then extend a coverage-based test generation with a non-functional criterion to increase the likelihood of detecting faults as well as helping developers to identify the locations of the faults. Finally, we propose several strategies and tools to efficiently apply search-based test generation techniques in large and evolving software projects. Our results show that, overall, the optimisation of several coverage criteria is efficient, there is indeed an evolutionary algorithm that clearly works better for test generation problem than others, the extended coverage-based test generation is effective at revealing and localising faults, and our proposed strategies, specifically designed to test entire software projects in a continuous way, improve efficiency and lead to higher code coverage. Consequently, the techniques and toolset presented in this thesis - which provides support to all contributions here described - brings search-based software testing one step closer to practical usage, by equipping software engineers with the state of the art in automated test generation
Mathematics in Software Reliability and Quality Assurance
This monograph concerns the mathematical aspects of software reliability and quality assurance and consists of 11 technical papers in this emerging area. Included are the latest research results related to formal methods and design, automatic software testing, software verification and validation, coalgebra theory, automata theory, hybrid system and software reliability modeling and assessment