Predicting prime path coverage using regression analysis at method level

Test coverage criteria help the tester in analyzing the quality of the test suite, especially in an evolving system where it can be used to guide the prioritization of regression tests and the testing effort of new code. However, coverage analysis of more powerful cri teria such as path coverage is still challenging due to the lack of supporting tools. As a consequence, the tester evaluates a test suite quality employing more basic coverage criteria (e.g., node coverage and edge coverage), which are the ones that are supported by tools. In this work, we evaluate the opportunity of using machine learning algorithms to estimate the prime-path coverage of a test suite at the method level. We followed the Knowledge Discovery in Database process and a dataset built from 9 real-world projects to devise three regression models for prime-path prediction. We compare four different machine learning algorithms and conduct a fine-grained feature analysis to investigate the factors that most impact the prediction accuracy. Our experimental results show that a suitable predictive model uses as input data only five source code metrics and one basic test coverage metric. Our evaluation shows that the best model achieves an MAE of 0.016 (1,6%) on the cross-validation (internal validation) and an MAE of 0.06 (6%) on the ex ternal validation. Finally, we observed that good prediction models can be generated from common code metrics although the use of a simple test metric such as branch coverage can improve even more the prediction performance of the model.Os critérios de cobertura de teste auxiliam o testador na análise da qualidade do conjunto de testes, em especial em sistemas em evolução onde pode ser utilizado para orientar a priorização dos testes de regressão e o esforço de teste de um novo código. No entanto, a análise da cobertura de critérios mais poderosos, tais como a cobertura de caminhos, continua a ser desafiante devido à falta de ferramentas de apoio. Como consequência, o testador avalia a qualidade de um conjunto de testes utilizando critérios de cobertura mais básicos (por exemplo, cobertura de nós e cobertura de arcos), que são os que são suporta dos por ferramentas. Neste trabalho, avaliou-se a oportunidade de utilizar algoritmos de aprendizagem de máquina para estimar a cobertura de caminhos primos de um conjunto de testes em nível de método. Seguiu-se o processo de descoberta de conhecimento em base de dados e um conjunto de dados construído a partir de 9 projetos do mundo real para se criarem três modelos de regressão para a previsão do valor de cobertura do critério de caminhos primos a partir de métricas de código. Compararam-se quatro algoritmos dife rentes de aprendizagem de máquina e realizou-se uma análise detalhada de características para identificar aquelas que mais afetam o desempenho da predição. Os resultados experi mentais mostraram que modelos preditivos de boa acurácia podem ser gerados a partir de um conjunto de métricas de código pequeno e de fácil obtenção. O melhor modelo gerado utiliza como dados de entrada apenas cinco métricas de código fonte e uma métrica básica de cobertura de teste e atinge um MAE de 0,016 (1,6%) na validação cruzada (validação interna) e um MAE de 0,06 (6%) na validação externa. Por fim, observou-se que modelos preditivos adequados podem ser gerados a partir de métricas de código comuns, embora o uso da métrica de cobertura de arcos, quando disponível, possa melhorar ainda mais o desempenho de predição

Automatização de testes de mutação em Java

Projeto de mestrado em Engenharia Informática, apresentada à Universidade de Lisboa, através da Faculdade de Ciências, 2014A importância da execução de testes durante o processo de desenvolvimento de software tem vindo a crescer, uma vez que estes são cruciais para garantir a qualidade do software desenvolvido, representando o principal método para a redução de ocorrência de erros. Isto tem um carater fundamental, pois no nosso dia-a-dia o software é ubíquo, e indispensável à tecnologia que dependemos. É, pois, fundamental garantir que o software possua um nível de qualidade elevado, pois uma falha poderá causar graves consequências a nível financeiro ou até mesmo ao nível humano. Torna-se assim imprescindível que os engenheiros de software dominem as técnicas de testes de software. O presente trabalho tem como objetivo construir uma ferramenta para apoiar o ensino de testes de mutação. Esta tem sido muito explorada recentemente e tem uma eficácia comprovada na atividade de testes. Os testes de mutação são uma técnica baseada em faltas e são utilizados para testar a robustez de um conjunto de casos de teste, com base no número de falhas identificadas. Cada uma das faltas produzidas por esta técnica representa um requisito de teste, que neste modelo de testes é designado por mutante. No entanto, o número de mutantes gerados é de tal ordem extenso que não pode ser sistematicamente tratado de forma manual. Daí a importância da utilização de uma ferramenta para este tipo de testes. A concretização deste trabalho resultou no desenvolvimento de uma ferramenta integrada com o ambiente de desenvolvimento Eclipse, de grande utilização tanto a nível profissional como a nível académico. Este documento apresenta aspetos relacionados com a ferramenta desenvolvida, nomeadamente, os conceitos teóricos essenciais à sua implementação; a sua integração na plataforma Eclipse e as dependências com outros elementos desta plataforma; a descrição da sua estrutura interna: a apresentação de detalhes da implementação de processos fundamentais para a concretização das funcionalidades presentes na ferramenta; e a descrição dos elementos da interface gráfica que permitem acompanhar as várias fases que compõem o processo de testes de mutação, como por exemplo, apresentação dos mutantes gerados, a pontuação dos testes de mutação, entre outras informações.The importance of running tests during the development process of software has been growing, as these are crucial to ensuring the quality of software developed, representing the primary method for reducing errors. This has a fundamental character, because in our day- to-day software is ubiquitous and essential to the technology we depend. It is therefore essential to ensure that the software has a high level of quality because a failure may cause serious consequences in financial terms or even the human level. It thus becomes essential that software engineers master the techniques of software testing. This paper aims to build a tool to support the teaching of mutation testing. This has been explored very recently and has a proven efficacy in testing activity. Changing tests are based on technical faults and are used to test the robustness of a set of test cases based on the number of identified faults. Each of faults produced by this technique is a requirement of testing in this test model is designated mutant. However, the number of mutants generated is so extensive that the order cannot be systematically handled manually. Hence the importance of using a tool for this type of testing. The completion of this work resulted in the development of an integrated tool with the Eclipse development environment, great use both professionally and academically. This paper presents aspects related to the developed tool, in particular, the essential theoretical concepts to their implementation; its integration into the Eclipse platform and dependencies with other elements of this platform; a description of their internal structure: the submission of details of the implementation of key processes for the realization of the features present in the tool; and the description of graphical interface elements for monitoring the various stages that make up the process of mutation tests, such as presentation of mutants generated, the scores of the mutation tests, among other information

Testing and test-driven development of conceptual schemas

The traditional focus for Information Systems (IS) quality assurance relies on the evaluation of its implementation. However, the quality of an IS can be largely determined in the first stages of its development. Several studies reveal that more than half the errors that occur during systems development are requirements errors. A requirements error is defined as a mismatch between requirements specification and stakeholders¿ needs and expectations. Conceptual modeling is an essential activity in requirements engineering aimed at developing the conceptual schema of an IS. The conceptual schema is the general knowledge that an IS needs to know in order to perform its functions. A conceptual schema specification has semantic quality when it is valid and complete. Validity means that the schema is correct (the knowledge it defines is true for the domain) and relevant (the knowledge it defines is necessary for the system). Completeness means that the conceptual schema includes all relevant knowledge. The validation of a conceptual schema pursues the detection of requirements errors in order to improve its semantic quality. Conceptual schema validation is still a critical challenge in requirements engineering. In this work we contribute to this challenge, taking into account that, since conceptual schemas of IS can be specified in executable artifacts, they can be tested. In this context, the main contributions of this Thesis are (1) an approach to test conceptual schemas of information systems, and (2) a novel method for the incremental development of conceptual schemas supported by continuous test-driven validation. As far as we know, this is the first work that proposes and implements an environment for automated testing of UML/OCL conceptual schemas, and the first work that explores the use of test-driven approaches in conceptual modeling. The testing of conceptual schemas may be an important and practical means for their validation. It allows checking correctness and completeness according to stakeholders¿ needs and expectations. Moreover, in conjunction with the automatic check of basic test adequacy criteria, we can also analyze the relevance of the elements defined in the schema. The testing environment we propose requires a specialized language for writing tests of conceptual schemas. We defined the Conceptual Schema Testing Language (CSTL), which may be used to specify automated tests of executable schemas specified in UML/OCL. We also describe a prototype implementation of a test processor that makes feasible the approach in practice. The conceptual schema testing approach supports test-last validation of conceptual schemas, but it also makes sense to test incomplete conceptual schemas while they are developed. This fact lays the groundwork of Test-Driven Conceptual Modeling (TDCM), which is our second main contribution. TDCM is a novel conceptual modeling method based on the main principles of Test-Driven Development (TDD), an extreme programming method in which a software system is developed in short iterations driven by tests. We have applied the method in several case studies, in the context of Design Research, which is the general research framework we adopted. Finally, we also describe an integration approach of TDCM into a broad set of software development methodologies, including the Unified Process development methodology, MDD-based approaches, storytest-driven agile methods and goal and scenario-oriented requirements engineering methods.Els enfocaments per assegurar la qualitat deis sistemes d'informació s'han basal tradicional m en! en l'avaluació de la seva implementació. No obstan! aix6, la qualitat d'un sis tema d'informació pot ser ampliament determinada en les primeres fases del seu desenvolupament. Diversos estudis indiquen que més de la meitat deis errors de software són errors de requisits . Un error de requisit es defineix com una desalineació entre l'especificació deis requisits i les necessitats i expectatives de les parts im plicades (stakeholders ). La modelització conceptual és una activitat essencial en l'enginyeria de requisits , l'objectiu de la qual és desenvolupar !'esquema conceptual d'un sistema d'informació. L'esquema conceptual és el coneixement general que un sistema d'informació requereix per tal de desenvolupar les seves funcions . Un esquema conceptual té qualitat semantica quan és va lid i complet. La valides a implica que !'esquema sigui correcte (el coneixement definit és cert peral domini) i rellevant (el coneixement definit és necessari peral sistema). La completes a significa que !'esquema conceptual inclou tot el coneixement rellevant. La validació de !'esquema conceptual té coma objectiu la detecció d'errors de requisits per tal de millorar la qualitat semantica. La validació d'esquemes conceptuals és un repte crític en l'enginyeria de requisits . Aquesta te si contribueix a aquest repte i es basa en el fet que els es quemes conceptuals de sistemes d'informació poden ser especificats en artefactes executables i, per tant, poden ser provats. Les principals contribucions de la te si són (1) un enfocament pera les pro ves d'esquemes conceptuals de sistemes d'informació, i (2) una metodología innovadora pel desenvolupament incremental d'esquemes conceptuals assistit per una validació continuada basada en proves . Les pro ves d'esquemes conceptuals poden ser una im portant i practica técnica pera la se va validació, jaque permeten provar la correctesa i la completesa d'acord ambles necessitats i expectatives de les parts interessades. En conjunció amb la comprovació d'un conjunt basic de criteris d'adequació de les proves, també podem analitzar la rellevancia deis elements definits a !'esquema. L'entorn de test proposat inclou un llenguatge especialitzat per escriure proves automatitzades d'esquemes conceptuals, anomenat Conceptual Schema Testing Language (CSTL). També hem descrit i implementa! a un prototip de processador de tes tos que fa possible l'aplicació de l'enfocament proposat a la practica. D'acord amb l'estat de l'art en validació d'esquemes conceptuals , aquest és el primer treball que proposa i implementa un entorn pel testing automatitzat d'esquemes conceptuals definits en UML!OCL. L'enfocament de proves d'esquemes conceptuals permet dura terme la validació d'esquemes existents , pero també té sentit provar es quemes conceptuals incomplets m entre estant sent desenvolupats. Aquest fet és la base de la metodología Test-Driven Conceptual Modeling (TDCM), que és la segona contribució principal. El TDCM és una metodología de modelització conceptual basada en principis basics del Test-Driven Development (TDD), un métode de programació en el qual un sistema software és desenvolupat en petites iteracions guiades per proves. També hem aplicat el métode en diversos casos d'estudi en el context de la metodología de recerca Design Science Research. Finalment, hem proposat enfocaments d'integració del TDCM en diverses metodologies de desenvolupament de software

Code Coverage Aid

Το θέμα της παρούσας πτυχιακής εργασίας είναι η ανάπτυξη ενός εργαλείου για την μέτρηση της κάλυψης κώδικα (code coverage), ένα από τα σημαντικότερα ζητήματα που αφορούν τη διαδικασία ελέγχου-δοκιμής του λογισμικού. Το εργαλείο αυτό υπολογίζει το ποσοστό του εκτελεσμένου κώδικα ενός προγράμματος έπειτα από μια σειρά δοκιμών. Περιγράφουμε και συγκρίνουμε τις πιο διαδεδομένες τεχνικές για την πραγματοποίηση της μέτρησης του code coverage. Έπειτα από μία σύντομη ανασκόπηση των διαθέσιμων εργαλείων για αυτή τη μέτρηση, παρουσιάζουμε την σχεδίαση του εργαλείου Code Coverage Aid (CCA) που δημιουργήθηκε από τον συγγραφέα και λειτουργεί σε προγράμματα γραμμένα σε C++. Το CCA χρησιμοποιεί την πιο ισχυρή τεχνική για την μέτρηση της κάλυψης κώδικα που ονομάζεται Source Instrumentation, η οποία ενεργεί πάνω στον πηγαίο κώδικα του προγράμματος πριν την μεταγλώττιση τοποθετώντας μία κλήση συνάρτησης σε κάθε διακλάδωση της ροής εκτέλεσης. Η τεχνική αυτή παρέχει έτσι τη δυνατότητα να παρακολουθεί ο προγραμματιστής ποιες γραμμές κώδικα εκτελέστηκαν σε κάθε εκτέλεση του προγράμματος. Το CCA παρέχει δυο σύνολα αποτελεσμάτων: τα στατιστικά της τελευταίας εκτέλεσης και τα συνολικά στατιστικά όλων των εκτελέσεων. Ο χρήστης μπορεί να επεξεργαστεί τα στατιστικά μέσω ενός γραφικού περιβάλοντος διεπαφής χρήστη και να αναγνωρίσει εύκολα τα κομμάτια του κώδικα που δεν έχουν εκτελεστεί. Τέλος, συγκρίνουμε το CCA με το εργαλείο GCOV που είναι ελεύθερα διαθέσιμο και παρουσιάζουμε τα αποτελέσματα της εν λόγω σύγκρισης.In this thesis we address the issue of code coverage, one of the most important issues of software testing, which describes the degree to which the source code of a program is tested. We compile a review on the most common techniques to perform code coverage analysis and we compare their pros and cons. Following a survey of several tools available for measuring the code coverage, and their characteristics, we present the design of the Code Coverage Aid tool (CCA) for C++ programs, developed by the author, to assist software developers and testers in the measuring of the amount of code coverage. Code Coverage Aid uses the most powerful technique, source Instrumentation, to instrument the source code upon compilation by placing a function call on every branch decision and allows to keep track of the lines of code that were executed during each run. CCA provides two sets of results; the statistics of the latest run, as well as cumulative statistics of all tests performed on the program so far. The user can review the code coverage results through a graphical interface and easily spot the areas (functions or code blocks) that were not executed. Based on this information, the user can decide to design additional tests to cover the code blocks indicated by CCA as ill-tested. Written in portable C++, and licensed under GNU EULA, CCA may be easily become a routine task of any C++ development team, without additional cost or development effort

Qualitätssicherung von Modelltransformationen - Über das dynamische Testen programmierter Graphersetzungssysteme

Modelle und Metamodelle repräsentieren Kernkonzepte der modellgetriebenen Softwareentwicklung (MDSD). Programme, die Modelle (unter Bezugnahme auf ihre Metamodelle) manipulieren oder ineinander überführen, werden als Modelltransformationen (MTs) bezeichnet und bilden ein weiteres Kernkonzept. Für dieses klar umrissene Aufgabenfeld wurden und werden speziell angepasste, domänenspezifische Transformationssprachen entwickelt und eingesetzt. Aufgrund der Bedeutung von MTs für das MDSD-Paradigma ist deren Korrektheit essentiell und eine gründliche Qualitätssicherung somit angeraten. Entsprechende Ansätze sind allerdings rar. In der Praxis erweisen sich die vornehmlich erforschten formalen Verifikationsansätze häufig als ungeeignet, da sie oft zu komplex oder zu teuer sind. Des Weiteren skalieren sie schlecht in Abhängigkeit zur Größe der betrachteten MT oder sind auf Abstraktionen bezogen auf die Details konkreter Implementierungen angewiesen. Demgegenüber haben testende Verfahren diese Nachteile nicht. Allerdings lassen sich etablierte Testverfahren für traditionelle Programmiersprachen aufgrund der Andersartigkeit der MT-Sprachen nicht oder nur sehr eingeschränkt wiederverwenden. Zudem sind angepasste Testverfahren grundsätzlich wünschenswert, da sie typische Eigenschaften von MTs berücksichtigen können. Zurzeit existieren hierzu überwiegend funktionsbasierte (Black-Box-)Verfahren. Das Ziel dieser Arbeit besteht in der Entwicklung eines strukturbasierten (White-Box-)Testansatzes für eine spezielle Klasse von Modelltransformationen, den sog. programmierten Graphtransformationen. Dafür ist anhand einer konkreten Vertreterin dieser Sprachen ein strukturelles Überdeckungskonzept zu entwickeln, um so den Testaufwand begrenzen oder die Güte der Tests bewerten zu können. Auch müssen Aspekte der Anwendbarkeit sowie der Leistungsfähigkeit der resultierenden Kriterien untersucht werden. Hierzu wird ein auf Graphmustern aufbauendes Testüberdeckungskriterium in der Theorie entwickelt und im Kontext des eMoflon-Werkzeugs für die dort genutzte Story- Driven-Modeling-Sprache (SDM) praktisch umgesetzt. Als Basis für eine Wiederverwendung des etablierten Ansatzes der Mutationsanalyse zur Leistungsabschätzung des Kriteriums hinsichtlich der Fähigkeiten zur Fehlererkennung werden Mutationen zur synthetischen Einbringung von Fehlern identifiziert und in Form eines Mutationstestrahmenwerks realisiert. Letzteres ermöglicht es, Zusammenhänge zwischen dem Überdeckungskonzept und der Mutationsadäquatheit zu untersuchen. Im Rahmen einer umfangreichen Evaluation wird anhand zweier nichttrivialer Modelltransformationen die Anwendbarkeit und die Leistungsfähigkeit des Ansatzes in der Praxis untersucht und eine Abgrenzung gegenüber einer quellcodebasierten Testüberdeckung durchgeführt. Es zeigt sich, dass das entwickelte Überdeckungskonzept praktisch umsetzbar ist und zu einer brauchbaren Überdeckungsmetrik führt. Die Visualisierbarkeit einzelner Überdeckungsanforderungen ist der grafischen Programmierung bei Graphtransformationen besonders nahe, so dass u. a. die Konstruktion sinnvoller Tests erleichtert wird. Die Mutationsanalyse stützt die These, dass die im Hinblick auf Steigerungen der Überdeckungsmaße optimierten Testmengen mehr Fehler erkennen als vor der Optimierung. Vergleiche mit quellcodebasierten Überdeckungskriterien weisen auf die Existenz entsprechender Korrelationen hin. Die Experimente belegen, dass die vorgestellte Überdeckung klassischen, codebasierten Kriterien vielfach überlegen ist und sich so insbesondere auch für das Testen von durch einen Interpreter ausgeführte Transformationen anbietet

Higher Order Mutation Testing

Mutation testing is a fault-based software testing technique that has been studied widely for over three decades. To date, work in this field has focused largely on first order mutants because it is believed that higher order mutation testing is too computationally expensive to be practical. This thesis argues that some higher order mutants are potentially better able to simulate real world faults and to reveal insights into programming bugs than the restricted class of first order mutants. This thesis proposes a higher order mutation testing paradigm which combines valuable higher order mutants and non-trivial first order mutants together for mutation testing. To overcome the exponential increase in the number of higher order mutants a search process that seeks fit mutants (both first and higher order) from the space of all possible mutants is proposed. A fault-based higher order mutant classification scheme is introduced. Based on different types of fault interactions, this approach classifies higher order mutants into four categories: expected, worsening, fault masking and fault shifting. A search-based approach is then proposed for locating subsuming and strongly subsuming higher order mutants. These mutants are a subset of fault mask and fault shift classes of higher order mutants that are more difficult to kill than their constituent first order mutants. Finally, a hybrid test data generation approach is introduced, which combines the dynamic symbolic execution and search based software testing approaches to generate strongly adequate test data to kill first and higher order mutants