Testes incrementais em um desenvolvimento guiado por testes baseados em modelo

Orientador: Eliane MartinsDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: O desenvolvimento de sistemas pode ser realizado seguindo diversos modelos de processo. Os métodos ágeis propõem realizar implementações iterativas e incrementais e testes antecipados, buscando uma validação antecipada do sistema. Algumas técnicas ágeis adicionam a característica de um desenvolvimento de sistema baseado em testes, como as técnicas de Desenvolvimento Baseado em Teste (do inglês Test Driven Development (TDD)) e Desenvolvimento Baseado em Comportamento (do inglês Behaviour Driven Development (BDD)). Recentemente algumas técnicas propõem a união de técnicas ágeis de desenvolvimento baseado em testes com técnicas consolidadas da área de testes, com o objetivo principal de auxiliar na etapa de criação de testes, que serão utilizados para guiar o desenvolvimento do sistema. Um exemplo é a técnica de Desenvolvimento Guiado por Testes Baseados em Modelo (do inglês Model-Based Test Driven Development (MBTDD)) que une os conceitos de Testes Baseados em Modelo (do inglês Model-Based Testing (MBT)) e Desenvolvimento Baseado em Teste (TDD). Portanto em MBTDD, testes são derivados de modelos que representam os comportamentos esperados do sistema, e baseado nesses testes, o desenvolvimento iterativo e incremental ocorre. Entretanto quando lidamos com processos iterativos e incrementais, surgem problemas decorrente da evolução do sistema, como por exemplo: como reutilizar os artefatos de testes, e como selecionar os testes relevantes para a codificação da nova versão do sistema. Nesse contexto, este trabalho explora um processo no qual o desenvolvimento ágil de sistema é guiado por testes baseados em modelos, com o enfoque no auxílio do reúso dos artefatos de testes e no processo de identificação de testes relevantes para o desenvolvimento de uma nova versão do sistema. Para tanto, características do processo de MBTDD são unidas com características de uma técnica que busca o reúso de artefatos de testes baseado em princípios de testes de regressão, denominada Testes de Regressão SPL Baseados em Modelo Delta (do inglês Delta-Oriented Model-Based SPL Regression Testing). Para realizar a avaliação da solução proposta, ela foi aplicada em exemplos existentes e comparada com a abordagem no qual nenhum caso de teste é reutilizadoAbstract: Systems can be developed following different process models. Agile methods propose iterative and incremental implementations and anticipating tests, in order to anticipate system validation. Some agile techniques add the characteristic of development based on tests, like in Test Driven Development (TDD) and Behaviour Driven Development (BDD). Recently some techniques proposed joining the agile techniques of development based on tests with techniques consolidated in the field of testing, with the main purpose of aiding in the test creation stage, which are used to guide the development of the system. An example is Model-Based Test Driven Development (MBTDD) which joins the concepts of Model-Based Testing (MBT) and Test Driven Development (TDD). Therefore in MBTDD, tests are derived from models that represent the expected behaviour of the system, and based on those tests, iterative and incremental development is performed. However, when iterative and incremental processes are used, problems appear as the consequence of the evolution of the system, such as: how to reuse the test artefacts, and how to select the relevant tests for implementing the new version of the system. In this context, this work proposes a process in which the agile development of a system is guided by model based tests, focusing on helping with the reuse of test artefacts and on the process of identifying tests relevant to development. To achieve this goal, MBTDD process characteristics are joined with characteristics from a technique that aims to find reusability of test artefacts based on principles of regression tests, called Delta-Oriented Model-Based SPL Regression Testing. To evaluate the proposed solution, it was applied to existing examples and compared to the approach without any test case reuseMestradoCiência da ComputaçãoMestra em Ciência da Computação151647/2013-5CNP

Supporting the grow-and-prune model for evolving software product lines

207 p.Software Product Lines (SPLs) aim at supporting the development of a whole family of software products through a systematic reuse of shared assets. To this end, SPL development is separated into two interrelated processes: (1) domain engineering (DE), where the scope and variability of the system is defined and reusable core-assets are developed; and (2) application engineering (AE), where products are derived by selecting core assets and resolving variability. Evolution in SPLs is considered to be more challenging than in traditional systems, as both core-assets and products need to co-evolve. The so-called grow-and-prune model has proven great flexibility to incrementally evolve an SPL by letting the products grow, and later prune the product functionalities deemed useful by refactoring and merging them back to the reusable SPL core-asset base. This Thesis aims at supporting the grow-and-prune model as for initiating and enacting the pruning. Initiating the pruning requires SPL engineers to conduct customization analysis, i.e. analyzing how products have changed the core-assets. Customization analysis aims at identifying interesting product customizations to be ported to the core-asset base. However, existing tools do not fulfill engineers needs to conduct this practice. To address this issue, this Thesis elaborates on the SPL engineers' needs when conducting customization analysis, and proposes a data-warehouse approach to help SPL engineers on the analysis. Once the interesting customizations have been identified, the pruning needs to be enacted. This means that product code needs to be ported to the core-asset realm, while products are upgraded with newer functionalities and bug-fixes available in newer core-asset releases. Herein, synchronizing both parties through sync paths is required. However, the state of-the-art tools are not tailored to SPL sync paths, and this hinders synchronizing core-assets and products. To address this issue, this Thesis proposes to leverage existing Version Control Systems (i.e. git/Github) to provide sync operations as first-class construct

Supporting the grow-and-prune model for evolving software product lines

207 p.Software Product Lines (SPLs) aim at supporting the development of a whole family of software products through a systematic reuse of shared assets. To this end, SPL development is separated into two interrelated processes: (1) domain engineering (DE), where the scope and variability of the system is defined and reusable core-assets are developed; and (2) application engineering (AE), where products are derived by selecting core assets and resolving variability. Evolution in SPLs is considered to be more challenging than in traditional systems, as both core-assets and products need to co-evolve. The so-called grow-and-prune model has proven great flexibility to incrementally evolve an SPL by letting the products grow, and later prune the product functionalities deemed useful by refactoring and merging them back to the reusable SPL core-asset base. This Thesis aims at supporting the grow-and-prune model as for initiating and enacting the pruning. Initiating the pruning requires SPL engineers to conduct customization analysis, i.e. analyzing how products have changed the core-assets. Customization analysis aims at identifying interesting product customizations to be ported to the core-asset base. However, existing tools do not fulfill engineers needs to conduct this practice. To address this issue, this Thesis elaborates on the SPL engineers' needs when conducting customization analysis, and proposes a data-warehouse approach to help SPL engineers on the analysis. Once the interesting customizations have been identified, the pruning needs to be enacted. This means that product code needs to be ported to the core-asset realm, while products are upgraded with newer functionalities and bug-fixes available in newer core-asset releases. Herein, synchronizing both parties through sync paths is required. However, the state of-the-art tools are not tailored to SPL sync paths, and this hinders synchronizing core-assets and products. To address this issue, this Thesis proposes to leverage existing Version Control Systems (i.e. git/Github) to provide sync operations as first-class construct

Modellbasiertes Regressionstesten von Varianten und Variantenversionen

The quality assurance of software product lines (SPL) achieved via testing is a crucial and challenging activity of SPL engineering. In general, the application of single-software testing techniques for SPL testing is not practical as it leads to the individual testing of a potentially vast number of variants. Testing each variant in isolation further results in redundant testing processes by means of redundant test-case executions due to the shared commonality. Existing techniques for SPL testing cope with those challenges, e.g., by identifying samples of variants to be tested. However, each variant is still tested separately without taking the explicit knowledge about the shared commonality and variability into account to reduce the overall testing effort. Furthermore, due to the increasing longevity of software systems, their development has to face software evolution. Hence, quality assurance has also to be ensured after SPL evolution by testing respective versions of variants. In this thesis, we tackle the challenges of testing redundancy as well as evolution by proposing a framework for model-based regression testing of evolving SPLs. The framework facilitates efficient incremental testing of variants and versions of variants by exploiting the commonality and reuse potential of test artifacts and test results. Our contribution is divided into three parts. First, we propose a test-modeling formalism capturing the variability and version information of evolving SPLs in an integrated fashion. The formalism builds the basis for automatic derivation of reusable test cases and for the application of change impact analysis to guide retest test selection. Second, we introduce two techniques for incremental change impact analysis to identify (1) changing execution dependencies to be retested between subsequently tested variants and versions of variants, and (2) the impact of an evolution step to the variant set in terms of modified, new and unchanged versions of variants. Third, we define a coverage-driven retest test selection based on a new retest coverage criterion that incorporates the results of the change impact analysis. The retest test selection facilitates the reduction of redundantly executed test cases during incremental testing of variants and versions of variants. The framework is prototypically implemented and evaluated by means of three evolving SPLs showing that it achieves a reduction of the overall effort for testing evolving SPLs.Testen ist ein wichtiger Bestandteil der Entwicklung von Softwareproduktlinien (SPL). Aufgrund der potentiell sehr großen Anzahl an Varianten einer SPL ist deren individueller Test im Allgemeinen nicht praktikabel und resultiert zudem in redundanten Testfallausführungen, die durch die Gemeinsamkeiten zwischen Varianten entstehen. Existierende SPL-Testansätze adressieren diese Herausforderungen z.B. durch die Reduktion der Anzahl an zu testenden Varianten. Jedoch wird weiterhin jede Variante unabhängig getestet, ohne dabei das Wissen über Gemeinsamkeiten und Variabilität auszunutzen, um den Testaufwand zu reduzieren. Des Weiteren muss sich die SPL-Entwicklung mit der Evolution von Software auseinandersetzen. Dies birgt weitere Herausforderungen für das SPL-Testen, da nicht nur für Varianten sondern auch für ihre Versionen die Qualität sichergestellt werden muss. In dieser Arbeit stellen wir ein Framework für das modellbasierte Regressionstesten von evolvierenden SPL vor, das die Herausforderungen des redundanten Testens und der Software-Evolution adressiert. Das Framework vereint Testmodellierung, Änderungsauswirkungsanalyse und automatische Testfallselektion, um einen inkrementellen Testprozess zu definieren, der Varianten und Variantenversionen unter Ausnutzung des Wissens über gemeinsame Funktionalität und dem Wiederverwendungspotential von Testartefakten und -resultaten effizient testet. Für die Testmodellierung entwickeln wir einen Ansatz, der Variabilitäts- sowie Versionsinformation von evolvierenden SPL gleichermaßen für die Modellierung einbezieht. Für die Änderungsauswirkungsanalyse definieren wir zwei Techniken, um zum einen Änderungen in Ausführungsabhängigkeiten zwischen zu testenden Varianten und ihren Versionen zu identifizieren und zum anderen die Auswirkungen eines Evolutionsschrittes auf die Variantenmenge zu bestimmen und zu klassifizieren. Für die Testfallselektion schlagen wir ein Abdeckungskriterium vor, das die Resultate der Auswirkungsanalyse einbezieht, um automatisierte Entscheidungen über einen Wiederholungstest von wiederverwendbaren Testfällen durchzuführen. Die abdeckungsgetriebene Testfallselektion ermöglicht somit die Reduktion der redundanten Testfallausführungen während des inkrementellen Testens von Varianten und Variantenversionen. Das Framework ist prototypisch implementiert und anhand von drei evolvierenden SPL evaluiert. Die Resultate zeigen, dass eine Aufwandsreduktion für das Testen evolvierender SPL erreicht wird

Modellbasierte Generierung und Reduktion von Testsuiten für Software-Produktlinien

Software-Produktlinienentwicklung ist ein Paradigma zur kostengünstigen Entwicklung vieler individueller aber sich ähnelnder Softwareprodukte aus einer gemeinsamen Softwareplattform heraus. Beispielsweise umfasst im Automotive-Bereich eine Software-Produktlinie (SPL) für ein Auto der Oberklasse typischerweise mehrere hunderttausend Softwaresystemvarianten. Um sicherzustellen, dass jede einzelne Produktvariante einer SPL in ihrer Funktionalität der Spezifikation entspricht, kann Testen verwendet werden. Da separates Testen jeder einzelnen Produktvariante meistens zu aufwändig ist, versuchen SPL-Testansätze die Gemeinsamkeiten der Produktvarianten beim Testen auszunutzen. So versuchen diese Ansätze geeignete Testartefakte wiederzuverwenden oder nur eine kleine repräsentative Menge von Produktvarianten stellvertretend für die ganze SPL zu testen. Da Software-Produktlinienentwicklung erst seit einigen Jahren verstärkt eingesetzt wird, sind im SPL-Test noch einige praxisnahe Probleme ungelöst. Beispielsweise existiert bisher kein Testansatz, mit dem sich eine gewisse Abdeckung bezüglich eines gewählten Überdeckungskriteriums auf allen Produktvarianten einer SPL effizient erreichen lässt. In dieser Arbeit wird ein Black-Box-Testfallgenerierungsansatz für Software-Produktlinien vorgestellt. Mit diesem Ansatz lassen sich für alle Produktvarianten einer SPL eine Menge von Testfällen aus einer formalen Spezifikation (Testmodell), die mit Variabilität angereichert wurde, effizient generieren. Diese Testfallmenge, im Folgenden als vollständige SPL-Testsuite bezeichnet, erreicht auf jeder Produktvariante der SPL eine vollständige Abdeckung bzgl. eines strukturellen Modell-Überdeckungskriteriums. Die Effizienz des Ansatzes beruht auf der Generierung von Testfällen, die variantenübergreifend wiederverwendbar sind. Dadurch müssen mit dem neuen Ansatz weniger Testfälle generiert werden als wenn dies für jede Produktvariante separat geschieht. Um bei Bedarf die Anzahl der generierten Testfälle reduzieren zu können, werden außerdem drei Algorithmen zur Testsuite-Reduktion vorgestellt. Die Neuerung der vorgestellten Algorithmen liegt im Vergleich zu existierenden Reduktionsalgorithmen für Testsuiten von Einzel-Softwaresystemen darin, dass die Existenz von variantenübergreifend verwendbaren Testfällen in einer SPL-Testsuite berücksichtig wird. Dadurch wird sichergestellt, dass trotz Testsuite-Reduktion die vollständige Testmodellabdeckung einer jeden Produktvariante durch die SPL-Testsuite erhalten bleibt. Sollte es aufgrund limitierter Ressourcen nicht möglich sein jede Produktvariante mit den in der vollständigen SPL-Testsuite enthaltenen Testfällen zu testen, kann mittels einer SPL-Testsuite eine kleine repräsentative Produktmenge aus der SPL bestimmt werden, deren Testergebnis (im begrenzten Rahmen) Rückschlüsse auf die Qualität der restlichen Produktvarianten zulässt. Zur Evaluation des Ansatzes wurde dieser prototypisch implementiert und auf zwei Fallbeispiele angewendet

Black-Box Testfall-Selektion und -Priorisierung für Software-Varianten und -Versionen

Software testing is a fundamental task in software quality assurance. Especially when dealing with several product variants or software versions under test, testing everything for each variant and version is infeasible due to limited testing resources. To cope with increasing complexity both in time (i.e., versions) and space (i.e., variants), new techniques have to be developed to focus on the most important parts for testing. In the past, regression testing techniques such as test case selection and prioritization have emerged to tackle these issues for single-software systems. However, testing of variants and versions is still a challenging task, especially when no source code is available. Most existing regression testing techniques analyze source code to identify important changes to be retested, i.e., they are likely to reveal a failure. To this end, this thesis contributes different techniques for both, variants and versions, to allow more efficient and effective testing in difficult black-box scenarios by identifying important test cases to be re-executed. Four major contributions in software testing are made. (1) We propose a test case prioritization framework for software product lines based on delta-oriented test models to reduce the redundancy in testing between different product variants.(2) We introduce a risk-based testing technique for software product lines. Our semi-automatic test case prioritization approach is able to compute risk values for test model elements and scales with large numbers of product variants. (3) For black-box software versions, we provide a test case selection technique based on genetic algorithms. In particular, seven different black-box selection objectives are defined, thus, we perform a multi-objective test case selection finding Pareto optimal test sets to reduce the testing effort. (4) We propose a novel test case prioritization technique based on supervised machine learning. It is able to imitate decisions made by experts based on different features, such as natural language test case descriptions and black-box meta-data. All of these techniques have been evaluated using the Body Comfort System case study. For testing of software versions, we also assesses our testing techniques using an industrial system. Our evaluation results indicate that our black-box testing approaches for software variants and versions are able to successfully reduce testing effort compared to existing techniques.Testen ist eine fundamentale Aufgabe zur Qualitätssicherung von modernen Softwaresystemen. Mangels limitierter Ressourcen ist das Testen von vielen Produktvarianten oder Versionen sehr herausfordernd und das wiederholte Ausführen aller Testfälle nicht wirtschaftlich. Um mit der Raum- (Varianten) und Zeitdimension (Versionen) in der Entwicklung umzugehen, wurden in der Vergangenheit verschiedene Testansätze entwickelt. Es existieren jedoch nach wie vor große Herausforderungen, welche es zu lösen gilt. Dies ist vor allem der Fall, wenn der Quellcode der getesteten Softwaresysteme unbekannt ist. Das Testen von Black-Box-Systemen erschwert die Identifikation von zu testenden Unterschieden zu vorher getesteten Varianten oder Versionen. In der Literatur finden sich wenige Ansätze, welche versuchen diese Herausforderungen zu lösen. Daher werden in dieser Dissertation neue Ansätze entwickelt und vorgestellt, welche beim Black-Box Testen von Software-Varianten und -Versionen helfen, wichtige Testfälle zur erneuten Ausführung zu identifizieren. Dies erspart die Ausführung von Testfällen, welche weder neues Verhalten testen noch mit hoher Wahrscheinlichkeit neue Fehler zu finden. Insgesamt leistet diese Dissertation die folgenden vier wissenschaftlichen Beiträge: (1) Ein modell-basiertes Framework zur Definition von Testfallpriorisierungsfunktionen für variantenreiche Systeme. Das Framework ermöglicht eine flexible Priorisierung von Testfällen für individuelle Produktvarianten. (2) Einen risiko-basierten Testfallpriorisierungsansatz für variantenreiche Systeme. Das Verfahren ermöglicht eine semi-automatisierte Berechnung von Risikowerten für Elemente von Produktvarianten und skaliert mit großen Produktzahlen. (3) Ein multi-kriterielles Testfallselektionsverfahren für den Regressionstest von Black-Box Software-Versionen. Es werden Black-Box Testkriterien aufgestellt und mittels eines genetischen Algorithmus optimiert um Pareto-optimale Testsets zu berechnen. (4) Ein Testfallpriorisierungsverfahren für Black-Box Regressionstests mit Hilfe von Machine Learning. Der verwendete Algorithmus imitiert Entscheidungen von Testexperten um wichtige Testfälle zu identifizieren. Diese Ansätze wurden alle mit Hilfe von Fallstudien evaluiert. Die resultierenden Ergebnisse zeigen, dass die Ansätze die gewünschten Ziele erreichen und helfen, wichtige Testfälle effektiv zu identifizieren. Insgesamt wird der Testaufwand im Vergleich zu existierenden Techniken verringert

Supporting Change in Product Lines Within the Context of Use Case-driven Development and Testing

Product Line Engineering (PLE) is a crucial practice in many software development environments where systems are complex and developed for multiple customers with varying needs. At the same time, many business contexts are use case-driven where use cases are the main artifacts driving requirements elicitation and many other development activities. In these contexts, variability information is often not explicitly represented, which leads to ad-hoc change management for use cases, domain models and test cases in product families. In this thesis, we address the problems of modeling variability in requirements with additional traceability to feature models and the manual and error prone requirements configuration and regression testing in product families. We provide the following contributions: - A modeling method for capturing variability information in product line use case and domain models by relying exclusively on commonly used artifacts in use-case driven development, thus avoiding unnecessary modeling overhead. - An approach for automated configuration of product specific use case and domain models that guides customers in making configuration decisions and automatically generates use case diagrams, use case specifications, and domain models for configured products. - A change impact analysis approach for evolving configuration decisions in product line use case models that automatically identifies the impact of decision changes on other decisions, and incrementally reconfigures product specific use case diagrams and specifications for evolving decisions. - An approach for automated classification and prioritization of system test cases in a family of products that automatically classifies and prioritizes, for each new product, system test cases of previous product(s) in a product line, and provides guidance in modifying existing system test cases to cover new use case scenarios that have not been tested in the product line before. All our approaches have been developed and evaluated in close collaboration with our industry partner IEE