Modellbasiertes Regressionstesten von Varianten und Variantenversionen

The quality assurance of software product lines (SPL) achieved via testing is a crucial and challenging activity of SPL engineering. In general, the application of single-software testing techniques for SPL testing is not practical as it leads to the individual testing of a potentially vast number of variants. Testing each variant in isolation further results in redundant testing processes by means of redundant test-case executions due to the shared commonality. Existing techniques for SPL testing cope with those challenges, e.g., by identifying samples of variants to be tested. However, each variant is still tested separately without taking the explicit knowledge about the shared commonality and variability into account to reduce the overall testing effort. Furthermore, due to the increasing longevity of software systems, their development has to face software evolution. Hence, quality assurance has also to be ensured after SPL evolution by testing respective versions of variants. In this thesis, we tackle the challenges of testing redundancy as well as evolution by proposing a framework for model-based regression testing of evolving SPLs. The framework facilitates efficient incremental testing of variants and versions of variants by exploiting the commonality and reuse potential of test artifacts and test results. Our contribution is divided into three parts. First, we propose a test-modeling formalism capturing the variability and version information of evolving SPLs in an integrated fashion. The formalism builds the basis for automatic derivation of reusable test cases and for the application of change impact analysis to guide retest test selection. Second, we introduce two techniques for incremental change impact analysis to identify (1) changing execution dependencies to be retested between subsequently tested variants and versions of variants, and (2) the impact of an evolution step to the variant set in terms of modified, new and unchanged versions of variants. Third, we define a coverage-driven retest test selection based on a new retest coverage criterion that incorporates the results of the change impact analysis. The retest test selection facilitates the reduction of redundantly executed test cases during incremental testing of variants and versions of variants. The framework is prototypically implemented and evaluated by means of three evolving SPLs showing that it achieves a reduction of the overall effort for testing evolving SPLs.Testen ist ein wichtiger Bestandteil der Entwicklung von Softwareproduktlinien (SPL). Aufgrund der potentiell sehr großen Anzahl an Varianten einer SPL ist deren individueller Test im Allgemeinen nicht praktikabel und resultiert zudem in redundanten Testfallausführungen, die durch die Gemeinsamkeiten zwischen Varianten entstehen. Existierende SPL-Testansätze adressieren diese Herausforderungen z.B. durch die Reduktion der Anzahl an zu testenden Varianten. Jedoch wird weiterhin jede Variante unabhängig getestet, ohne dabei das Wissen über Gemeinsamkeiten und Variabilität auszunutzen, um den Testaufwand zu reduzieren. Des Weiteren muss sich die SPL-Entwicklung mit der Evolution von Software auseinandersetzen. Dies birgt weitere Herausforderungen für das SPL-Testen, da nicht nur für Varianten sondern auch für ihre Versionen die Qualität sichergestellt werden muss. In dieser Arbeit stellen wir ein Framework für das modellbasierte Regressionstesten von evolvierenden SPL vor, das die Herausforderungen des redundanten Testens und der Software-Evolution adressiert. Das Framework vereint Testmodellierung, Änderungsauswirkungsanalyse und automatische Testfallselektion, um einen inkrementellen Testprozess zu definieren, der Varianten und Variantenversionen unter Ausnutzung des Wissens über gemeinsame Funktionalität und dem Wiederverwendungspotential von Testartefakten und -resultaten effizient testet. Für die Testmodellierung entwickeln wir einen Ansatz, der Variabilitäts- sowie Versionsinformation von evolvierenden SPL gleichermaßen für die Modellierung einbezieht. Für die Änderungsauswirkungsanalyse definieren wir zwei Techniken, um zum einen Änderungen in Ausführungsabhängigkeiten zwischen zu testenden Varianten und ihren Versionen zu identifizieren und zum anderen die Auswirkungen eines Evolutionsschrittes auf die Variantenmenge zu bestimmen und zu klassifizieren. Für die Testfallselektion schlagen wir ein Abdeckungskriterium vor, das die Resultate der Auswirkungsanalyse einbezieht, um automatisierte Entscheidungen über einen Wiederholungstest von wiederverwendbaren Testfällen durchzuführen. Die abdeckungsgetriebene Testfallselektion ermöglicht somit die Reduktion der redundanten Testfallausführungen während des inkrementellen Testens von Varianten und Variantenversionen. Das Framework ist prototypisch implementiert und anhand von drei evolvierenden SPL evaluiert. Die Resultate zeigen, dass eine Aufwandsreduktion für das Testen evolvierender SPL erreicht wird

Fundamental Approaches to Software Engineering

computer software maintenance; computer software selection and evaluation; formal logic; formal methods; formal specification; programming languages; semantics; software engineering; specifications; verificatio

Automated analysis of feature models: Quo vadis?

Feature models have been used since the 90's to describe software product lines as a way of reusing common parts in a family of software systems. In 2010, a systematic literature review was published summarizing the advances and settling the basis of the area of Automated Analysis of Feature Models (AAFM). From then on, different studies have applied the AAFM in different domains. In this paper, we provide an overview of the evolution of this field since 2010 by performing a systematic mapping study considering 423 primary sources. We found six different variability facets where the AAFM is being applied that define the tendencies: product configuration and derivation; testing and evolution; reverse engineering; multi-model variability-analysis; variability modelling and variability-intensive systems. We also confirmed that there is a lack of industrial evidence in most of the cases. Finally, we present where and when the papers have been published and who are the authors and institutions that are contributing to the field. We observed that the maturity is proven by the increment in the number of journals published along the years as well as the diversity of conferences and workshops where papers are published. We also suggest some synergies with other areas such as cloud or mobile computing among others that can motivate further research in the future.Ministerio de Economía y Competitividad TIN2015-70560-RJunta de Andalucía TIC-186

Software diversity: state of the art and perspectives

International audienceDiversity is prevalent in modern software systems to facilitate adapting the software to customer requirements or the execution environment. Diversity has an impact on all phases of the software development process. Appropriate means and organizational structures are required to deal with the additional complexity introduced by software variability. This introductory article to the special section "Software Diversity--Modeling, Analysis and Evolution" provides an overview of the current state of the art in diverse systems development and discusses challenges and potential solutions. The article covers requirements analysis, design, implementation, verification and validation, maintenance and evolution as well as organizational aspects. It also provides an overview of the articles which are part of this special section and addresses particular issues of diverse systems development

Exploring regression testing and software product line testing - research and state of practice

In large software organizations with a product line development approach a selective testing of product variants is necessary in order to keep pace with the decreased development time for new products, enabled by the systematic reuse. The close relationship between products in product line indicates an option to reduce the testing effort due to redundancy. In many cases test selection is performed manually, based on test leaders’ expertise. This makes the cost and quality of the testing highly dependent on the skills and experience of the test leaders. There is a need in industry for systematic approaches to test selection. The goal of our research is to improve the control of the testing and reduce the amount of redundant testing in the product line context by applying regression test selection strategies. In this thesis, the state of art of regression testing and software product line testing are explored. Two extensive systematic reviews are conducted as well as an industrial survey of regression testing state of practice and an industrial evaluation of a pragmatic regression test selection strategy. Regression testing is not an isolated one-off activity, but rather an activity of varying scope and preconditions, strongly dependent on the context in which it is applied. Several techniques for regression test selection are proposed and evaluated empirically but in many cases the context is too specific for a technique to be easily applied directly by software developers. In order to improve the possibility for generalizing empirical results on regression test selection, guidelines for reporting the testing context are discussed in this thesis. Software product line testing is a relatively new research area. The understanding about challenges is well established but when looking for solutions to these challenges, we mostly find proposals, and empirical evaluations are sparse. Regression test selection strategies proposed in literature are not easily applicable in the product line context. Instead, control may be increased by increased visibility of the effects of testing and proper measurements of software quality. Focus of our future work will be on how to guide the planning and assessment of regression testing activities in large, complex reuse based systems, by visualizing the quality achieved in different parts of the system and evaluating the effects of different selection strategies when applied in various regression testing situations

Seamless Variability Management With the Virtual Platform

Customization is a general trend in software engineering, demanding systems that support variable stakeholder requirements. Two opposing strategies are commonly used to create variants: software clone & own and software configuration with an integrated platform. Organizations often start with the former, which is cheap, agile, and supports quick innovation, but does not scale. The latter scales by establishing an integrated platform that shares software assets between variants, but requires high up-front investments or risky migration processes. So, could we have a method that allows an easy transition or even combine the benefits of both strategies? We propose a method and tool that supports a truly incremental development of variant-rich systems, exploiting a spectrum between both opposing strategies. We design, formalize, and prototype the variability-management framework virtual platform. It bridges clone & own and platform-oriented development. Relying on programming-language-independent conceptual structures representing software assets, it offers operators for engineering and evolving a system, comprising: traditional, asset-oriented operators and novel, feature-oriented operators for incrementally adopting concepts of an integrated platform. The operators record meta-data that is exploited by other operators to support the transition. Among others, they eliminate expensive feature-location effort or the need to trace clones. Our evaluation simulates the evolution of a real-world, clone-based system, measuring its costs and benefits.Comment: 13 pages, 10 figures; accepted for publication at the 43rd International Conference on Software Engineering (ICSE 2021), main technical trac

Interdisziplinäre Variabilitätsmodellierung und Performance Analyse für langlebige Systeme in der Automatisierungstechnik

In this day and age, automation systems have to deal with differing customer needs, environmental requirements and multiple application contexts. Automation systems have to be variable enough to satisfy all of these demands. The development and maintenance of such highly-customizable systems is a challenging task and becomes increasingly more difficult considering multiple involved engineering disciplines and long lifetimes, which is characteristic for industrial systems of the automation domain. Software product line engineering provides developers with fundamental concepts to manage the variability of such systems. However, these concepts are not established in the domain of automation systems. In addition, the involvement of multiple engineering disciplines poses a threat to existing SPL techniques. This thesis contributes novel approaches to improve the development and maintenance of software-intensive automation product lines. In total, three major contributions are made, spanning across the complete design phase of an automation system. (1) The feature modeling process is improved by detecting hidden dependencies between interrelated feature models from separate engineering disciplines. Furthermore, hidden dependencies and occurring defects in the feature models are explained in a user-friendly manner. (2) A model-driven development approach is introduced consisting of UML models, which are extended with delta modeling to manage variability in the automation product line. The models encompass information that is needed to automatically derive and analyze a performance model. (3) Subsequently, an efficient family-product-based performance analysis is proposed for the previously derived UML models that is vastly superior compared to common product-based approaches. All of these techniques have been evaluated using multiple case studies, with one being a real-world automation system.In der heutigen Zeit sehen sich Automatisierungssysteme mit einer steigenden Komplexität konfrontiert. Einzelne Kunden haben unterschiedliche Ansprüche an das System und ebenso müssen Umweltbedingungen der verschiedenen Betriebsumgebungen sowie abweichende Anwendungsgebiete bei der Entwicklung eines Automatisierungssystems berücksichtigt werden. Diese Komplexitätsaspekte werden unter dem Stichwort Variabilität zusammengefasst. Ein Automatisierungssystem muss in der Lage sein, sämtliche Anforderungen zu erfüllen. Die Entwicklung und Wartung dieser Systeme wird jedoch durch die stetig wachsende Variabilität und eine potentiell lange Lebensdauer immer schwieriger. Zusätzlich sind an dem Entwicklungsprozess eines Automatisierungssystems mehrere Ingenieursdisziplinen beteiligt. Die Techniken aus dem Bereich der Software-Produktlinienentwicklung bilden Lösungen, um die Variabilität beherrschbar zu machen. In der Automatisierungstechnik sind diese Techniken weitgehend unbekannt und durch den interdisziplinären Charakter oft nicht ausreichend. Daher werden in dieser Dissertation neue Ansätze entwickelt und vorgestellt, die auf die Domäne der Automatisierungstechnik zugeschnitten sind. Insgesamt leistet diese Dissertation folgende drei wissenschaftlichen Beiträge: (1) Die Entwicklung von Feature-Modellen wird durch die Detektion von verborgenen Abhängigkeiten, die zwischen Feature-Modellen der unterschiedlichen Ingenieursdisziplinen existieren, verbessert. Gleichzeitig liefert der vorgestellte Algorithmus die Erklärung für die Existenz dieser Abhängigkeiten. Dieses Konzept wird auf weitere Defekte in Feature-Modellen ausgeweitet. (2) Einen modell-basierten Ansatz zur Entwicklung eines Automatisierungssystems. Der Ansatz basiert auf Modellen aus der UML, die mit Hilfe der Delta Modellierung Variabilität abbilden können. Zusätzlich sind die Modelle mit Informationen über Performance Eigenschaften angereichert und erlauben die automatische Ableitung eines Performance-Modells. (3) Eine effiziente Performance Analyse von allen Varianten des Automatisierungssystems, die auf den zuvor abgeleiteten Performance-Modellen basiert. Alle Beiträge wurden mit Fallstudien evaluiert. Eine Fallstudie repräsentiert ein reales Automatisierungssystem

Software and systems traceability for safety-critical projects: report from Dagstuhl Seminar 15162

This report documents the program and the outcomes of Dagstuhl Seminar 15162 on “Software and Systems Traceability for Safety-Critical Projects”. The event brought together researchers and industrial practitioners working in the field of safety critical software to explore the needs, challenges, and solutions for Software and Systems Traceability in this domain. The goal was to explore the gap between the traceability prescribed by guidelines and that delivered by manufacturers, and starting from a clean slate, to clearly articulate traceability needs for safety-critical software systems, to identify challenges, explore solutions, and to propose a set of principles and domain-specific exemplars for achieving traceability in safety critical systems

Feature-based Approach to Bridge the Information Technology and Business Gap

The gap between business goals (problem domain), such as cost reduction, new business processes, increasing competitive advantage, etc., and the supporting Information Technology infrastructure (solution domain), such as the ability to implement software solutions to achieve these goals, is complex and challenging to bridge. This gap emerges for many reasons; for instance, inefficient communication, domain terminology misunderstanding or external factors, e.g. business change. As most business and software products can be described by a set of features, a promising solution would be to link both the problem and solution domains based on these features. Thus, the proposed approach aims to bridge the gap between the problem and the solution domains by using a feature-based technique in order to provide a quick and efficient means for understanding the relationships between IT solutions and business goals. The novelty of the proposed framework emanates from the three characteristics of the business-IT gap: the problem domain, the solution domain and the matching process. Besides the proposed feature-based IT-business framework, other contributions are proposed: a feature extracting method and feature matching algorithms. The proposed approach is achieved in three phases. The first phase is to decompose business needs and transform them into a feature model (presented in UML diagrams); this is represented as a top-to-middle process. The second phase is a reverse engineering process. A system program code is sliced into modules and transformed into feature-based models (again, in UML diagrams); these are represented as a bottom-to-middle process. The third phase is a model-driven engineering process. It uses model comparison techniques to match the UML feature models of the top-to-middle and bottom-to-middle phases. The presented approach in this research shows that features elicited from the business goals can be matched to features extracted from in the IT side. This proposed approach is feasible and able to provide a quick and efficient means for improving feature-based business IT matching. Two case studies are presented to demonstrate that the feature-oriented view of features from the users' perspective can be matched to the feature-oriented view of features in the IT side. This matching can serve to remove any ambiguities that may cause difficulties in the cases of system maintenance or system evolution, in particular when there are changes in requirements, which is to be expected when there is any business change