Spinal Test Suites for Software Product Lines

A major challenge in testing software product lines is efficiency. In particular, testing a product line should take less effort than testing each and every product individually. We address this issue in the context of input-output conformance testing, which is a formal theory of model-based testing. We extend the notion of conformance testing on input-output featured transition systems with the novel concept of spinal test suites. We show how this concept dispenses with retesting the common behavior among different, but similar, products of a software product line.Comment: In Proceedings MBT 2014, arXiv:1403.704

Software product line testing - a systematic mapping study

Context: Software product lines (SPL) are used in industry to achieve more efficient software development. However, the testing side of SPL is underdeveloped. Objective: This study aims at surveying existing research on SPL testing in order to identify useful approaches and needs for future research. Method: A systematic mapping study is launched to find as much literature as possible, and the 64 papers found are classified with respect to focus, research type and contribution type. Results: A majority of the papers are of proposal research types (64 %). System testing is the largest group with respect to research focus (40%), followed by management (23%). Method contributions are in majority. Conclusions: More validation and evaluation research is needed to provide a better foundation for SPL testing

Colored model based testing for software product lines (CMBT-SWPL)

Over the last decade, the software product line domain has emerged as one of the mostpromising software development paradigms. The main benefits of a software product lineapproach are improvements in productivity, time to market, product quality, and customersatisfaction.Therefore, one topic that needs greater emphasis is testing of software product lines toachieve the required software quality assurance. Our concern is how to test a softwareproduct line as early as possible in order to detect errors, because the cost of error detectedIn early phases is much less compared to the cost of errors when detected later.The method suggested in this thesis is a model-based, reuse-oriented test technique calledColored Model Based Testing for Software Product Lines (CMBT-SWPL). CMBT-SWPLis a requirements-based approach for efficiently generating tests for products in a soft-ware product line. This testing approach is used for validation and verification of productlines. It is a novel approach to test product lines using a Colored State Chart (CSC), whichconsiders variability early in the product line development process. More precisely, the vari-ability will be introduced in the main components of the CSC. Accordingly, the variabilityis preserved in test cases, as they are generated from colored test models automatically.During domain engineering, the CSC is derived from the feature model. By coloring theState Chart, the behavior of several product line variants can be modeled simultaneouslyin a single diagram and thus address product line variability early. The CSC representsthe test model, from which test cases using statistical testing are derived.During application engineering, these colored test models are customized for a specificapplication of the product line. At the end of this test process, the test cases are generatedagain using statistical testing, executed and the test results are ready for evaluation. Inxaddition, the CSC will be transformed to a Colored Petri Net (CPN) for verification andsimulation purposes.The main gains of applying the CMBT-SWPL method are early detection of defects inrequirements, such as ambiguities incompleteness and redundancy which is then reflectedin saving the test effort, time, development and maintenance costs

Exploring regression testing and software product line testing - research and state of practice

In large software organizations with a product line development approach a selective testing of product variants is necessary in order to keep pace with the decreased development time for new products, enabled by the systematic reuse. The close relationship between products in product line indicates an option to reduce the testing effort due to redundancy. In many cases test selection is performed manually, based on test leaders’ expertise. This makes the cost and quality of the testing highly dependent on the skills and experience of the test leaders. There is a need in industry for systematic approaches to test selection. The goal of our research is to improve the control of the testing and reduce the amount of redundant testing in the product line context by applying regression test selection strategies. In this thesis, the state of art of regression testing and software product line testing are explored. Two extensive systematic reviews are conducted as well as an industrial survey of regression testing state of practice and an industrial evaluation of a pragmatic regression test selection strategy. Regression testing is not an isolated one-off activity, but rather an activity of varying scope and preconditions, strongly dependent on the context in which it is applied. Several techniques for regression test selection are proposed and evaluated empirically but in many cases the context is too specific for a technique to be easily applied directly by software developers. In order to improve the possibility for generalizing empirical results on regression test selection, guidelines for reporting the testing context are discussed in this thesis. Software product line testing is a relatively new research area. The understanding about challenges is well established but when looking for solutions to these challenges, we mostly find proposals, and empirical evaluations are sparse. Regression test selection strategies proposed in literature are not easily applicable in the product line context. Instead, control may be increased by increased visibility of the effects of testing and proper measurements of software quality. Focus of our future work will be on how to guide the planning and assessment of regression testing activities in large, complex reuse based systems, by visualizing the quality achieved in different parts of the system and evaluating the effects of different selection strategies when applied in various regression testing situations

Testing variability-intensive systems using automated analysis: an application to Android

Software product lines are used to develop a set of software products that, while being different, share a common set of features. Feature models are used as a compact representation of all the products (e.g., possible configurations) of the product line. The number of products that a feature model encodes may grow exponentially with the number of features. This increases the cost of testing the products within a product line. Some proposals deal with this problem by reducing the testing space using different techniques. However, a daunting challenge is to explore how the cost and value of test cases can be modeled and optimized in order to have lower-cost testing processes. In this paper, we present TESting vAriAbiLity Intensive Systems (TESALIA), an approach that uses automated analysis of feature models to optimize the testing of variability-intensive systems. We model test value and cost as feature attributes, and then we use a constraint satisfaction solver to prune, prioritize and package product line tests complementing prior work in the software product line testing literature. A prototype implementation of TESALIA is used for validation in an Android example showing the benefits of maximizing the mobile market share (the value function) while meeting a budgetary constraint.Ministerio de Economía y Competitividad TIN2012-32273Junta de Andalucía TIC-5906Junta de Andalucía TIC-186

Architecture-Based Unit Testing of the Flight Software Product Line

This paper presents an analysis of the unit testing approach developed and used by the Core Flight Software (CFS) product line team at the NASA GSFC. The goal of the analysis is to understand, review, and reconunend strategies for improving the existing unit testing infrastructure as well as to capture lessons learned and best practices that can be used by other product line teams for their unit testing. The CFS unit testing framework is designed and implemented as a set of variation points, and thus testing support is built into the product line architecture. The analysis found that the CFS unit testing approach has many practical and good solutions that are worth considering when deciding how to design the testing architecture for a product line, which are documented in this paper along with some suggested innprovennents

Optimizing Model-Based Software Product Line Testing with Graph Transformations

Software Product Lines (SPLs) are increasing in relevance and importance as various domains strive to cope with the challenges of supporting a high degree of variability in modern software systems.  Especially the systematic testing of SPLs is non-trivial as a high degree of variability implies a vast number of possible products.As testing every valid product individually quickly becomes infeasible, heuristics are often used to choose a representative subset of products to be tested.  MoSo-PoLiTe (Model-Based Software Product Line Testing) is a framework for SPL testing that combines and applies combinatorial (in particular pairwise) and model-based testing to SPL feature models.  In this paper, we (1) present MoSo-PoLiTe as a novel case study for graph transformations in general and Story Driven Modelling (SDM) in particular, (2) show why we consider SDMs to be ideal for rapid prototyping optimization strategies in this context, and (3) evaluate our implemented optimizations and quantify the realized improvements for MoSo-PoLiTe

A parallel evolutionary algorithm for prioritized pairwise testing of software product lines

Lopez-Herrejon, R. Erick, Ferrer J., Chicano F., Haslinger E. Nicole, Egyed A., & Alba E. (2014). A parallel evolutionary algorithm for prioritized pairwise testing of software product lines. (Arnold, D. V., Ed.).Genetic and Evolutionary Computation Conference, GECCO '14, Vancouver, BC, Canada, July 12-16, 2014. 1255–1262.Software Product Lines (SPLs) are families of related software systems, which provide different feature combinations. Different SPL testing approaches have been proposed. However, despite the extensive and successful use of evolutionary computation techniques for software testing, their application to SPL testing remains largely unexplored. In this paper we present the Parallel Prioritized product line Genetic Solver (PPGS), a parallel genetic algorithm for the generation of prioritized pairwise testing suites for SPLs. We perform an extensive and comprehensive analysis of PPGS with 235 feature models from a wide range of number of features and products, using 3 different priority assignment schemes and 5 product prioritization selection strategies. We also compare PPGS with the greedy algorithm prioritized-ICPL. Our study reveals that overall PPGS obtains smaller covering arrays with an acceptable performance difference with prioritized-ICPL.Austrian Science Fund (FWF) project P25289-N15 and Lise Meitner Fellowship M1421-N15. Spanish Ministry of Economy and Competitiveness and FEDER under contract TIN2011-28194 and fellowship Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech.BES-2012-055967. Project 8.06/5.47.4142 in collaboration with the VSB-Tech. Univ. of Ostrava and Universidad de Málaga, Andalucía Tech