Optimal Minimisation of Pairwise-covering Test Configurations Using Constraint Programming

International audienceContext: Testing highly-configurable software systems is challenging due to a large number of test configurations that have to be carefully selected in order to reduce the testing effort as much as possible, while maintaining high software quality. Finding the smallest set of valid test configurations that ensure sufficient coverage of the system's feature interactions is thus the objective of validation engineers, especially when the execution of test configurations is costly or time-consuming. However, this problem is NP-hard in general and approximation algorithms have often been used to address it in practice. Objective: In this paper, we explore an alternative approach based on constraint programming that will allow engineers to increase the effectiveness of configuration testing while keeping the number of configurations as low as possible. Method: Our approach consists in using a (time-aware) minimisation algorithm based on constraint programming. Given the amount of time, our solution generates a minimised set of valid test configurations that ensure coverage of all pairs of feature values (a.k.a. pairwise coverage). The approach has been implemented in a tool called PACOGEN. Results: PACOGEN was evaluated on 224 feature models in comparison with the two existing tools that are based on a greedy algorithm. For 79% of 224 feature models, PACOGEN generated up to 60% fewer test configurations than the competitor tools. We further evaluated PACOGEN in the case study of large industrial highly-configurable video conferencing software with a feature model of 169 features, and found 60% fewer configurations compared with the manual approach followed by test engineers. The set of test configurations generated by PACOGEN decreased the time required by test engineers in manual test configuration by 85%, increasing the feature-pairs coverage at the same time. Conclusion: Extensive evaluation concluded that optimal minimisation of pairwise-covering test configurations is efficiently addressed using constraint programming techniques

Approche à contraintes pour la sélection de Covering Array

Aujourd'hui, les éditeurs logiciels ne conçoivent, développent et ne maintiennent plus leur offre logicielle avec comme cible un client unique. Au contraire, les offres logicielles sont conçues pour cibler plusieurs entités. Par conséquent, ces applications doivent s'intégrer dans des environnements différents et s'adapter aux besoins des clients. Ainsi, les produits logiciels développés ne sont plus des programmes uniques, mais des familles de produits. Les systèmes configurables facilitent la création de ces familles de produits. Grâce à eux il est possible de créer un produit logiciel en sélectionnant les fonctionnalités qui seront intégrées. Cependant, la validation de ces systèmes est une tâche complexe. Un système configurable peut générer plusieurs millions de configurations possibles. Il ne s'agit donc plus de valider un seul et unique produit, mais un ensemble de produits. Cet important nombre de configurations est un problème pour les personneschargées de la validation. Nous proposons trois contributions qui visent à mieux répondre aux problématiques liées à la variabilité lors des projets de test: une présentation détaillée de deux projets de test industriels faisant face à des problématiques de variabilité issus de deux entreprises : Cisco et Orange; une méthode originale basée sur les techniques de programmation par contraintes pour extraire des configurations de test qui respectent le critère Pairwise à partir d'un modèle explicite de la variabilité; une comparaison de cette approche par rapport aux techniques de l'état de l'art et une étude de l'application de cette technique de test sur deux projets de tests industriels.Nowadays, software companies develop and maintain their software for several clients. Consequently, these applications have to be integrated in heterogenous context and adapt to the user requriements. All these products are sharing commonalities but also differ in certain point due to business specific constraints. Configurable systems facilitate the creation of these product families. With them it is possible to create a software product by selecting the features that will be integrated, thus, the creation of a product is greatly simplified. However, the validation of these systems is a complex task. A configurable system can generate millions of possible configurations. Thus, validation process doesn't consist in validating a single product but in validating a set of products. This large number of configurations is a problem for those responsible of the validation. In this thesis we propose three contributions that aim to solve issues raised by variability during test projects : A detailled presentation of two industrial test projects coping tat variaibility issues; an original methodology based on constraint programming techniques to select test configurations that respect pairwise criteria from a feature model; an exhaustive comparison of this approach with the existing approches and a detailled study of the application of a such techniques on the two industrials projects.RENNES1-Bibl. électronique (352382106) / SudocSudocFranceF

A Technique for Agile and Automatic Interaction Testing for Product Lines

Part 2: Testing in PracticeInternational audienceProduct line developers must ensure that existing and new features work in all products. Adding to or changing a product line might break some of its features. In this paper, we present a technique for automatic and agile interaction testing for product lines. The technique enables developers to know if features work together with other features in a product line, and it blends well into a process of continuous integration. The technique is evaluated with two industrial applications, testing a product line of safety devices and the Eclipse IDEs. The first case shows how existing test suites are applied to the products of a 2-wise covering array to identify two interaction faults. The second case shows how over 400,000 test executions are performed on the products of a 2-wise covering array using over 40,000 existing automatic tests to identify potential interactions faults