Towards Statistical Prioritization for Software Product Lines Testing

Software Product Lines (SPL) are inherently difficult to test due to the combinatorial explosion of the number of products to consider. To reduce the number of products to test, sampling techniques such as combinatorial interaction testing have been proposed. They usually start from a feature model and apply a coverage criterion (e.g. pairwise feature interaction or dissimilarity) to generate tractable, fault-finding, lists of configurations to be tested. Prioritization can also be used to sort/generate such lists, optimizing coverage criteria or weights assigned to features. However, current sampling/prioritization techniques barely take product behavior into account. We explore how ideas of statistical testing, based on a usage model (a Markov chain), can be used to extract configurations of interest according to the likelihood of their executions. These executions are gathered in featured transition systems, compact representation of SPL behavior. We discuss possible scenarios and give a prioritization procedure illustrated on an example.Comment: Extended version published at VaMoS '14 (http://dx.doi.org/10.1145/2556624.2556635

Quantitative Analysis of Probabilistic Models of Software Product Lines with Statistical Model Checking

We investigate the suitability of statistical model checking techniques for analysing quantitative properties of software product line models with probabilistic aspects. For this purpose, we enrich the feature-oriented language FLan with action rates, which specify the likelihood of exhibiting particular behaviour or of installing features at a specific moment or in a specific order. The enriched language (called PFLan) allows us to specify models of software product lines with probabilistic configurations and behaviour, e.g. by considering a PFLan semantics based on discrete-time Markov chains. The Maude implementation of PFLan is combined with the distributed statistical model checker MultiVeStA to perform quantitative analyses of a simple product line case study. The presented analyses include the likelihood of certain behaviour of interest (e.g. product malfunctioning) and the expected average cost of products.Comment: In Proceedings FMSPLE 2015, arXiv:1504.0301

Towards quality assurance of software product lines with adversarial configurations

International audienceSoftware product line (SPL) engineers put a lot of effort to ensure that, through the setting of a large number of possible configuration options, products are acceptable and well-tailored to customers’ needs. Unfortunately, options and their mutual interactions create a huge configuration space which is intractable to exhaustively explore. Instead of testing all products, machine learning is increasingly employed to approximate the set of acceptable products out of a small training sample of configurations. Machine learning (ML) techniques can refine a software product line through learned constraints and a priori prevent non-acceptable products to be derived. In this paper, we use adversarial ML techniques to generate adversarial configurations fooling ML classifiers and pinpoint incorrect classifications of products (videos) derived from an industrial video generator. Our attacks yield (up to) a 100% misclassification rate and a drop in accuracy of 5%. We discuss the implications these results have on SPL quality assurance

Isolated Features Detection in Feature Models

Feature models are commonly used to describe software product lines in terms of features. Features are linked by relations, which may introduce errors in the model. This paper gives a description of isolated features and states the detection of them, as the first step in their treatment. Two implementations are given to automatically support isolated features detection and a third one that uses both and improves the performance

Domain Specific Languages for Managing Feature Models: Advances and Challenges

International audienceManaging multiple and complex feature models is a tedious and error-prone activity in software product line engineering. Despite many advances in formal methods and analysis techniques, the supporting tools and APIs are not easily usable together, nor unified. In this paper, we report on the development and evolution of the Familiar Domain-Specific Language (DSL). Its toolset is dedicated to the large scale management of feature models through a good support for separating concerns, composing feature models and scripting manipulations. We overview various applications of Familiar and discuss both advantages and identified drawbacks. We then devise salient challenges to improve such DSL support in the near future

Uniform and scalable SAT-sampling for configurable systems

Several relevant analyses on configurable software systems remain intractable because they require examining vast and highly-constrained configuration spaces. Those analyses could be addressed through statistical inference, i.e., working with a much more tractable sample that later supports generalizing the results obtained to the entire configuration space. To make this possible, the laws of statistical inference impose an indispensable requirement: each member of the population must be equally likely to be included in the sample, i.e., the sampling process needs to be "uniform". Various SAT-samplers have been developed for generating uniform random samples at a reasonable computational cost. Unfortunately, there is a lack of experimental validation over large configuration models to show whether the samplers indeed produce genuine uniform samples or not. This paper (i) presents a new statistical test to verify to what extent samplers accomplish uniformity and (ii) reports the evaluation of four state-of-the-art samplers: Spur, QuickSampler, Unigen2, and Smarch. According to our experimental results, only Spur satisfies both scalability and uniformity.Ministerio de Ciencia, Innovación y Universidades VITAL-3D DPI2016-77677-PMinisterio de Ciencia, Innovación y Universidades OPHELIA RTI2018-101204-B-C22Comunidad Autónoma de Madrid CAM RoboCity2030 S2013/MIT-2748Agencia Estatal de Investigación TIN2017-90644-RED

Modeling Variability in the Video Domain: Language and Experience Report

This paper reports about a new domain-specific variability modeling language, called VM, resulting from the close collaboration with industrial partners in the video domain. We expose the requirements and advanced variability constructs required to characterize and realize variations of physical properties of a video (such as objects' speed or scene illumination). The results of our experiments and industrial experience show that VM is effective to model complex variability information and can be exploited to synthesize video variants. We concluded that basic variability mechanisms are useful but not enough, attributes and multi-features are of prior importance, and meta-information is relevant for efficient variability analysis. In addition, we questioned the existence of one-size-fits-all variability modeling solution applicable in any industry. Yet, some common needs for modeling variability are becoming apparent such as support for attributes and multi-features.Ce document décrit un nouveau langage de modélisation dédiée à la variabilité, appelé VM, résultant de la collaboration avec des partenaires industriels dans le domaine de la vidéo. Nous exposons les exigences et les constructions de la variabilité avancées requises pour caractériser et implémenter les variations des propriétés physiques d'une vidéo (tels que la vitesse des objets ou l'illumination de la scène). Les résultats de nos expérimentations et de l'expérience industrielle montrent que VM est efficace pour modéliser l'information de variabilité complexe et peut être exploitée pour synthétiser des variantes de vidéo. Nous avons conclu que les mécanismes basiques de la variabilité sont certes utiles, mais insuffisants. Les attributs et multi-caractéristiques sont nécessaires alors que les méta-informations sont pertinentes pour une analyse efficace de la variabilité. En s'appuyant sur notre expérience, nous mettons en doute l'existence d'une solution de modélisation de la variabilité applicable à n'importe quelle industrie et domaine. Néanmoins, certains besoins communs pour la modélisation de la variabilité à sont apparents, comme le support pour les attributs et multi-caractéristiques