Evolution, testing and configuration of variability intensive systems

Tesis descargada desde ResearchGateOne of the key characteristics of software is its ability to be adapted and configured to different scenarios. Recently, software variability has been studied as a first-class concept in different domains ranging from software product lines to pervasive systems. Variability is the ability of a software product to vary depending on different circumstances. Variability intensive systems are those software products where variability management is a core engineering activity. The varying parts of those systems are commonly modeled by us- ing different variability model flavors, being feature modeling one of the most common ones. Feature models were first introduced by Kang et al. back in 1990 and are a compact representation of a set of configurations in a variability intensive system. The large number of configurations that a feature model can encode makes the manual analysis of feature models an error prone and costly task. Then, computer-aided mechanisms appeared as a solution to extract useful information from feature models. This process of extracting information from feature models is known as ¿Automated Analysis of Feature models¿ that has been one of the main areas of research in the last years where more than thirty analysis operations have been proposed.Premio Extraordinario de Doctorado U

The Drupal Framework: a Case Study to Evaluate Variability Testing Techniques

Variability testing techniques search for effective but manageable test suites that lead to the rapid detection of faults in systems with high variability. Evaluating the effectiveness of these techniques in real settings is a must but challenging due to the lack of variability-intensive systems with available code, automated tests and fault reports. in this paper, we propose using the Drupal framework as a case study to evaluate variability testing techniques. First, we represent the framework variability as a feature model. Then, we report on extensive data extracted from the Drupal git repository and the Drupal issue tracking system. Among other results, we identified 378 faults in single features and 11 faults triggered by the interaction between two of the features of Drupal v7.23, reported during a one-year period. These data may give a new insight into the distribution of faults in variability-intensive systems and the fault propensity of features. To show the feasibility of our work, we used the case study to evaluate the effectiveness of a historybased test case prioritization criterion. Results suggest that this technique could contribute to accelerate the detection of faults of test suites based on combinatorial testing.CICYT TIN2009-07366CICYT TIN2012-32273Junta de Andalucía TIC-5906Junta de Andalucía TIC-186

Testing variability-intensive systems using automated analysis: an application to Android

Software product lines are used to develop a set of software products that, while being different, share a common set of features. Feature models are used as a compact representation of all the products (e.g., possible configurations) of the product line. The number of products that a feature model encodes may grow exponentially with the number of features. This increases the cost of testing the products within a product line. Some proposals deal with this problem by reducing the testing space using different techniques. However, a daunting challenge is to explore how the cost and value of test cases can be modeled and optimized in order to have lower-cost testing processes. In this paper, we present TESting vAriAbiLity Intensive Systems (TESALIA), an approach that uses automated analysis of feature models to optimize the testing of variability-intensive systems. We model test value and cost as feature attributes, and then we use a constraint satisfaction solver to prune, prioritize and package product line tests complementing prior work in the software product line testing literature. A prototype implementation of TESALIA is used for validation in an Android example showing the benefits of maximizing the mobile market share (the value function) while meeting a budgetary constraint.Ministerio de Economía y Competitividad TIN2012-32273Junta de Andalucía TIC-5906Junta de Andalucía TIC-186

Automated analysis of feature models: Quo vadis?

Feature models have been used since the 90's to describe software product lines as a way of reusing common parts in a family of software systems. In 2010, a systematic literature review was published summarizing the advances and settling the basis of the area of Automated Analysis of Feature Models (AAFM). From then on, different studies have applied the AAFM in different domains. In this paper, we provide an overview of the evolution of this field since 2010 by performing a systematic mapping study considering 423 primary sources. We found six different variability facets where the AAFM is being applied that define the tendencies: product configuration and derivation; testing and evolution; reverse engineering; multi-model variability-analysis; variability modelling and variability-intensive systems. We also confirmed that there is a lack of industrial evidence in most of the cases. Finally, we present where and when the papers have been published and who are the authors and institutions that are contributing to the field. We observed that the maturity is proven by the increment in the number of journals published along the years as well as the diversity of conferences and workshops where papers are published. We also suggest some synergies with other areas such as cloud or mobile computing among others that can motivate further research in the future.Ministerio de Economía y Competitividad TIN2015-70560-RJunta de Andalucía TIC-186

Variability testing in the wild: the Drupal case study

Variability testing techniques search for effective and manageable test suites that lead to the rapid detection of faults in systems with high variability. Evaluating the effectiveness of these techniques in realistic settings is a must, but challenging due to the lack of variability intensive systems with available code, automated tests and fault reports. In this article, we propose using the Drupal framework as a case study to evaluate variability testing techniques. First, we represent the framework variability using a feature model. Then, we report on extensive non–functional data extracted from the Drupal Git repository and the Drupal issue tracking system. Among other results, we identified 3,392 faults in single features and 160 faults triggered by the interaction of up to 4 features in Drupal v7.23. We also found positive correlations relating the number of bugs in Drupal features to their size, cyclomatic complexity, number of changes and fault history. To show the feasibility of our work, we evaluated the effectiveness of non–functional data for test case prioritization in Drupal. Results show that non–functional attributes are effective at accelerating the detection of faults, outperforming related prioritization criteria as test case similarity.Ministerio de Economía y Competitividad IPT-2012-0890-390000Ministerio de Economía y Competitividad TIN2012-3227

Automated metamorphic testing of variability analysis tools

Variability determines the capability of software applications to be configured and customized. A common need during the development of variability–intensive systems is the automated analysis of their underlying variability models, e.g. detecting contradictory configuration options. The analysis operations that are performed on variability models are often very complex, which hinders the testing of the corresponding analysis tools and makes difficult, often infeasible, to determine the correctness of their outputs, i.e. the well–known oracle problem in software testing. In this article, we present a generic approach for the automated detection of faults in variability analysis tools overcoming the oracle problem. Our work enables the generation of random variability models together with the exact set of valid configurations represented by these models. These test data are generated from scratch using step–wise transformations and assuring that certain constraints (a.k.a. metamorphic relations) hold at each step. To show the feasibility and generalizability of our approach, it has been used to automatically test several analysis tools in three variability domains: feature models, CUDF documents and Boolean formulas. Among other results, we detected 19 real bugs in 7 out of the 15 tools under test.CICYT TIN2012-32273CICYT IPT-2012- 0890-390000Junta de Andalucía TIC-5906Junta de Andalucía P12-TIC- 186

Towards Automated Execution and Evaluation of Simulated Prototype HRI Experiments

Lier F, Lütkebohle I, Wachsmuth S. Towards Automated Execution and Evaluation of Simulated Prototype HRI Experiments. In: HRI '14 Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction. New York, NY, USA: ACM; 2014: 230-231.Autonomous robots are highly relevant targets for interaction studies, but can exhibit behavioral variability that confounds experimental validity. Currently, testing on real systems is the only means to prevent this, but remains very labour-intensive and often happens too late. To improve this situation, we are working towards early testing by means of partial simulation, with automated assessment, and based upon continuous software integration to prevent regressions. We will introduce the concept and describe a proof-of-concept that demonstrates fast feedback and coherent experiment results across repeated trials

Spinal Test Suites for Software Product Lines

A major challenge in testing software product lines is efficiency. In particular, testing a product line should take less effort than testing each and every product individually. We address this issue in the context of input-output conformance testing, which is a formal theory of model-based testing. We extend the notion of conformance testing on input-output featured transition systems with the novel concept of spinal test suites. We show how this concept dispenses with retesting the common behavior among different, but similar, products of a software product line.Comment: In Proceedings MBT 2014, arXiv:1403.704