Towards Statistical Prioritization for Software Product Lines Testing

Software Product Lines (SPL) are inherently difficult to test due to the combinatorial explosion of the number of products to consider. To reduce the number of products to test, sampling techniques such as combinatorial interaction testing have been proposed. They usually start from a feature model and apply a coverage criterion (e.g. pairwise feature interaction or dissimilarity) to generate tractable, fault-finding, lists of configurations to be tested. Prioritization can also be used to sort/generate such lists, optimizing coverage criteria or weights assigned to features. However, current sampling/prioritization techniques barely take product behavior into account. We explore how ideas of statistical testing, based on a usage model (a Markov chain), can be used to extract configurations of interest according to the likelihood of their executions. These executions are gathered in featured transition systems, compact representation of SPL behavior. We discuss possible scenarios and give a prioritization procedure illustrated on an example.Comment: Extended version published at VaMoS '14 (http://dx.doi.org/10.1145/2556624.2556635

A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation

Otsingupõhine tarkvara testimine kasutab metaheuristilisi algoritme, et automatiseerida testide genereerimist. Selle töö eesmärgiks on osaliselt taasluua 2010. aastal kirjutatud Ali et al. artikkel, et uurida, kuidas on aastatel 2008-2015 kasutatud metaheuristilisi algoritme testide loomiseks. See töö analüüsib, kuidas on antud artiklid koostatud ning kuidas neis on algoritmide maksumust ja efektiivsust hinnatud. Kogutud tulemusi võrreldakse Ali et al. tulemustega.Search based software testing uses metaheuristic algorithms to automate the generation of test cases. This thesis partially replicates a literature study published in 2010 by Ali et al. to determine how studies published in 2008-2015 use metaheuristic algorithms to automate the generation of test cases. The thesis analyses how these studies were conducted and how the cost-effectiveness is assessed in these papers. The trends detected in the new publications are compared to those presented in Ali et al

SmarTest: A Test Case Prioritization Tool for Drupal

Test case prioritization techniques aim to identify the optimal ordering of tests to accelerate the detection of faults. The importance of these techniques has been recognized in the context of Software Product Lines (SPLs), where the potentially huge number of products makes testing extremely challenging. We found that the open source Drupal framework shares most of the principles and challenges of SPL development and it can be considered a real-world example of family of products. In a previous work, we represented the Drupal configuration space as a feature model and we collected extra functional information about its features from open repositories. Part of this data proved to be a good indicator of faults propensity in Drupal features. Thus, they become valuable assets to prioritize tests in individual Drupal products. In this paper, we present SmarTest, a test prioritization tool for accelerating the detection of faults in Drupal. SmarTest has been developed as an extension of the Drupal core testing system. SmarTest supports the prioritization of tests providing faster feedback and letting testers begin correcting critical faults earlier. Different test prioritization criteria can be selected in SmarTest, such as prioritization based on the number of commits made in the code, or based on the tests that failed in last executions. A customizable dashboard with significant system information to guide the testing is also provided by SmarTest at run-time. This work represents an interesting application of SPL-inspired testing techniques to real-world software systems, which could be applicable to other open-source SPLs.Ministerio de Economía y Competitividad BELI (TIN2015-70560-R)Junta de Andalucía P12-TIC-186

Automated generation of computationally hard feature models using evolutionary algorithms

This is the post-print version of the final paper published in Expert Systems with Applications. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2014 Elsevier B.V.A feature model is a compact representation of the products of a software product line. The automated extraction of information from feature models is a thriving topic involving numerous analysis operations, techniques and tools. Performance evaluations in this domain mainly rely on the use of random feature models. However, these only provide a rough idea of the behaviour of the tools with average problems and are not sufficient to reveal their real strengths and weaknesses. In this article, we propose to model the problem of finding computationally hard feature models as an optimization problem and we solve it using a novel evolutionary algorithm for optimized feature models (ETHOM). Given a tool and an analysis operation, ETHOM generates input models of a predefined size maximizing aspects such as the execution time or the memory consumption of the tool when performing the operation over the model. This allows users and developers to know the performance of tools in pessimistic cases providing a better idea of their real power and revealing performance bugs. Experiments using ETHOM on a number of analyses and tools have successfully identified models producing much longer executions times and higher memory consumption than those obtained with random models of identical or even larger size.European Commission (FEDER), the Spanish Government and the Andalusian Government

Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of Software Product Lines

Lopez-Herrejon, R. Erick, Ferrer J., Chicano F., Egyed A., & Alba E. (2014). Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of Software Product Lines. Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2014, Beijing, China, July 6-11, 2014. 387–396.Software Product Lines (SPLs) are families of related software products, each with its own set of feature combinations. Their commonly large number of products poses a unique set of challenges for software testing as it might not be technologically or economically feasible to test of all them individually. SPL pairwise testing aims at selecting a set of products to test such that all possible combinations of two features are covered by at least one selected product. Most approaches for SPL pairwise testing have focused on achieving full coverage of all pairwise feature combinations with the minimum number of products to test. Though useful in many contexts, this single-objective perspective does not reflect the prevailing scenario where software engineers do face trade-offs between the objectives of maximizing the coverage or minimizing the number of products to test. In contrast and to address this need, our work is the first to propose a classical multi-objective formalisation where both objectives are equally important. In this paper, we study the application to SPL pairwise testing of four classical multi-objective evolutionary algorithms. We developed three seeding strategies — techniques that leverage problem domain knowledge — and measured their performance impact on a large and diverse corpus of case studies using two well-known multi-objective quality measures. Our study identifies the performance differences among the algorithms and corroborates that the more domain knowledge leveraged the better the search results. Our findings enable software engineers to select not just one solution (as in the case of single-objective techniques) but instead to select from an array of test suite possibilities the one that best matches the economical and technological constraints of their testing context.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech. Austrian Science Fund (FWF) project P25289- N15 and Lise Meitner Fellowship M1421-N15. Spanish Ministry of Economy and Competitiveness and FEDER under contract TIN2011-28194 and fellowship BES-2012-055967. Project 8.06/5.47.4142 in collaboration with the VSB-Tech. Univ. of Ostrava and Universidad de Málaga, Andalucía Tech

A Survey of Constrained Combinatorial Testing

Combinatorial Testing (CT) is a potentially powerful testing technique, whereas its failure revealing ability might be dramatically reduced if it fails to handle constraints in an adequate and efficient manner. To ensure the wider applicability of CT in the presence of constrained problem domains, large and diverse efforts have been invested towards the techniques and applications of constrained combinatorial testing. In this paper, we provide a comprehensive survey of representations, influences, and techniques that pertain to constraints in CT, covering 129 papers published between 1987 and 2018. This survey not only categorises the various constraint handling techniques, but also reviews comparatively less well-studied, yet potentially important, constraint identification and maintenance techniques. Since real-world programs are usually constrained, this survey can be of interest to researchers and practitioners who are looking to use and study constrained combinatorial testing techniques