Test case prioritization in highly-configurable systems

Highly-configurable software systems (HCSs) provide a common core functionality and a set of optional features, where a feature represents an increment in system functionality. Development tools as Eclipse (with more than 3,000 plug-ins) or operating systems as Debian Wheezy (with more than 37,000 packages) have been reported as examples of HCSs. The large number of features that can be combined leading to thousands or even millions of individual software systems (a.k.a. configurations) makes the testing of HCSs a challenge. That is, testing every single configuration is too expensive in general. To overcome this problem, researchers have proposed numerous techniques mainly focused on reducing the space of testing to a manageable but representative subset of configurations to be tested. Nevertheless, even after reduce the test space, the number of configurations under test may still be large and expensive to run. In this dissertation, we address the testing of HCSs using test case prioritization, which schedules test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal, typically detect faults as quickly as possible. Test case prioritization approaches help to improve the effectiveness of testing allowing faster feedback to software testers and ensuring that test cases with the highest fault detection ability will have been executed if testing is stopped by any circumstance. Test case prioritization in HCSs can be driven by different functional and non–functional objectives, being an objective the order criterion used to accelerate the detection of faults. Functional prioritization objectives are based on the functional features of the system and their interactions. Non–functional prioritization objectives consider extra–functional information such as user preferences or cost. In this thesis we present thirteen objectives, techniques and tools for test case prioritization in HCSs. In particular, we define six objectives based on functional properties and seven objectives based on non–functional properties of the HCS under test. These objectives are evaluated using real data extracted from HCS Git repositories and bug tracking systems, which reinforces the validity of our conclusions. Regarding test case prioritization techniques, we present single–objective and multi-objective approaches based on evolutionary algorithms. Additionally, we report a comparison of 63 different com binations of up to three objectives to determine which combinations perform better in accelerating the detection of faults in an HCS. Furthermore, we propose an industrystrength HCS case study with more than 2 billions of configurations to be used as a realistic subject for further and reproducible validation of variability testing techniques. These contributions have been evaluated using extensive and rigorous experiments that reveal the efficacy and efficiency of our approach. Part of our contributions have been integrated into a tool called SmarTest for testing Drupal, a well-known web content management framework. SmarTest is a testing module that supports the analysis of the Drupal system to provide useful information to guide the testing. Also, it allows applying different prioritization testing techniques to reveals bugs faster in Drupal.Los sistemas software altamente configurables proporcionan una funcionalidad básica común y un conjunto de características opcionales para adaptar todas las variantes de un sistema de acuerdo a un conjunto determinado de requisitos. Sistemas operativos como Linux (con más de 37,000 paquetes) o herramientas de desarrollo como Eclipse (con más de 3,000 plugins) han sido presentados como ejemplos de sistemas altamente configurables. Las pruebas en sistemas software altamente configurables suponen un gran reto debido al elevado número de configuraciones que deben probarse. Por ejemplo, linux con más de 37,000 paquetes puede dar lugar a miles de millones de configuraciones diferentes. Esto hace que probar cada configuración individual de un sistema altamente configurable sea demasiado costoso. Para paliar este problema, se han propuesto numerosos trabajos de investigación para reducir el espacio de pruebas a un subconjunto razonable y representativo de configuraciones a probar. Sin embargo, incluso reduciendo el espacio de pruebas, el número de configurationes resultante puede seguir siendo demasiado grande y costoso para llevar a cabo las pruebas. En esta tesis, se aborda el problema anteriormente mencionado utilizando priorización de casos de prueba para sistemas altamente configurables. Las técnicas de priorización de casos de prueba ordenan las pruebas de manera que se ejecuten primero aquellas que permitan maximizar un determinado objetivo de rendimiento, normalmente, detectar errores lo antes posible. La priorización de casos de prueba ayuda a mejorar la efectividad del proceso de pruebas permitiendo un feedback más rápido a los ingenieros del software y asegurando que las pruebas con mayor capacidad de detectar errores habrán sido ejecutadas si el proceso se parase por alguna circunstancia. La priorización de pruebas en sistemas software altamente configurables puede guiarse por diferentes objetivos funcionales y no funcionales, entendiéndose un objetivo como el criterio de ordenación elegido para acelerar la detectión de errores. Los objetivos de priorización funcionales son aquellos basados en características funcionales del sistema y sus interacciones. Los objetivos no funcionales consideran información extra-funcional tales como las preferencias de usuario. En esta tesis se presentan un conjunto de objetivos, técnicas y herramientas para la priorización de casos de prueba en sistemas altamente configurables. En concreto, se definen 6 objetivos basados en propiedades funcionales y 7 objetivos basados en propiedades no funcionales del sistema para guiar el proceso de pruebas. Estos objetivos se evalúan utilizando datos reales extraídos de los repositiorios Git y sistemas de gestión de errores de sistemas altamente configurables, reforzándo así la validez de nuestras conclusiones. Con respecto a las técnicas de priorización de pruebas, se presentan propuestas basadas en un sólo objetivo y basadas en múltiples objetivos utilizando algoritmos evolutivos. Adicionalmente, se presenta una comparación de 63 combinaciones diferentes de 1, 2 y 3 objetivos con la intención de encontrar las mejores combinaciones para accelerar la detección de errores en un sistema altamente configurable. Además, proponemos un caso de estudio basado en un sistema real altamente configurable con más de 2,000 millones de configuraciones para evaluar y validar las técnicas de pruebas. Estas contribuciones han sido evaluadas realizando rigurosos experimentos que han revalado la eficacia y efficiancia de nuestra propuesta. Parte de nuestras contribuciones se han integrado en una herramienta denominada SmarTest para llevar acabo las pruebas en Drupal. Drupal es un sistema de gestión de contenidos web y framework ampliamente conocido. SmarTest es un módulo de pruebas para Drupal que permite analizar el sistema y proporcionar al ingeniero de calidad información relevante de Drupal que le permita guiar mejor el proceso de pruebas. Además, SmarTest también permite aplicar diferentes técnicas de priorización para detectar antes los errores en Drupal

SmarTest: A Test Case Prioritization Tool for Drupal

Test case prioritization techniques aim to identify the optimal ordering of tests to accelerate the detection of faults. The importance of these techniques has been recognized in the context of Software Product Lines (SPLs), where the potentially huge number of products makes testing extremely challenging. We found that the open source Drupal framework shares most of the principles and challenges of SPL development and it can be considered a real-world example of family of products. In a previous work, we represented the Drupal configuration space as a feature model and we collected extra functional information about its features from open repositories. Part of this data proved to be a good indicator of faults propensity in Drupal features. Thus, they become valuable assets to prioritize tests in individual Drupal products. In this paper, we present SmarTest, a test prioritization tool for accelerating the detection of faults in Drupal. SmarTest has been developed as an extension of the Drupal core testing system. SmarTest supports the prioritization of tests providing faster feedback and letting testers begin correcting critical faults earlier. Different test prioritization criteria can be selected in SmarTest, such as prioritization based on the number of commits made in the code, or based on the tests that failed in last executions. A customizable dashboard with significant system information to guide the testing is also provided by SmarTest at run-time. This work represents an interesting application of SPL-inspired testing techniques to real-world software systems, which could be applicable to other open-source SPLs.Ministerio de Economía y Competitividad BELI (TIN2015-70560-R)Junta de Andalucía P12-TIC-186

Automated testing on the analysis of variability-intensive artifacts: An exploratory study with SAT Solvers

The automated detection of faults on variability analysis tools is a challenging task often infeasible due to the combinatorial com plexity of the analyses. In previous works, we successfully automated the generation of test data for feature model analysis tools using metamor phic testing. The positive results obtained have encouraged us to explore the applicability of this technique for the efficient detection of faults in other variability-intensive domains. In this paper, we present an auto mated test data generator for SAT solvers that enables the generation of random propositional formulas (inputs) and their solutions (expected output). In order to show the feasibility of our approach, we introduced 100 artificial faults (i.e. mutants) in an open source SAT solver and com pared the ability of our generator and three related benchmarks to detect them. Our results are promising and encourage us to generalize the tech nique, which could be potentially applicable to any tool dealing with variability such as Eclipse repositories or Maven dependencies analyzers.Comisión Interministerial de Ciencia y Tecnología (CICYT) SETI (TIN2009-07366)Junta de Andalucía P07-TIC-2533 (Isabel)Junta de Andalucía P10-TIC-5906 (THEOS

The Drupal Framework: a Case Study to Evaluate Variability Testing Techniques

Variability testing techniques search for effective but manageable test suites that lead to the rapid detection of faults in systems with high variability. Evaluating the effectiveness of these techniques in real settings is a must but challenging due to the lack of variability-intensive systems with available code, automated tests and fault reports. in this paper, we propose using the Drupal framework as a case study to evaluate variability testing techniques. First, we represent the framework variability as a feature model. Then, we report on extensive data extracted from the Drupal git repository and the Drupal issue tracking system. Among other results, we identified 378 faults in single features and 11 faults triggered by the interaction between two of the features of Drupal v7.23, reported during a one-year period. These data may give a new insight into the distribution of faults in variability-intensive systems and the fault propensity of features. To show the feasibility of our work, we used the case study to evaluate the effectiveness of a historybased test case prioritization criterion. Results suggest that this technique could contribute to accelerate the detection of faults of test suites based on combinatorial testing.CICYT TIN2009-07366CICYT TIN2012-32273Junta de Andalucía TIC-5906Junta de Andalucía TIC-186

A Comparison of Test Case Prioritization Criteria for Software Product Lines

Software Product Line (SPL) testing is challenging due to the potentially huge number of derivable products. To alleviate this problem, numerous contributions have been proposed to reduce the number of products to be tested while still having a good coverage. However, not much attention has been paid to the order in which the products are tested. Test case prioritization techniques reorder test cases to meet a certain performance goal. For instance, testers may wish to order their test cases in order to detect faults as soon as possible, which would translate in faster feedback and earlier fault correction. in this paper, we explore the applicability of test case prioritization techniques to SPL testing. We propose five different prioritization criteria based on common metrics of feature models and we compare their effectiveness in increasing the rate of early fault detection, i.e. a measure of how quickly faults are detected. The results show that different orderings of the same SPL suite may lead to significant differences in the rate of early fault detection. They also show that our approach may contribute to accelerate the detection of faults of SPL test suites based on combinatorial testingMinisterio de Ciencia e Innovación TIN2009-07366 (SETI)Ministerio de Economía y Competitividad TIN2012-32273Junta de Andalucía P10-TIC-590

Automated Variability Analysis and Testing of an E-Commerce Site. An Experience Report

In this paper, we report on our experience on the development of La Hilandera, an e-commerce site selling haberdashery products and craft supplies in Europe. The store has a huge input space where customers can place almost three millions of different orders which made testing an ex-tremely di cult task. To address the challenge, we explored the applicability of some of the practices for variability management in software product lines. First, we used a feature model to represent the store input space which provided us with a variability view easy to understand, share and discuss with all the stakeholders. Second, we used techniques for the automated analysis of feature models for the detection and repair of inconsistent and missing con guration settings. Finally, we used test selection and prioritization techniques for the generation of a manageable and effective set of test cases. Our ndings, summarized in a set of lessons learnt, suggest that variability techniques could successfully address many of the challenges found when developing e-commerce sites.CICYT TIN2012-32273Junta de Andalucía TIC-5906Junta de Andalucía P12-TIC- 186

Performance mutation testing

Performance bugs are known to be a major threat to the success of software products. Performance tests aim to detect performance bugs by executing the program through test cases and checking whether it exhibits a noticeable performance degradation. The principles of mutation testing, a well-established testing technique for the assessment of test suites through the injection of artificial faults, could be exploited to evaluate and improve the detection power of performance tests. However, the application of mutation testing to assess performance tests, henceforth called performance mutation testing (PMT), is a novel research topic with numerous open challenges. In previous papers, we identified some key challenges related to PMT. In this work, we go a step further and explore the feasibility of applying PMT at the source-code level in general purpose languages. To do so, we revisit concepts associated with classical mutation testing, and design seven novel mutation operators to model known bug-inducing patterns. As a proof of concept, we applied traditional mutation operators as well as performance mutation operators to open-source C++ programs. The results reveal the potential of the new performance-mutants to help assess and enhance performance tests when compared to traditional mutants. A review of live mutants in these programs suggests that they can induce the design of special test inputs. In addition to these promising results, our work brings a whole new set of challenges related to PMT, which will hopefully serve as a starting point for new contributions in the areaMinisterio de Economía y Competitividad TIN2015-65845-C3-3-RMinisterio de Economía y Competitividad RTI2018- 093608-B-C33Ministerio de Economía y Competitividad BELI (TIN2015-70560-R)Ministerio de Economía y Competitividad (HORATIO) RTI2018-101204-B-C2

Search-based mutation testing to improve performance tests

Performance bugs are common and can cause a significant deterio ration in the behaviour of a program, leading to costly issues. To detect them and reduce their impact, performance tests are typi cally applied. However, there is a lack of mechanisms to evaluate the quality of performance tests, causing many of these bugs re main unrevealed. Mutation testing, a fault-based technique to assess and improve test suites, has been successfully studied with func tional tests. In this paper, we propose the use of mutation testing together with a search-based strategy (evolutionary algorithm) to find mutants that simulate performance issues. This novel approach contributes to enhance the confidence on performance tests while reducing the cost of mutation testing.Ministerio de Economía y Competitividad BELI (TIN2015-70560-R)Ministerio de Economía y Competitividad TIN2015-65845-C3-3-

Mutation testing in the wild: findings from GitHub

Mutation testing exploits artificial faults to measure the adequacy of test suites and guide their improvement. It has become an extremely popular testing technique as evidenced by the vast literature, numerous tools, and research events on the topic. Previous survey papers have successfully compiled the state of research, its evolution, problems, and challenges. However, the use of mutation testing in practice is still largely unexplored. In this paper, we report the results of a thorough study on the use of mutation testing in GitHub projects. Specifically, we first performed a search for mutation testing tools, 127 in total, and we automatically searched the GitHub repositories including evidence of their use. Then, we focused on the top ten most widely used tools, based on the previous results, and manually revised and classified over 3.5K GitHub active repositories importing them. Among other findings, we observed a recent upturn in interest and activity, with Infection (PHP), PIT (Java) and Humbug (PHP) being the most widely used mutation tools in recent years. The predominant use of mutation testing is development, followed by teaching and learning, and research projects, although with significant differences among mutation tools found in the literature—less adopted and largely used in teaching and research—and those found in GitHub only—more popular and more widely used in development. Our work provides a new and encouraging perspective on the state of practice of mutation testing.Junta de Andalucía US-1264651 (APOLO)Junta de Andalucía P18-FR-2895 (EKIPMENT-PLUS)Ministerio de Ciencia, Innovación y Universidades RTI2018-101204-B-C21 (HORATIO)Ministerio de Ciencia, Innovación y Universidades RTI2018-093608-BC33 (FAME

Variability testing in the wild: the Drupal case study

Variability testing techniques search for effective and manageable test suites that lead to the rapid detection of faults in systems with high variability. Evaluating the effectiveness of these techniques in realistic settings is a must, but challenging due to the lack of variability intensive systems with available code, automated tests and fault reports. In this article, we propose using the Drupal framework as a case study to evaluate variability testing techniques. First, we represent the framework variability using a feature model. Then, we report on extensive non–functional data extracted from the Drupal Git repository and the Drupal issue tracking system. Among other results, we identified 3,392 faults in single features and 160 faults triggered by the interaction of up to 4 features in Drupal v7.23. We also found positive correlations relating the number of bugs in Drupal features to their size, cyclomatic complexity, number of changes and fault history. To show the feasibility of our work, we evaluated the effectiveness of non–functional data for test case prioritization in Drupal. Results show that non–functional attributes are effective at accelerating the detection of faults, outperforming related prioritization criteria as test case similarity.Ministerio de Economía y Competitividad IPT-2012-0890-390000Ministerio de Economía y Competitividad TIN2012-3227