Enabling Testing of Large Scale Highly Configurable Systems with Search-based Software Engineering: the Case of Model-based Software Product Lines

Complex situations formed by mixes of versatile environments, various user needs and time-to-market constraints led to the development of highly configurable systems. In line with the emergence of such systems, software development is increasingly moving from the production of a single, yet configurable software to the development of families of software products. Such families of related software are called Software Product Lines (SPLs), and they allow the automation of the configuration, deployment and management of tailored software products through the combination of software features. These features and the constraints defining their legal combinations are usually encoded in a feature model (FM), which is used to represent a SPL. One main challenge with SPLs is testing them, a task which is even more difficult as the number of features proposed is important. Ideally, all the possible products that can be configured from a SPL should be tested. This, however, is unfeasible in practice since only 270 optional features allows configuring more products than the number of atoms in the universe. Considering that realistic SPLs involve thousands of features and that testing capabilities are limited by time and budget constraints, only a subset of all the configurable products can actually be tested, introducing the needs for strategies to test such SPLs. To reduce the testing effort, techniques using combinatorial interaction testing (CIT) have been proposed and proven to be successful. However, they fail at scaling to large and heavily constrained SPLs. In addition, CIT is costly to apply due to the combinatorial explosion induced by calculating the feature combinations. Besides, existing approaches do not consider multiple and potentially conflicting testing objectives such as minimizing the number of configurations and their cost. In this respect, this dissertation introduces scalable techniques for both generating and prioritizing relevant SPL product configurations for CIT by using a similarity heuristic which avoids the combinatorial explosion. In a second step, methods for handling multiple testing objectives are presented. The following part of this thesis focuses on the quality assessment of given product configurations prior to testing. The objective here is to evaluate how good is a given set of configurations according to different testing criteria, whatever the way these configurations have been selected. This situation arises when configurations that have to be tested are already available. Since testing these software products individually is a costly and time consuming task, methodologies to evaluate them prior testing are introduced, thus allowing to discard unnecessary ones and save testing sessions. In particular, an approach based on mutation of the SPL FM which can form viable and cheaper alternative to CIT is presented. The next part of this dissertation investigates the reverse-engineering of a SPL and its FM from existing source code of software product variants. Since SPLs allows us to reduce development costs and quickly derive tailored products for specific market share, automated techniques to migrate similar product variants into a whole SPL are necessary. In particular, the challenge of reverse-engineering a SPL which is concordant with the underlying software products is tackled by a fully automated approach. In addition, since reverse-engineering approaches (whether manually or automatically performed) are inherently error-prone, a methodology for evaluating and fixing reverse-engineered SPL FM is presented. The final part of this dissertation describes the application of the introduced theoretical advances to an industrial case with the CETREL company. In this project, a credit card authorization system is tested by using credit card authorizations. The testing process is optimized by modeling credit card authorizations as a SPL, enabling the application of the above-mentioned generation and evaluation approaches. All the proposed approaches use search-based techniques combined with constraint solvers and have been validated through rigorous experiments performed on moderate to large scale SPLs

Comparing White-box and Black-box Test Prioritization

Although white-box regression test prioritization has been well-studied, the more recently introduced black-box prioritization approaches have neither been compared against each other nor against more well-established white-box techniques. We present a comprehensive experimental comparison of several test prioritization techniques, including well-established white-box strategies and more recently introduced black-box approaches. We found that Combinatorial Interaction Testing and diversity-based techniques (Input Model Diversity and Input Test Set Diameter) perform best among the black-box approaches. Perhaps surprisingly, we found little difference between black-box and white-box performance (at most 4% fault detection rate difference). We also found the overlap between black- and white-box faults to be high: the first 10% of the prioritized test suites already agree on at least 60% of the faults found. These are positive findings for practicing regression testers who may not have source code available, thereby making white-box techniques inapplicable. We also found evidence that both black-box and white-box prioritization remain robust over multiple system releases

Towards a Language-Independent Approach for Reverse-Engineering of Software Product Lines

International audienceCommon industrial practices lead to the development of similar software products. These products are usually managed in an ad-hoc way which gradually results in a low product quality. To overcome this problem, it is essential to migrate these products into a Software Product Line (SPL). Towards this direction, this paper proposes a language-independent approach capable of reverse-engineering an SPL from the source code of product variants. A prototype tool and a case study show the feasibility and the practicality of the proposed approach