Automated Variability Analysis and Testing of an E-Commerce Site. An Experience Report

In this paper, we report on our experience on the development of La Hilandera, an e-commerce site selling haberdashery products and craft supplies in Europe. The store has a huge input space where customers can place almost three millions of different orders which made testing an ex-tremely di cult task. To address the challenge, we explored the applicability of some of the practices for variability management in software product lines. First, we used a feature model to represent the store input space which provided us with a variability view easy to understand, share and discuss with all the stakeholders. Second, we used techniques for the automated analysis of feature models for the detection and repair of inconsistent and missing con guration settings. Finally, we used test selection and prioritization techniques for the generation of a manageable and effective set of test cases. Our ndings, summarized in a set of lessons learnt, suggest that variability techniques could successfully address many of the challenges found when developing e-commerce sites.CICYT TIN2012-32273Junta de Andalucía TIC-5906Junta de Andalucía P12-TIC- 186

Statistical prioritization for software product line testing: an experience report

Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion™, an embedded system providing helicopter landing assistance.</p

Modellbasiertes Regressionstesten von Varianten und Variantenversionen

The quality assurance of software product lines (SPL) achieved via testing is a crucial and challenging activity of SPL engineering. In general, the application of single-software testing techniques for SPL testing is not practical as it leads to the individual testing of a potentially vast number of variants. Testing each variant in isolation further results in redundant testing processes by means of redundant test-case executions due to the shared commonality. Existing techniques for SPL testing cope with those challenges, e.g., by identifying samples of variants to be tested. However, each variant is still tested separately without taking the explicit knowledge about the shared commonality and variability into account to reduce the overall testing effort. Furthermore, due to the increasing longevity of software systems, their development has to face software evolution. Hence, quality assurance has also to be ensured after SPL evolution by testing respective versions of variants. In this thesis, we tackle the challenges of testing redundancy as well as evolution by proposing a framework for model-based regression testing of evolving SPLs. The framework facilitates efficient incremental testing of variants and versions of variants by exploiting the commonality and reuse potential of test artifacts and test results. Our contribution is divided into three parts. First, we propose a test-modeling formalism capturing the variability and version information of evolving SPLs in an integrated fashion. The formalism builds the basis for automatic derivation of reusable test cases and for the application of change impact analysis to guide retest test selection. Second, we introduce two techniques for incremental change impact analysis to identify (1) changing execution dependencies to be retested between subsequently tested variants and versions of variants, and (2) the impact of an evolution step to the variant set in terms of modified, new and unchanged versions of variants. Third, we define a coverage-driven retest test selection based on a new retest coverage criterion that incorporates the results of the change impact analysis. The retest test selection facilitates the reduction of redundantly executed test cases during incremental testing of variants and versions of variants. The framework is prototypically implemented and evaluated by means of three evolving SPLs showing that it achieves a reduction of the overall effort for testing evolving SPLs.Testen ist ein wichtiger Bestandteil der Entwicklung von Softwareproduktlinien (SPL). Aufgrund der potentiell sehr großen Anzahl an Varianten einer SPL ist deren individueller Test im Allgemeinen nicht praktikabel und resultiert zudem in redundanten Testfallausführungen, die durch die Gemeinsamkeiten zwischen Varianten entstehen. Existierende SPL-Testansätze adressieren diese Herausforderungen z.B. durch die Reduktion der Anzahl an zu testenden Varianten. Jedoch wird weiterhin jede Variante unabhängig getestet, ohne dabei das Wissen über Gemeinsamkeiten und Variabilität auszunutzen, um den Testaufwand zu reduzieren. Des Weiteren muss sich die SPL-Entwicklung mit der Evolution von Software auseinandersetzen. Dies birgt weitere Herausforderungen für das SPL-Testen, da nicht nur für Varianten sondern auch für ihre Versionen die Qualität sichergestellt werden muss. In dieser Arbeit stellen wir ein Framework für das modellbasierte Regressionstesten von evolvierenden SPL vor, das die Herausforderungen des redundanten Testens und der Software-Evolution adressiert. Das Framework vereint Testmodellierung, Änderungsauswirkungsanalyse und automatische Testfallselektion, um einen inkrementellen Testprozess zu definieren, der Varianten und Variantenversionen unter Ausnutzung des Wissens über gemeinsame Funktionalität und dem Wiederverwendungspotential von Testartefakten und -resultaten effizient testet. Für die Testmodellierung entwickeln wir einen Ansatz, der Variabilitäts- sowie Versionsinformation von evolvierenden SPL gleichermaßen für die Modellierung einbezieht. Für die Änderungsauswirkungsanalyse definieren wir zwei Techniken, um zum einen Änderungen in Ausführungsabhängigkeiten zwischen zu testenden Varianten und ihren Versionen zu identifizieren und zum anderen die Auswirkungen eines Evolutionsschrittes auf die Variantenmenge zu bestimmen und zu klassifizieren. Für die Testfallselektion schlagen wir ein Abdeckungskriterium vor, das die Resultate der Auswirkungsanalyse einbezieht, um automatisierte Entscheidungen über einen Wiederholungstest von wiederverwendbaren Testfällen durchzuführen. Die abdeckungsgetriebene Testfallselektion ermöglicht somit die Reduktion der redundanten Testfallausführungen während des inkrementellen Testens von Varianten und Variantenversionen. Das Framework ist prototypisch implementiert und anhand von drei evolvierenden SPL evaluiert. Die Resultate zeigen, dass eine Aufwandsreduktion für das Testen evolvierender SPL erreicht wird

Cloud adoption: a goal-oriented requirements engineering approach

The enormous potential of cloud computing for improved and cost-effective service has generated unprecedented interest in its adoption. However, a potential cloud user faces numerous risks regarding service requirements, cost implications of failure and uncertainty about cloud providers’ ability to meet service level agreements. These risks hinder the adoption of cloud computing. We motivate the need for a new requirements engineering methodology for systematically helping businesses and users to adopt cloud services and for mitigating risks in such transition. The methodology is grounded in goal-oriented approaches for requirements engineering. We argue that Goal-Oriented Requirements Engineering (GORE) is a promising paradigm to adopt for goals that are generic and flexible statements of users’ requirements, which could be refined, elaborated, negotiated, mitigated for risks and analysed for economics considerations. The methodology can be used by small to large scale organisations to inform crucial decisions related to cloud adoption. We propose a risk management framework based on the principle of GORE. In this approach, we liken risks to obstacles encountered while realising cloud user goals, therefore proposing cloud-specific obstacle resolution tactics for mitigating identified risks. The proposed framework shows benefits by providing a principled engineering approach to cloud adoption and empowering stakeholders with tactics for resolving risks when adopting the cloud. We extend the work on GORE and obstacles for informing the adoption process. We argue that obstacles’ prioritisation and their resolution is core to mitigating risks in the adoption process. We propose a novel systematic method for prioritising obstacles and their resolution tactics using Analytical Hierarchy Process (AHP). To assess the AHP choice of the resolution tactics we support the method by stability and sensitivity analysis

Auditing for ISO 9001 requirements in the context of agile software processes

ISO 9001 demands of (software) organizations that a rigorous demonstration of their software processes be implemented and a set of guidelines followed at various levels of abstraction. What these organizations need to show, in other words, is that their software processes have been designed and implemented in a way that allows for a level of configuration and operation that complies with ISO 9001 requirements. For software organizations needing ISO 9001 certification, it is important that they establish a software process life cycle that can manage the requirements imposed by this certification standard. However, software organizations that develop their software products using the agile software processes, such as Extreme Programming (agile-XP), face a number of challenges in their effort to demonstrate that their process activities conform to ISO 9001 requirements, major ones being: product construction, traceability, and measurement. Agile software organizations must provide evidence of ISO 9001 conformity, and they need to develop their own procedures, tools, and methodologies to do so. As yet, there is no consensus on how to audit the agile software organization to ensure that their software processes have been designed and implemented in conformity with ISO 9001 requirements. Moreover, it is challenging to ensure that such lightweight documentation methodologies meet these requirements for certification purposes. The motivation of this research is to help software organizations that use agile software processes in their effort to meet the ISO 9001 certification requirements. This research project is also aimed at helping IS auditors extract auditing evidence that demonstrates conformity to the ISO 9001 requirements that must be met by agile software organizations. Extreme programming (agile-XP) has been selected for improvement as a candidate agile process. This selection was based on the literature indicating a higher adoption of agile-XP over other agile software processes. The goal of this research project is to improve the ability of the agile-XP process to meet the auditing requirements of ISO 9001. The goal of the research also focuses on helping agile software organizations in their effort to become ISO 9001 certified. The main objective of this research project is to design an auditing model that covers the measurement and traceability requirements of ISO 9001. The auditing model should provide IS auditors with auditing evidence that the software projects developed with the agile-XP process have fulfilled the requirements of ISO 9001. The objective also proposes several sub processes to enhance the early planning activities of agile-XP according to ISO 9001 requirements. To achieve these objectives, the main phases of the research methodology are: Investigation of the capability of agile-XP to achieve the requirements of ISO 9001 software process certification; modification of the early phases of agile-XP (i.e. release planning phase) using CMMI-DEV; and design of an auditing model for ISO 9001 traceability and measurement requirements. The main outcome of this research study, which is an auditing model that is aligned with the principles of agile-XP and focuses on ISO 9001 traceability and measurement requirements to provide the IS auditors with a methodological approach for the auditing process. The auditing model has been assessed based on case studies selected from the literature

Quality Assessment and Prediction in Software Product Lines

At the heart of product line development is the assumption that through structured reuse later products will be of a higher quality and require less time and effort to develop and test. This thesis presents empirical results from two case studies aimed at assessing the quality aspect of this claim and exploring fault prediction in the context of software product lines. The first case study examines pre-release faults and change proneness of four products in PolyFlow, a medium-sized, industrial software product line; the second case study analyzes post-release faults using pre-release data over seven releases of four products in Eclipse, a very large, open source software product line.;The goals of our research are (1) to determine the association between various software metrics, as well as their correlation with the number of faults at the component/package level; (2) to characterize the fault and change proneness of components/packages at various levels of reuse; (3) to explore the benefits of the structured reuse found in software product lines; and (4) to evaluate the effectiveness of predictive models, built on a variety of products in a software product line, to make accurate predictions of pre-release software faults (in the case of PolyFlow) and post-release software faults (in the case of Eclipse).;The research results of both studies confirm, in a software product line setting, the findings of others that faults (both pre- and post-release) are more highly correlated to change metrics than to static code metrics, and are mostly contained in a small set of components/ packages. The longitudinal aspect of our research indicates that new products do benefit from the development and testing of previous products. The results also indicate that pre-existing components/packages, including the common components/packages, undergo continuous change, but tend to sustain low fault densities. However, this is not always true for newly developed components/packages. Finally, the results also show that predictions of pre-release faults in the case of PolyFlow and post-release faults in the case of Eclipse can be done accurately from pre-release data, and furthermore, that these predictions benefit from information about additional products in the software product lines

Genomics-driven breeding for local adaptation of durum wheat is enhanced by farmers’ traditional knowledge

In the smallholder, low-input farming systems widespread in sub-Saharan Africa, farmers select and propagate crop varieties based on their traditional knowledge and experience. A data-driven integration of their knowledge into breeding pipelines may support the sustainable intensification of local farming. Here, we combine genomics with participatory research to tap into traditional knowledge in smallholder farming systems, using durum wheat (Triticum durum Desf.) in Ethiopia as a case study. We developed and genotyped a large multiparental population, called the Ethiopian NAM (EtNAM), that recombines an elite international breeding line with Ethiopian traditional varieties maintained by local farmers. A total of 1,200 EtNAM lines were evaluated for agronomic performance and farmers’ appreciation in three locations in Ethiopia, finding that women and men farmers could skillfully identify the worth of wheat genotypes and their potential for local adaptation. We then trained a genomic selection (GS) model using farmer appreciation scores and found that its prediction accuracy over grain yield (GY) was higher than that of a benchmark GS model trained on GY. Finally, we used forward genetics approaches to identify marker–trait associations for agronomic traits and farmer appreciation scores. We produced genetic maps for individual EtNAM families and used them to support the characterization of genomic loci of breeding relevance with pleiotropic effects on phenology, yield, and farmer preference. Our data show that farmers’ traditional knowledge can be integrated in genomics-driven breeding to support the selection of best allelic combinations for local adaptation

Combining SOA and BPM Technologies for Cross-System Process Automation

This paper summarizes the results of an industry case study that introduced a cross-system business process automation solution based on a combination of SOA and BPM standard technologies (i.e., BPMN, BPEL, WSDL). Besides discussing major weaknesses of the existing, custom-built, solution and comparing them against experiences with the developed prototype, the paper presents a course of action for transforming the current solution into the proposed solution. This includes a general approach, consisting of four distinct steps, as well as specific action items that are to be performed for every step. The discussion also covers language and tool support and challenges arising from the transformation

Exploiting Similarity Patterns to Build Generic Test Case Templates for Software Product Line Testing

Ph.DDOCTOR OF PHILOSOPH