Spectrum-based feature localization for families of systems

In large code bases, locating the elements that implement concrete features of a system is challenging. This information is paramount for maintenance and evolution tasks, although not always explicitly available. In this work, motivated by the needs of locating features as a first step for feature-based Software Product Line adoption, we propose a solution for improving the performance of existing approaches. For this, relying on an automatic feature localization approach to locate features in single-systems, we propose approaches to deal with feature localization in the context of families of systems, e.g., variants created through opportunistic reuse such as clone-and-own. Our feature localization approaches are built on top of Spectrum-based feature localization (SBFL) techniques, supporting both dynamic feature localization (i.e., using execution traces as input) and static feature localization (i.e., relying on the structural decomposition of the variants’ implementation). Concretely, we provide (i) a characterization of different settings for dynamic SBFL in single systems, (ii) an approach to improve accuracy of dynamic SBFL for families of systems, and (iii) an approach to use SBFL as a static feature localization technique for families of systems. The proposed approaches are evaluated using the consolidated ArgoUML SPL feature localization benchmark. The results suggest that some settings of SBFL favor precision such as using the ranking metrics Wong2, Ochiai2, or Tarantula with high threshold values, while most of the ranking metrics with low thresholds favor recall. The approach to use information from variants increase the precision of dynamic SBFL while maintaining recall even with few number of variants, namely two or three. Finally, the static SBFL approach performs equally in terms of accuracy to other state-of-the-art approaches, such as Formal Concept Analysis and Interdependent Elements

Uncertainty-aware Robustness Assessment of Industrial Elevator Systems

Industrial elevator systems are commonly used software systems in our daily lives, which operate in uncertain environments such as unpredictable passenger traffic, uncertain passenger attributes and behaviors, and hardware delays. Understanding and assessing the robustness of such systems under various uncertainties enable system designers to reason about uncertainties, especially those leading to low system robustness, and consequently improve their designs and implementations in terms of handling uncertainties. To this end, we present a comprehensive empirical study conducted with industrial elevator systems provided by our industrial partner Orona, which focuses on assessing the robustness of a dispatcher, i.e., a software component responsible for elevators’ optimal scheduling. In total, we studied 90 industrial dispatchers in our empirical study. Based on the experience gained from the study, we derived an uncertainty-aware robustness assessment method (named UncerRobua) comprising a set of guidelines on how to conduct the robustness assessment and a newly proposed ranking algorithm, for supporting the robustness assessment of industrial elevator systems against uncertainties

Genetic Algorithm-based Testing of Industrial Elevators under Passenger Uncertainty

Elevators, as other cyber-physical systems, need to deal with uncertainty during their operation due to several factors such as passengers and hardware. Such uncertainties could affect the quality of service promised by elevators and in the worst case lead to safety hazards. Thus, it is important that elevators are extensively tested by considering uncertainty during their development to ensure their safety in operation. To this end, we present an uncertainty testing methodology supported with a tool to test industrial dispatching systems at the Software-in-the-Loop (SiL) test level. In particular, we focus on uncertainties in passenger data and employ a Genetic Algorithm (GA) with specifically designed genetic operators to significantly reduce the quality of service of elevators, thus aiming to find uncertain situations that are difficult to extract by users. An initial experiment with an industrial dispatcher revealed that the GA significantly decreased the quality of service as compared to not considering uncertainties. The results can be used to further improve the implementation of dispatching algorithms to handle various uncertainties

Model-Based Testing in Practice: An Industrial Case Study using GraphWalker

Model-based testing (MBT) is a test design technique that supports the automation of software testing processes and generates test artefacts based on a system model representing behavioural aspects of the system under test (SUT). Previous research has shown some positive aspects of MBT such as low-cost test case generation and fault detection effectiveness. However, it is still a challenge for both practitioners and researchers to evaluate MBT tools and techniques in real, industrial settings. Consequently, the empirical evidence regarding the mainstream use, including the modelling and test case generation using MBT tools, is limited. In this paper, we report the results of a case study on applying GraphWalker, an open-source tool for MBT, on an industrial cyber-physical system (i.e., a Train Control Management System developed by Bombardier Transportation in Sweden), from modelling of real-world requirements and test specifications to test case generation. We evaluate the models of the SUT for completeness and representativeness, compare MBT with manual test cases written by practitioners using multiple attributes as well as share our experiences of selecting and using GraphWalker for industrial application. The results show that a model of the SUT created using both requirements and test specifications provides better understanding of the SUT from testers’ perspective, making it more complete and representative than the model created based only on the requirements specification alone. The generated model-based test cases are longer in terms of the number of test steps, achieve better edge coverage and can cover requirements more frequently in different orders while achieving the same level of requirements coverage as manually created test cases

Evolutionary generation of metamorphic relations for cyber-physical systems

A problem when testing Cyber-Physical Systems (CPS) is the difficulty of determining whether a particular system output or behaviour is correct or not. Metamorphic testing alleviates such a problem by reasoning on the relations expected to hold among multiple executions of the system under test, which are known as Metamorphic Relations (MRs). However, the development of effective MRs is often challenging and requires the involvement of domain experts. This paper summarizes our recent publication: "Generating Metamorphic Relations for Cyber-Physical Systems with Genetic Programming: An Industrial Case Study", presented at ESEC/FSE 2021. In that publication we presented GAssertMRs, the first technique to automatically generate MRs for CPS, leveraging GP to explore the space of candidate solutions. We evaluated GAssertMRs in an industrial case study, outperforming other baselines

Automated Misconfiguration Repair of Configurable Cyber-Physical Systems with Search: an Industrial Case Study on Elevator Dispatching Algorithms

Real-world Cyber-Physical Systems (CPSs) are usually configurable. Through parameters, it is possible to configure, select or unselect different system functionalities. While this provides high flexibility, it also becomes a source for failures due to misconfigurations. The large number of parameters these systems have and the long test execution time in this context due to the use of simulation-based testing make the manual repair process a cumbersome activity. Subsequently, in this context, automated repairing methods are paramount. In this paper, we propose an approach to automatically repair CPSs' misconfigurations. Our approach is evaluated with an industrial CPS case study from the elevation domain. Experiments with a real building and data obtained from operation suggests that our approach outperforms a baseline algorithm as well as the state of the practice (i.e., manual repair carried out by domain experts).Comment: To be published in the 45th International Conference on Software Engineering, SEIP trac

Microservices for Continuous Deployment, Monitoring and Validation in Cyber-Physical Systems: an Industrial Case Study for Elevators Systems

Cyber-Physical Systems (CPSs) are systems that integrate digital cyber computations with physical processes. The software embedded in CPSs has a long life-cycle, requiring constant evolution to support new requirements, bug fixes, and deal with hardware obsolescence. To date, the development of software for CPSs is fragmented, which makes it extremely expensive. This could be substantially enhanced by tightly connecting the development and operation phases, as is done in other software engineering domains (e.g., web engineering through DevOps). Nevertheless, there are still complex issues that make it difficult to use DevOps techniques in the CPS domain, such as those related to hardware-software co-design. To pave the way towards DevOps in the CPS domain, in this paper we instantiate part of the reference architecture presented in the H2020 Adeptness project, which is based on microservices that allow for the continuous deployment, monitoring and validation of CPSs. To this end, we elaborate a systematic methodology that considers as input both domain expertise and a previously defined taxonomy for DevOps in the CPS domain. We obtain a generic microservice template that can be used in any kind of CPS. In addition, we instantiate this architecture in the context of an industrial case study from the elevation domain

Applying and Extending the Delta Debugging Algorithm for Elevator Dispatching Algorithms (Experience Paper)

Elevator systems are one kind of Cyber-Physical Systems (CPSs), and as such, test cases are usually complex and long in time. This is mainly because realistic test scenarios are employed (e.g., for testing elevator dispatching algorithms, typically a full day of passengers traveling through a system of elevators is used). However, in such a context, when needing to reproduce a failure, it is of high benefit to provide the minimal test input to the software developers. This way, analyzing and trying to localize the root-cause of the failure is easier and more agile. Delta debugging has been found to be an efficient technique to reduce failure-inducing test inputs. In this paper, we enhance this technique by first monitoring the environment at which the CPS operates as well as its physical states. With the monitored information, we search for stable states of the CPS during the execution of the simulation. In a second step, we use such identified stable states to help the delta debugging algorithm isolate the failure-inducing test inputs more efficiently. We report our experience of applying our approach into an industrial elevator dispatching algorithm. An empirical evaluation carried out with real operational data from a real installation of elevators suggests that the proposed environment-wise delta debugging algorithm is between 1.3 to 1.8 times faster than the traditional delta debugging, while producing a larger reduction in the failure-inducing test inputs. The results provided by the different implemented delta debugging algorithm versions are qualitatively assessed with domain experts. This assessment provides new insights and lessons learned, such as, potential applications of the delta debugging algorithm beyond debugging

Generating metamorphic relations for cyber-physical systems with genetic programming: an industrial case study

One of the major challenges in the verification of complex industrial Cyber-Physical Systems is the difficulty of determining whether a particular system output or behaviour is correct or not, the socalled test oracle problem. Metamorphic testing alleviates the oracle problem by reasoning on the relations that are expected to hold among multiple executions of the system under test, which are known as Metamorphic Relations (MRs). However, the development of effective MRs is often challenging and requires the involvement of domain experts. In this paper, we present a case study aiming at automating this process. To this end,we implemented GAssertMRs, a tool to automatically generate MRs with genetic programming. We assess the cost-effectiveness of this tool in the context of an industrial case study from the elevation domain. Our experimental results show that in most cases GAssertMRs outperforms the other baselines, including manually generated MRs developed with the help of domain experts. We then describe the lessons learned from our experiments and we outline the future work for the adoption of this technique by industrial practitioners