15 research outputs found
Automated Test Suite Generation for Time-Continuous Simulink Models
All engineering disciplines are founded and rely on models, al- though they may differ on purposes and usages of modeling. Inter- disciplinary domains such as Cyber Physical Systems (CPSs) seek approaches that incorporate different modeling needs and usages. Specifically, the Simulink modeling platform greatly appeals to CPS engineers due to its seamless support for simulation and code generation. In this paper, we propose a test generation approach that is applicable to Simulink models built for both purposes of simulation and code generation. We define test inputs and outputs as signals that capture evolution of values over time. Our test gener- ation approach is implemented as a meta-heuristic search algorithm and is guided to produce test outputs with diverse shapes according to our proposed notion of diversity. Our evaluation, performed on industrial and public domain models, demonstrates that: (1) In con- trast to the existing tools for testing Simulink models that are only applicable to a subset of code generation models, our approach is applicable to both code generation and simulation Simulink mod- els. (2) Our new notion of diversity for output signals outperforms random baseline testing and an existing notion of signal diversity in revealing faults in Simulink models. (3) The fault revealing ability of our test generation approach outperforms that of the Simulink Design Verifier, the only testing toolbox for Simulink
Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions
Search-based software testing (SBST) typically relies on fitness functions to
guide the search exploration toward software failures. There are two main
techniques to define fitness functions: (a) automated fitness function
computation from the specification of the system requirements and (b) manual
fitness function design. Both techniques have advantages. The former uses
information from the system requirements to guide the search toward portions of
the input domain that are more likely to contain failures. The latter uses the
engineers' domain knowledge. We propose ATheNA, a novel SBST framework that
combines fitness functions that are automatically generated from requirements
specifications and manually defined by engineers. We design and implement
ATheNA-S, an instance of ATheNA that targets Simulink models. We evaluate
ATheNA-S by considering a large set of models and requirements from different
domains. We compare our solution with an SBST baseline tool that supports
automatically generated fitness functions, and another one that supports
manually defined fitness functions. Our results show that ATheNA-S generates
more failure-revealing test cases than the baseline tools and that the
difference between the performance of ATheNA-S and the baseline tools is not
statistically significant. We also assess whether ATheNA-S could generate
failure-revealing test cases when applied to a large case study from the
automotive domain. Our results show that ATheNA-S successfully revealed a
requirement violation in our case study
Evaluating Model Testing and Model Checking for Finding Requirements Violations in Simulink Models
Matlab/Simulink is a development and simulation language that is widely used
by the Cyber-Physical System (CPS) industry to model dynamical systems. There
are two mainstream approaches to verify CPS Simulink models: model testing that
attempts to identify failures in models by executing them for a number of
sampled test inputs, and model checking that attempts to exhaustively check the
correctness of models against some given formal properties. In this paper, we
present an industrial Simulink model benchmark, provide a categorization of
different model types in the benchmark, describe the recurring logical patterns
in the model requirements, and discuss the results of applying model checking
and model testing approaches to identify requirements violations in the
benchmarked models. Based on the results, we discuss the strengths and
weaknesses of model testing and model checking. Our results further suggest
that model checking and model testing are complementary and by combining them,
we can significantly enhance the capabilities of each of these approaches
individually. We conclude by providing guidelines as to how the two approaches
can be best applied together.Comment: 10 pages + 2 page reference
Software Engineering Research and Industry: A Symbiotic Relationship to Foster Impact
Software engineering is not only an increasingly challenging endeavor that goes beyond the intellectual capabilities of any single individual engineer, but is also an intensely human one. Tools and methods to develop software are employed by engineers of varied backgrounds within a large variety of organizations and application domains. As a result, the variation in challenges and practices in system requirements, architecture, and quality assurance is staggering. Human, domain and organizational factors define the context within which software engineering methodologies and technologies are to be applied and therefore the context that research needs to account for, if it is to be impactful. This paper provides an assessment of the current challenges faced by software engineering research in achieving its potential, a description of the root causes of such challenges, and a proposal for the field to move forward and become more impactful through collaborative research and innovation between public research and industry
Seeding Strategies for Multi-Objective Test Case Selection: An Application on Simulation-based Testing
The time it takes software systems to be tested is usually long. This is often caused by the time it takes the entire test suite to be executed. To optimize this, regression test selection approaches have allowed for improvements to the cost-effectiveness of verification and validation activities in the software industry. In this area, multi-objective algorithms have played a key role in selecting the appropriate subset of test cases from the entire test suite. In this paper, we propose a set of seeding strategies for the test case selection problem that generate the initial population of multi-objective algorithms.We integrated these seeding strategies with an NSGA-II algorithm for solving the test case selection problem in the context of simulation-based testing. We evaluated the strategies with six case studies and a total of 21 fitness combinations for each case study (i.e., a total of 126 problems). Our evaluation suggests that these strategies are indeed helpful for solving the multi-objective test case selection problem. In fact, two of the proposed seeding strategies outperformed the NSGA-II algorithm without seeding population with statistical significance for 92.8 and 96% of the problems
Faster Multi-Goal Simulation-Based Testing Using DoLesS (Domination with Least Square Approximation)
For cyber-physical systems, finding a set of test cases with the least cost
by exploring multiple goals is a complex task. For example, Arrieta et al.
reported that state-of-the-art optimizers struggle to find minimal test suites
for this task. To better manage this task, we propose DoLesS (Domination with
Least Squares Approximation) which uses a domination predicate to sort the
space of possible goals to a small number of representative examples.
Multi-objective domination then divides these examples into a "best" set and
the remaining "rest" set. After that, DoLesS applies an inverted least squares
approximation approach to learn a minimal set of tests that can distinguish
best from rest in the reduced example space. DoLesS has been tested on four
cyber-physical models: a tank flow model; a model of electric car windows; a
safety feature of an AC engine; and a continuous PID controller combined with a
discrete state machine. Comparing to the recent state-of-the-art paper
attempted the same task, DoLesS performs as well or even better as
state-of-the-art, while running 80-360 times faster on average (seconds instead
of hours). Hence, we recommend DoLesSas a fast method to find minimal test
suites for multi-goal cyber-physical systems. For replication purposes, all our
code is on-line:https://github.com/hellonull123/Test_Selection_2021.Comment: 10 pages, 4 figures, 6 tables. Submitted to ICSE 202
Applying and Extending the Delta Debugging Algorithm for Elevator Dispatching Algorithms (Experience Paper)
Elevator systems are one kind of Cyber-Physical Systems (CPSs), and as such,
test cases are usually complex and long in time. This is mainly because
realistic test scenarios are employed (e.g., for testing elevator dispatching
algorithms, typically a full day of passengers traveling through a system of
elevators is used). However, in such a context, when needing to reproduce a
failure, it is of high benefit to provide the minimal test input to the
software developers. This way, analyzing and trying to localize the root-cause
of the failure is easier and more agile. Delta debugging has been found to be
an efficient technique to reduce failure-inducing test inputs. In this paper,
we enhance this technique by first monitoring the environment at which the CPS
operates as well as its physical states. With the monitored information, we
search for stable states of the CPS during the execution of the simulation. In
a second step, we use such identified stable states to help the delta debugging
algorithm isolate the failure-inducing test inputs more efficiently.
We report our experience of applying our approach into an industrial elevator
dispatching algorithm. An empirical evaluation carried out with real
operational data from a real installation of elevators suggests that the
proposed environment-wise delta debugging algorithm is between 1.3 to 1.8 times
faster than the traditional delta debugging, while producing a larger reduction
in the failure-inducing test inputs. The results provided by the different
implemented delta debugging algorithm versions are qualitatively assessed with
domain experts. This assessment provides new insights and lessons learned, such
as, potential applications of the delta debugging algorithm beyond debugging
Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: An Approach Based on System Identification
Black-box testing has been extensively applied to test models of Cyber-Physical systems (CPS) since these models are not often amenable to static and symbolic testing and verification. Black-box testing, however, requires to execute the model under test for a large number of candidate test inputs. This poses a challenge for a large and practically-important category of CPS models, known as compute-intensive CPS (CI-CPS) models, where a single simulation may take hours to complete. We propose a novel approach, namely ARIsTEO, to enable effective and efficient testing of CI-CPS models. Our approach embeds black-box testing into an iterative approximation-refinement loop. At the start, some sampled inputs and outputs of the CI-CPS model under test are used to generate a surrogate model that is faster to execute and can be subjected to black-box testing. Any failure-revealing test identified for the surrogate model is checked on the original model. If spurious, the test results are used to refine the surrogate model to be tested again. Otherwise, the test reveals a valid failure. We evaluated ARIsTEO by comparing it with S-Taliro, an open-source and industry-strength tool for testing CPS models. Our results, obtained based on five publicly-available CPS models, show that, on average, ARIsTEO is able to find 24% more requirements violations than S-Taliro and is 31% faster than S-Taliro in finding those violations. We further assessed the effectiveness and efficiency of ARIsTEO on a large industrial case study from the satellite domain. In contrast to S-Taliro, ARIsTEO successfully tested two different versions of this model and could identify three requirements violations, requiring four hours, on average, for each violation
Generating Automated and Online Test Oracles for Simulink Models with Continuous and Uncertain Behaviors
Test automation requires automated oracles to assess test outputs. For cyber physical systems (CPS), oracles, in addition to be automated, should ensure some key objectives: (i) they should check test outputs in an online manner to stop expensive test executions as soon as a failure is detected; (ii) they should handle time- and magnitude-continuous CPS behaviors; (iii) they should provide a quantitative degree of satisfaction or failure measure instead of binary pass/fail outputs; and (iv) they should be able to handle uncertainties due to CPS interactions with the environment. We propose an automated approach to translate CPS requirements specified in a logic-based language into test oracles specified in Simulink - a widely-used development and simulation language for CPS. Our approach achieves the objectives noted above through the identification of a fragment of Signal First Order logic (SFOL) to specify requirements, the definition of a quantitative semantics for this fragment and a sound translation of the fragment into Simulink. The results from applying our approach on 11 industrial case studies show that: (i) our requirements language can express all the 98 requirements of our case studies; (ii) the time and effort required by our approach are acceptable, showing potentials for the adoption of our work in practice, and (iii) for large models, our approach can dramatically reduce the test execution time compared to when test outputs are checked in an offline manner