94 research outputs found
Conformance Testing as Falsification for Cyber-Physical Systems
In Model-Based Design of Cyber-Physical Systems (CPS), it is often desirable
to develop several models of varying fidelity. Models of different fidelity
levels can enable mathematical analysis of the model, control synthesis, faster
simulation etc. Furthermore, when (automatically or manually) transitioning
from a model to its implementation on an actual computational platform, then
again two different versions of the same system are being developed. In all
previous cases, it is necessary to define a rigorous notion of conformance
between different models and between models and their implementations. This
paper argues that conformance should be a measure of distance between systems.
Albeit a range of theoretical distance notions exists, a way to compute such
distances for industrial size systems and models has not been proposed yet.
This paper addresses exactly this problem. A universal notion of conformance as
closeness between systems is rigorously defined, and evidence is presented that
this implies a number of other application-dependent conformance notions. An
algorithm for detecting that two systems are not conformant is then proposed,
which uses existing proven tools. A method is also proposed to measure the
degree of conformance between two systems. The results are demonstrated on a
range of models
Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions
Search-based software testing (SBST) typically relies on fitness functions to
guide the search exploration toward software failures. There are two main
techniques to define fitness functions: (a) automated fitness function
computation from the specification of the system requirements and (b) manual
fitness function design. Both techniques have advantages. The former uses
information from the system requirements to guide the search toward portions of
the input domain that are more likely to contain failures. The latter uses the
engineers' domain knowledge. We propose ATheNA, a novel SBST framework that
combines fitness functions that are automatically generated from requirements
specifications and manually defined by engineers. We design and implement
ATheNA-S, an instance of ATheNA that targets Simulink models. We evaluate
ATheNA-S by considering a large set of models and requirements from different
domains. We compare our solution with an SBST baseline tool that supports
automatically generated fitness functions, and another one that supports
manually defined fitness functions. Our results show that ATheNA-S generates
more failure-revealing test cases than the baseline tools and that the
difference between the performance of ATheNA-S and the baseline tools is not
statistically significant. We also assess whether ATheNA-S could generate
failure-revealing test cases when applied to a large case study from the
automotive domain. Our results show that ATheNA-S successfully revealed a
requirement violation in our case study
An Efficient Algorithm for Monitoring Practical TPTL Specifications
We provide a dynamic programming algorithm for the monitoring of a fragment
of Timed Propositional Temporal Logic (TPTL) specifications. This fragment of
TPTL, which is more expressive than Metric Temporal Logic, is characterized by
independent time variables which enable the elicitation of complex real-time
requirements. For this fragment, we provide an efficient polynomial time
algorithm for off-line monitoring of finite traces. Finally, we provide
experimental results on a prototype implementation of our tool in order to
demonstrate the feasibility of using our tool in practical applications
ViSpec: A graphical tool for elicitation of MTL requirements
One of the main barriers preventing widespread use of formal methods is the
elicitation of formal specifications. Formal specifications facilitate the
testing and verification process for safety critical robotic systems. However,
handling the intricacies of formal languages is difficult and requires a high
level of expertise in formal logics that many system developers do not have. In
this work, we present a graphical tool designed for the development and
visualization of formal specifications by people that do not have training in
formal logic. The tool enables users to develop specifications using a
graphical formalism which is then automatically translated to Metric Temporal
Logic (MTL). In order to evaluate the effectiveness of our tool, we have also
designed and conducted a usability study with cohorts from the academic student
community and industry. Our results indicate that both groups were able to
define formal requirements with high levels of accuracy. Finally, we present
applications of our tool for defining specifications for operation of robotic
surgery and autonomous quadcopter safe operation.Comment: Technical report for the paper to be published in the 2015 IEEE/RSJ
International Conference on Intelligent Robots and Systems held in Hamburg,
Germany. Includes 10 pages and 19 figure
Test Case Generation for Drivability Requirements of an Automotive Cruise Controller: An Experience with an Industrial Simulator
Automotive software development requires engineers to test their systems to
detect violations of both functional and drivability requirements. Functional
requirements define the functionality of the automotive software. Drivability
requirements refer to the driver's perception of the interactions with the
vehicle; for example, they typically require limiting the acceleration and jerk
perceived by the driver within given thresholds. While functional requirements
are extensively considered by the research literature, drivability requirements
garner less attention. This industrial paper describes our experience assessing
the usefulness of an automated search-based software testing (SBST) framework
in generating failure-revealing test cases for functional and drivability
requirements. Our experience concerns the VI-CarRealTime simulator, an
industrial virtual modeling and simulation environment widely used in the
automotive domain. We designed a Cruise Control system in Simulink for a
four-wheel vehicle, in an iterative fashion, by producing 21 model versions. We
used the SBST framework for each version of the model to search for
failure-revealing test cases revealing requirement violations. Our results show
that the SBST framework successfully identified a failure-revealing test case
for 66.7% of our model versions, requiring, on average, 245.9s and 3.8
iterations. We present lessons learned, reflect on the generality of our
results, and discuss how our results improve the state of practice.Comment: 10 pages papaer plus 2 of bibliography. 10 figures and 6 table
From Formal Requirement Analysis to Testing and Monitoring of Cyber-Physical Systems
abstract: Cyber-Physical Systems (CPS) are being used in many safety-critical applications. Due to the important role in virtually every aspect of human life, it is crucial to make sure that a CPS works properly before its deployment. However, formal verification of CPS is a computationally hard problem. Therefore, lightweight verification methods such as testing and monitoring of the CPS are considered in the industry. The formal representation of the CPS requirements is a challenging task. In addition, checking the system outputs with respect to requirements is a computationally complex problem. In this dissertation, these problems for the verification of CPS are addressed. The first method provides a formal requirement analysis framework which can find logical issues in the requirements and help engineers to correct the requirements. Also, a method is provided to detect tests which vacuously satisfy the requirement because of the requirement structure. This method is used to improve the test generation framework for CPS. Finally, two runtime verification algorithms are developed for off-line/on-line monitoring with respect to real-time requirements. These monitoring algorithms are computationally efficient, and they can be used in practical applications for monitoring CPS with low runtime overhead.Dissertation/ThesisDoctoral Dissertation Computer Science 201
- …