Using Valued Booleans to Find Simpler Counterexamples in Random Testing of Cyber-Physical Systems

We propose a new logic of valued Booleans for writing properties which are not just true or false but compute how severely they are falsified. The logic is reminiscent of STL or MTL but gives the tester control over what severity means in the particular problem domain. We use this logic to simplify failing test inputs in the context of random testing of cyber-physical systems and show that it improves the quality of counterexamples found. The logic of valued Booleans might also be used as an alternative to the standard robust semantics of STL formulas in optimization-based approaches to falsification

On Falsification of Large-Scale Cyber-Physical Systems

In the development of modern Cyber-Physical Systems, Model-Based Testingof the closed-loop system is an approach for finding potential faults andincreasing quality of developed products. Testing is done on many differentabstraction levels, and for large-scale industrial systems, there are severalchallenges. Executing tests on the systems can be time-consuming and largenumbers of complex specifications need to be thoroughly tested, while manyof the popular academic benchmarks do not necessarily reflect on this complexity.This thesis proposes new methods for analyzing and generating test casesas a means for being more certain that proper testing has been performed onthe system under test. For analysis, the proposed approach can automaticallyfind out how much of the physical parts of the system that the test suite hasexecuted.For test case generation, an approach to find errors is optimization-basedfalsification. This thesis attempts to close the gap between academia and industryby applying falsification techniques to real-world models from VolvoCar Corporation and adapting the falsification procedure where it has shortcomingsfor certain classes of systems. Specifically, the main contributionsof this thesis are (i) a method for automatically transforming a signal-basedspecification into a formal specification allowing an optimization-based falsificationapproach, (ii) a new collection of specifications inspired by large-scalespecifications from industry, (iii) an algorithm to perform optimization-basedfalsification for such a large set of specifications, and (iv) a new type of coveragecriterion for Cyber-Physical Systems that can help to assess when testingcan be concluded.The proposed methods have been evaluated for both academic benchmarkexamples and real-world industrial models. One of the main conclusions isthat the proposed additions and changes to the analysis and generation oftests can be useful, given that one has enough information about the systemunder test. The methods presented in this thesis have been applied to realworldmodels in a way that allows for higher-quality products by finding morefaults in early phases of development

On Optimization-Based Falsification of Cyber-Physical Systems

In what is commonly referred to as cyber-physical systems (CPSs), computational and physical resources are closely interconnected. An example is the closed-loop behavior of perception, planning, and control algorithms, executing on a computer and interacting with a physical environment. Many CPSs are safety-critical, and it is thus important to guarantee that they behave according to given specifications that define the correct behavior. CPS models typically include differential equations, state machines, and code written in general-purpose programming languages. This heterogeneity makes it generally not feasible to use analytical methods to evaluate the system’s correctness. Instead, model-based testing of a simulation of the system is more viable. Optimization-based falsification is an approach to, using a simulation model, automatically check for the existence of input signals that make the CPS violate given specifications. Quantitative semantics estimate how far the specification is from being violated for a given scenario. The decision variables in the optimization problems are parameters that determine the type and shape of generated input signals. This thesis contributes to the increased efficiency of optimization-based falsification in four ways. (i) A method for using multiple quantitative semantics during optimization-based falsification. (ii) A direct search approach, called line-search falsification that prioritizes extreme values, which are known to often falsify specifications, and has a good balance between exploration and exploitation of the parameter space. (iii) An adaptation of Bayesian optimization that allows for injecting prior knowledge and uses a special acquisition function for finding falsifying points rather than the global minima. (iv) An investigation of different input signal parameterizations and their coverability of the space and time and frequency domains. The proposed methods have been implemented and evaluated on standard falsification benchmark problems. Based on these empirical studies, we show the efficiency of the proposed methods. Taken together, the proposed methods are important contributions to the falsification of CPSs and in enabling a more efficient falsification process

Incorporating Measurement Uncertainty into OCL/UML Primitive Datatypes

Preprint publicado en la revista Software & System Modeling :Bertoa, M.F., Burgueño, L., Moreno, N., Vallecillo, A. "Incorporating measurement uncertainty into OCL/UML primitive datatypes." Softw Syst Model (2019). https://doi.org/10.1007/s10270-019-00741-0)The correct representation of the relevant properties of a system is an essential requirement for the effective use and wide adoption of model-based practices in industry. Uncertainty is one of the inherent properties of any measurement or estimation that is obtained in any physical setting; as such, it must be considered when modeling software systems that deal with real data. Although a few modeling languages enable the representation of measurement uncertainty, these aspects are not normally incorporated into their type systems. Therefore, operating with uncertain values and propagating their uncertainty become cumbersome processes, which hinder their realization in real environments. This paper proposes an extension of OCL/UML primitive datatypes that enables the representation of the uncertainty that comes from physical measurements or user estimates into the models, together with an algebra of operations that are defined for the values of these types.TIN2014-52034-R, TIN2016-75944-R and PGC2018-094905-B-I

A PVS-Simulink Integrated Environment for Model-Based Analysis of Cyber-Physical Systems

This paper presents a methodology, with supporting tool, for formal modeling and analysis of software components in cyber-physical systems. Using our approach, developers can integrate a simulation of logic-based specifications of software components and Simulink models of continuous processes. The integrated simulation is useful to validate the characteristics of discrete system components early in the development process. The same logic-based specifications can also be formally verified using the Prototype Verification System (PVS), to gain additional confidence that the software design complies with specific safety requirements. Modeling patterns are defined for generating the logic-based specifications from the more familiar automata-based formalism. The ultimate aim of this work is to facilitate the introduction of formal verification technologies in the software development process of cyber-physical systems, which typically requires the integrated use of different formalisms and tools. A case study from the medical domain is used to illustrate the approach. A PVS model of a pacemaker is interfaced with a Simulink model of the human heart. The overall cyber-physical system is co-simulated to validate design requirements through exploration of relevant test scenarios. Formal verification with the PVS theorem prover is demonstrated for the pacemaker model for specific safety aspects of the pacemaker design

Development of a Framework for CPS Open Standards and Platforms

This technical report describes a Framework we have developed through our research and investigations in this project, with the goal to facilitate creation of Open Standards and Platforms for CPS; a task that addresses a critical mission for NIST. The rapid development of information technology (in terms of processing power, embedded hardware and software systems, comprehensive IT management systems, networking and Internet growth, system design environments) is producing an increasing number of applications and opening new doors. In addition over the last decade we entered a new era where systems complexity has increased dramatically. Complexity is increased both by the number of components that are included in each system as well as by the dependencies between those components. Increasingly, systems tend to be more software dependent and that is a major challenge that engineers involved in the development of such systems face. The challenge is even greater when a safety critical system is considered, like an airplane or a passenger car. Software-intensive systems and devices have become everyday consumables. There is a need for development of software that is provably error-free. Thanks to their multifaceted support for networking and inclusion of data and services from global networks, systems are evolving to form integrated, overarching solutions that are increasingly penetrating all areas of life and work. When software dependent systems interact with the physical environment then we have the class of cyber-physical systems (CPS) [1, 2]. The challenge in CPS is to incorporate the inputs (and their characteristics and constraints) from the physical components in the logic of the cyber components (hardware and software). CPS are engineered systems constructed as networked interactions of physical and computational (cyber) components. In CPS, computations and communication are deeply embedded in and interacting with physical processes, and add new capabilities to physical systems. Competitive pressure and societal needs drive industry to design and deploy airplanes and cars that are more energy efficient and safe, medical devices and systems that are more dependable, defense systems that are more autonomous and secure. Whole industrial sectors are transformed by new product lines that are CPS-based. Modern CPSs are not simply the connection of two different kinds of components engineered by means of distinct design technology, but rather, a new system category that is both physical and computational [1, 2]. Current industrial experience tells us that, in fact, we have reached the limits of our knowledge of how to combine computers and physical systems. The shortcomings range from technical limitations in the foundations of cyber-physical systems to the way we organize our industries and educate engineers and scientists that support cyber-physical system design. If we continue to build systems using our very limited methods and tools but lack the science and technology foundations, we will create significant risks, produce failures and lead to loss of market. Nowadays, with increasing frequency we observe systems that cooperate to achieve a common goal, even though there were not built for that reason. These are called systems of systems. For example, the Global Positioning System (GPS) is a system by itself. However, it needs to cooperate with other systems when the air traffic control system of systems is under 3 consideration. The analysis and development of such systems should be done carefully because of the emergent behavior that systems exhibit when they are coupled with other systems. However, apart from the increasing complexity and the other technical challenges, there is a need to decrease time-to-market for new systems as well as the associated costs. This specific trend and associated requirements, which are an outcome of global competitiveness, are expected to continue and become even more stringent. If a successful contribution is to be made in shaping this change, the revolutionary potential of CPS must be recognized and incorporated into internal development processes at an early stage. For that Interoperability and Integratability of CPS is critical. In this Task we have developed a Framework to facilitate interoperability and integratability of CPS via Open Standards and Platforms. The purpose of this technical report is to introduce this Framework and its critical components, to provide various instantiations of it, and to describe initial successful applications of it in various important classes of CPS. An additional goal of publishing this technical report is to solicit feedback on the proposed Framework, and to catalyze discussions and interactions in the broader CPS technical community towards improving and strengthening this Framework. CPS integrate data and services from different systems which were developed independently and with disparate objectives, thereby enabling new functionalities and benefits. Currently there is a lack of well-defined interfaces that on the one hand define the standards for the form and content of the data being exchanged, but on the other hand take account of non-functional aspects of this data, such as differing levels of data quality or reliability. A similar situation exists with respect to tools and synthesis environments, although some work has been initiated in the latter. The technological prerequisite for the design of the aforementioned various functions and value added services of CPS is the interoperability and integratability of these systems as well as their capability to be adapted flexibly and application-specifically as well as extended at the different levels of abstraction. Dependent on the objective and scope of the application, it may be necessary to integrate component functions (Embedded Systems (ES), System of Systems (SoS), CPS), to establish communication and interfaces, and to ensure the required level of quality of interaction and also of the overall system behavior. This requires cross-domain concepts for architecture, communication and compatibility at all levels. The effects of these factors on existing or yet undeveloped systems and architectures represent a major challenge. Investigation into these factors is the objective of current national and international studies and research projects. CPS create core technological challenges for traditional system architectures, especially because of their high degree of connectivity. This is because CPS are not constructed for one specific purpose or function, but rather are open for many different services and processes, and must therefore be adaptable. In view of their evolutionary nature, they are only controllable to a limited extent. This creates new demands for greater interoperability and communication within CPS that cannot be met by current closed systems. In particular, the differences in the characteristics of embedded systems in relation to IT systems and services and data in networks lead to outstanding questions in relation to the form of architectures, the definition of system and communication interfaces and requirements for underlying CPS platforms with basic services and parallel architectures at different levels of abstraction. 4 The technological developments underlying CPS evolution require the development of standards in the individual application domains, as well as basic infrastructure investments that cannot be borne by individual companies alone. This is particularly significant for SMEs. The development and operation of uniform platforms to migrate individual services and products will therefore be as much of a challenge as joint specification standards. The creation of such quasi standards, less in the traditional mold of classic industry norms and standards and more in the sense of de facto standards that become established on the basis of technological and market dominance, will become an essential part of technological and market leadership. To summarize and emphasize, the complexity of the subject in terms of the required technologies and capabilities of CPS, as well as the capabilities and competences required to develop, control and design/ create innovative, usable CPS applications, demand fundamentally integrated action, interdisciplinarity (research and development, economy and society) and vertical and horizontal efforts in: The creation of open, cross-domain platforms with fundamental services (communication, networking, interoperability) and architectures (including domainspecific architectures); The complementary expansion and integration of application fields and environments with vertical experimentation platforms and correspondingly integrated interdisciplinary efforts; The systematic enhancement with respect to methods and technologies across all involved disciplines to create innovative CPS. The aim of our research and investigations under this Task of the project, was precisely to clarify these objectives and systematically develop detailed recommendations for action. Our research and investigations have identified the following essential and fundamental challenges for the modeling, design, synthesis and manufacturing of CPS: (i) The creation and demonstration of a framework for developing cross-domain integrated modeling hubs for CPS. (ii) The creation and demonstration of a framework for linking the integrated CPS modeling hub of (i) with powerful and diverse tradeoff analysis methods and tools for design exploration for CPS. (iii) The creation of a framework of linking the integrated CPS synthesis environment of (i) and (ii) with databases of modular component and process (manufacturing) models, backwards compatible with earlier legacy systems; (iv)The creation of a framework for translating textual requirements to mathematical representations as constraints, rules and metrics involving both logical and numerical variables and the automatic (at least to 75%) allocation of the resulting specifications to components of the CPS and of processes, in a way that allows traceability. 5 These challenges have been listed here in the order of increasing difficulty both conceptually and in terms of arriving at implementable solutions. The order also reflects the extent to which the current state of affairs has made progress towards developing at least some initial instantiations of the desired frameworks. In this context, it is useful to compare with the advanced state of development of similar frameworks and their instantiations for synthesis and manufacturing of complex microelectronic VLSI chips including distributed ones, which have been available as integrated tools by several vendors for at least a decade. Regarding challenge (i) we have performed extensive work and research in this project towards developing model-based systems engineering (MBSE) procedures for the design, integration, testing and operational management of cyber-physical systems, that is, physical systems with cyber potentially embedded in every physical component. Thus in the Framework, described in this report, for standards for integrated modeling hubs for CPS, MBSE methods and tools are prominent. Regarding the search for a framework for standards for CPS this selection has the additional advantage that it is also emerging as an accepted framework for systems engineering by all industry sectors with substantial interest in CPS [3, 7]. Regarding challenge (ii) we have performed extensive work and research in this project towards developing the foundations for such an integration, and we have developed and demonstrated the first ever integration of a powerful tradeoff analysis tool (and methodology) with our SysMLIntegrated system modeling environments for CPS synthesis [3, 7]. Primary applications of interest that we have instantiated this framework are: microgrids and power grids, wireless sensor networks (WSN) and applications to Smart Grid, energy efficient buildings, microrobotics and collaborative robotics, and the overarching (for all these applications) security and trust issues including our pioneering and innovative work on compositional security systems. A key concept here is the integration of multi-criteria, multi constraint optimization with constrained based reasoning. Regarding challenge (iii) we have only developed the conceptual Framework, as any required instantiations will require substantial commercial grade software development beyond the scope of this project. It is clear however that object-relational databases and database mediators (for both data and semantics) will have to be employed. Regarding challenge (iv) we have developed a Framework for checking and validating specifications, after they have been translated to their mathematical representations as constraints and metrics with logical and numerical variables. Various multi-criteria optimization, constrained based reasoning, model checking and automatic theorem proving tools will have to be combined. The automatic annotation of the system blocks with requirements and parameter specifications remains an open challenge.Research supported in part by Cooperative Agreement, NIST 70NANB11H148, to the University of Maryland College Park