Falsification of Cyber-Physical Systems with Robustness-Guided Black-Box Checking

For exhaustive formal verification, industrial-scale cyber-physical systems (CPSs) are often too large and complex, and lightweight alternatives (e.g., monitoring and testing) have attracted the attention of both industrial practitioners and academic researchers. Falsification is one popular testing method of CPSs utilizing stochastic optimization. In state-of-the-art falsification methods, the result of the previous falsification trials is discarded, and we always try to falsify without any prior knowledge. To concisely memorize such prior information on the CPS model and exploit it, we employ Black-box checking (BBC), which is a combination of automata learning and model checking. Moreover, we enhance BBC using the robust semantics of STL formulas, which is the essential gadget in falsification. Our experiment results suggest that our robustness-guided BBC outperforms a state-of-the-art falsification tool.Comment: Accepted to HSCC 202

Conformance-based doping detection for cyber-physical systems

We present a novel and generalised notion of doping cleanness for cyber-physical systems that allows for perturbing the inputs and observing the perturbed outputs both in the time– and value–domains. We instantiate our definition using existing notions of conformance for cyber-physical systems. We show that our generalised definitions are essential in a data-driven method for doping detection and apply our definitions to a case study concerning diesel emission tests

Conformance Testing for Stochastic Cyber-Physical Systems

Conformance is defined as a measure of distance between the behaviors of two dynamical systems. The notion of conformance can accelerate system design when models of varying fidelities are available on which analysis and control design can be done more efficiently. Ultimately, conformance can capture distance between design models and their real implementations and thus aid in robust system design. In this paper, we are interested in the conformance of stochastic dynamical systems. We argue that probabilistic reasoning over the distribution of distances between model trajectories is a good measure for stochastic conformance. Additionally, we propose the non-conformance risk to reason about the risk of stochastic systems not being conformant. We show that both notions have the desirable transference property, meaning that conformant systems satisfy similar system specifications, i.e., if the first model satisfies a desirable specification, the second model will satisfy (nearly) the same specification. Lastly, we propose how stochastic conformance and the non-conformance risk can be estimated from data using statistical tools such as conformal prediction. We present empirical evaluations of our method on an F-16 aircraft, an autonomous vehicle, a spacecraft, and Dubin's vehicle

On Optimization-Based Falsification of Cyber-Physical Systems

In what is commonly referred to as cyber-physical systems (CPSs), computational and physical resources are closely interconnected. An example is the closed-loop behavior of perception, planning, and control algorithms, executing on a computer and interacting with a physical environment. Many CPSs are safety-critical, and it is thus important to guarantee that they behave according to given specifications that define the correct behavior. CPS models typically include differential equations, state machines, and code written in general-purpose programming languages. This heterogeneity makes it generally not feasible to use analytical methods to evaluate the system’s correctness. Instead, model-based testing of a simulation of the system is more viable. Optimization-based falsification is an approach to, using a simulation model, automatically check for the existence of input signals that make the CPS violate given specifications. Quantitative semantics estimate how far the specification is from being violated for a given scenario. The decision variables in the optimization problems are parameters that determine the type and shape of generated input signals. This thesis contributes to the increased efficiency of optimization-based falsification in four ways. (i) A method for using multiple quantitative semantics during optimization-based falsification. (ii) A direct search approach, called line-search falsification that prioritizes extreme values, which are known to often falsify specifications, and has a good balance between exploration and exploitation of the parameter space. (iii) An adaptation of Bayesian optimization that allows for injecting prior knowledge and uses a special acquisition function for finding falsifying points rather than the global minima. (iv) An investigation of different input signal parameterizations and their coverability of the space and time and frequency domains. The proposed methods have been implemented and evaluated on standard falsification benchmark problems. Based on these empirical studies, we show the efficiency of the proposed methods. Taken together, the proposed methods are important contributions to the falsification of CPSs and in enabling a more efficient falsification process

Discovering Physical Interaction Vulnerabilities in IoT Deployments

Internet of Things (IoT) applications drive the behavior of IoT deployments according to installed sensors and actuators. It has recently been shown that IoT deployments are vulnerable to physical interactions, caused by design flaws or malicious intent, that can have severe physical consequences. Yet, extant approaches to securing IoT do not translate the app source code into its physical behavior to evaluate physical interactions. Thus, IoT consumers and markets do not possess the capability to assess the safety and security risks these interactions present. In this paper, we introduce the IoTSeer security service for IoT deployments, which uncovers undesired states caused by physical interactions. IoTSeer operates in four phases (1) translation of each actuation command and sensor event in an app source code into a hybrid I/O automaton that defines an app's physical behavior, (2) combining apps in a novel composite automaton that represents the joint physical behavior of interacting apps, (3) applying grid-based testing and falsification to validate whether an IoT deployment conforms to desired physical interaction policies, and (4) identification of the root cause of policy violations and proposing patches that guide users to prevent them. We use IoTSeer in an actual house with 13 actuators and six sensors with 37 apps and demonstrate its effectiveness and performance

Codifying Information Assurance Controls for Department of Defense (DoD) Supervisory Control and Data Acquisition (SCADA) Systems (U)

Protecting DoD critical infrastructure resources and Supervisory Control and Data Acquisition (SCADA) systems from cyber attacks is becoming an increasingly challenging task. DoD Information Assurance controls provide a sound framework to achieve an appropriate level of confidentiality, integrity, and availability. However, these controls have not been updated since 2003 and currently do not adequately address the security of DoD SCADA systems. This research sampled U.S. Air Force Civil Engineering subject matter experts representing eight Major Commands that manage and operate SCADA systems. They ranked 30 IA controls in three categories, and evaluated eight SCADA specific IA controls for inclusion into the DoD IA control framework. Spearman’s Rho ranking results (ρ = .972414) indicate a high preference for encryption, and system and information integrity as key IA Controls to mitigate cyber risk. Equally interesting was the strong agreement among raters on ranking certification and accreditation dead last as an effective IA control. The respondents strongly favored including four new IA controls of the eight considered

Evaluating web accessibility and usability for totally blind users at Thailand Cyber University

Thesis (Ed.D.)--Boston UniversityResearch suggests that web-based education increases opportunities for underserved populations to be integrated into educational activities (Schmetzke, 2001; Burgstahler, 2002; Opitz, Savenye, & Rowland, 2003). This may be true for students with disabilities because they have more flexibility to participate in formal education. However, Moisey (2004) found that people with disabilities had lower rates of enrollment and educational achievement than people without disabilities. These findings raise the question of whether or not web-based = education helps increase students with disabilities' access to learning opportunities and improve their learning outcome. This study investigated the degree of difficulty blind persons had in accessing and using web-based educational resources provided by Thailand Cyber University (TCU). Based on a mixed methods design, the data were collected in two phases. Quantitative data were collected first, in order to identify accessibility problems and conformance levels reported by automated web accessibility evaluation tools. Qualitative data was collected from interviews with blind participants in the second phase to expand the understanding of the accessibility problems and usability issues that were not discovered in the quantitative phase by the automated web accessibility evaluation tools. The findings indicate that all of the 13 selected web pages failed to meet a minimum requirement of WCAG 2.0. This means those selected web pages would be inaccessible for the blind. However, the findings indicate blind participants rated only one of the 13 pages as inaccessible. Moreover, their ratings of difficulty on "usability" were higher than their ratings of difficulty on "accessibility" on the same web page. On six out of 22 tasks, blind and sighted user groups agreed on the ratings. Nevertheless, the time that it took to complete each task varied greatly between the two user groups.2031-01-0