Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems

Due to the increasing usage of machine learning (ML) techniques in security- and safety-critical domains, such as autonomous systems and medical diagnosis, ensuring correct behavior of ML systems, especially for different corner cases, is of growing importance. In this paper, we propose a generic framework for evaluating security and robustness of ML systems using different real-world safety properties. We further design, implement and evaluate VeriVis, a scalable methodology that can verify a diverse set of safety properties for state-of-the-art computer vision systems with only blackbox access. VeriVis leverage different input space reduction techniques for efficient verification of different safety properties. VeriVis is able to find thousands of safety violations in fifteen state-of-the-art computer vision systems including ten Deep Neural Networks (DNNs) such as Inception-v3 and Nvidia's Dave self-driving system with thousands of neurons as well as five commercial third-party vision APIs including Google vision and Clarifai for twelve different safety properties. Furthermore, VeriVis can successfully verify local safety properties, on average, for around 31.7% of the test images. VeriVis finds up to 64.8x more violations than existing gradient-based methods that, unlike VeriVis, cannot ensure non-existence of any violations. Finally, we show that retraining using the safety violations detected by VeriVis can reduce the average number of violations up to 60.2%.Comment: 16 pages, 11 tables, 11 figure

Refining SCJ Mission Specifications into Parallel Handler Designs

Safety-Critical Java (SCJ) is a recent technology that restricts the execution and memory model of Java in such a way that applications can be statically analysed and certified for their real-time properties and safe use of memory. Our interest is in the development of comprehensive and sound techniques for the formal specification, refinement, design, and implementation of SCJ programs, using a correct-by-construction approach. As part of this work, we present here an account of laws and patterns that are of general use for the refinement of SCJ mission specifications into designs of parallel handlers used in the SCJ programming paradigm. Our notation is a combination of languages from the Circus family, supporting state-rich reactive models with the addition of class objects and real-time properties. Our work is a first step to elicit laws of programming for SCJ and fits into a refinement strategy that we have developed previously to derive SCJ programs.Comment: In Proceedings Refine 2013, arXiv:1305.563

Safety-related challenges and opportunities for GPUs in the automotive domain

GPUs have been shown to cover the computing performance needs of autonomous driving (AD) systems. However, since the GPUs used for AD build on designs for the mainstream market, they may lack fundamental properties for correct operation under automotive's safety regulations. In this paper, we analyze some of the main challenges in hardware and software design to embrace GPUs as the reference computing solution for AD, with the emphasis in ISO 26262 functional safety requirements.Authors would like to thank Guillem Bernat from Rapita Systems for his technical feedback on this work. The research leading to this work has received funding from the European Re-search Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 772773). This work has also been partially supported by the Spanish Ministry of Science and Innovation under grant TIN2015-65316-P and the HiPEAC Network of Excellence. Jaume Abella has been partially supported by the Ministry of Economy and Competitiveness under Ramon y Cajal postdoctoral fellowship number RYC-2013-14717. Carles Hernández is jointly funded by the Spanish Ministry of Economy and Competitiveness and FEDER funds through grant TIN2014-60404-JIN.Peer ReviewedPostprint (author's final draft

Assisted assignment of automotive safety requirements

ISO 26262, a functional-safety standard, uses Automotive Safety Integrity Levels (ASILs) to assign safety requirements to automotive-system elements. System designers initially assign ASILs to system-level hazards and then allocate them to elements of the refined system architecture. Through ASIL decomposition, designers can divide a function & rsquo;s safety requirements among multiple components. However, in practice, manual ASIL decomposition is difficult and produces varying results. To overcome this problem, a new tool automates ASIL allocation and decomposition. It supports the system and software engineering life cycle by enabling users to efficiently allocate safety requirements regarding systematic failures in the design of critical embedded computer systems. The tool is applicable to industries with a similar concept of safety integrity levels. © 1984-2012 IEEE

Exploring the impact of different cost heuristics in the allocation of safety integrity levels

Contemporary safety standards prescribe processes in which system safety requirements, captured early and expressed in the form of Safety Integrity Levels (SILs), are iteratively allocated to architectural elements. Different SILs reflect different requirements stringencies and consequently different development costs. Therefore, the allocation of safety requirements is not a simple problem of applying an allocation "algebra" as treated by most standards; it is a complex optimisation problem, one of finding a strategy that minimises cost whilst meeting safety requirements. One difficulty is the lack of a commonly agreed heuristic for how costs increase between SILs. In this paper, we define this important problem; then we take the example of an automotive system and using an automated approach show that different cost heuristics lead to different optimal SIL allocations. Without automation it would have been impossible to explore the vast space of allocations and to discuss the subtleties involved in this problem

Thermal oscillations in the decomposition of organic peroxides: Identification of a hazard, utilization, and suppression

The purpose of this research is to identify and characterize oscillatory thermal instability in organic peroxides that are used in vast quantities in industry and misused by terrorists. The explosive thermal decompositions of lauroyl peroxide, methyl ethyl ketone peroxide, and triacetone triperoxide are investigated computationally, using a continuous stirred tank reactor model and literature values of the kinetic and thermal parameters. Mathematical stability analysis is used to identify and track the oscillatory instability, which may be violent. In the mild oscillatory regime it is shown that, in principle, the oscillatory thermal signal may be used in microcalorimetry to detect and identify explosives. Stabilization of peroxide thermal decomposition via Endex coupling is investigated. It is usually assumed that initiation of explosive thermal decomposition occurs via classical (Semenov) ignition at a turning point or saddle-node bifurcation, but this work shows that oscillatory ignition is also characteristic of thermoreactive liquids and that Semenov theory and purely steady state analyses are inadequate for identifying a thermal hazard in such systems