Testing and verification of neural-network-based safety-critical control software: A systematic literature review

Context: Neural Network (NN) algorithms have been successfully adopted in a number of Safety-Critical Cyber-Physical Systems (SCCPSs). Testing and Verification (T&V) of NN-based control software in safety-critical domains are gaining interest and attention from both software engineering and safety engineering researchers and practitioners. Objective: With the increase in studies on the T&V of NN-based control software in safety-critical domains, it is important to systematically review the state-of-the-art T&V methodologies, to classify approaches and tools that are invented, and to identify challenges and gaps for future studies. Method: We retrieved 950 papers on the T&V of NN-based Safety-Critical Control Software (SCCS). To reach our result, we filtered 83 primary papers published between 2001 and 2018, applied the thematic analysis approach for analyzing the data extracted from the selected papers, presented the classification of approaches, and identified challenges. Conclusion: The approaches were categorized into five high-order themes: assuring robustness of NNs, assuring safety properties of NN-based control software, improving the failure resilience of NNs, measuring and ensuring test completeness, and improving the interpretability of NNs. From the industry perspective, improving the interpretability of NNs is a crucial need in safety-critical applications. We also investigated nine safety integrity properties within four major safety lifecycle phases to investigate the achievement level of T&V goals in IEC 61508-3. Results show that correctness, completeness, freedom from intrinsic faults, and fault tolerance have drawn most attention from the research community. However, little effort has been invested in achieving repeatability; no reviewed study focused on precisely defined testing configuration or on defense against common cause failure.Comment: This paper had been submitted to Journal of Information and Software Technology on April 20, 2019,Revised 5 December 2019, Accepted 6 March 2020, Available online 7 March 202

Remote and agile improvement of industrial control and safety systems processes

Digitalization and remote operations introduce new possibilities for continuous and agile improvements of products in operation by exploiting inherent possibilities in software which is easily changeable and deployable. This approach is driven by data analysis, customer expectations and the possibility of frequent deployment over the air of improved software. Adding functionality into software, combined with connectivity to products, opens possibilities for manufacturers and operators, enabling new features and new operational models. This has also become relevant for regulated environments like industrial control and safety systems used in critical infrastructures. Adapted agile processes like SafeScrum and DevOps may be used to achieve continuous improvement. They enable speed and a continuum between development, maintenance and operation. For instance, experience and data from operation on new cybersecurity threats, must be fed back to the maintenance process to be resolved fast. Hence, the DevOps concept, which is imperative in non-safety domains, is now highly relevant in regulated environments as well. The speed of this process is vital where in particular cybersecurity threats must be resolved fast to avoid safety threats. The Agile Safety Case is an enabler of ensuring structured proof of compliance of safety performance for the involved stakeholders. This paper proposes a solution for a safety case which may be applied for continuous product improvements during operation considering safety as well as security. The solution involves the relevant stakeholders and results in a shift in responsibilities.publishedVersio

Formal verification of automotive embedded UML designs

Software applications are increasingly dominating safety critical domains. Safety critical domains are domains where the failure of any application could impact human lives. Software application safety has been overlooked for quite some time but more focus and attention is currently directed to this area due to the exponential growth of software embedded applications. Software systems have continuously faced challenges in managing complexity associated with functional growth, flexibility of systems so that they can be easily modified, scalability of solutions across several product lines, quality and reliability of systems, and finally the ability to detect defects early in design phases. AUTOSAR was established to develop open standards to address these challenges. ISO-26262, automotive functional safety standard, aims to ensure functional safety of automotive systems by providing requirements and processes to govern software lifecycle to ensure safety. Each functional system needs to be classified in terms of safety goals, risks and Automotive Safety Integrity Level (ASIL: A, B, C and D) with ASIL D denoting the most stringent safety level. As risk of the system increases, ASIL level increases and the standard mandates more stringent methods to ensure safety. ISO-26262 mandates that ASILs C and D classified systems utilize walkthrough, semi-formal verification, inspection, control flow analysis, data flow analysis, static code analysis and semantic code analysis techniques to verify software unit design and implementation. Ensuring software specification compliance via formal methods has remained an academic endeavor for quite some time. Several factors discourage formal methods adoption in the industry. One major factor is the complexity of using formal methods. Software specification compliance in automotive remains in the bulk heavily dependent on traceability matrix, human based reviews, and testing activities conducted on either actual production software level or simulation level. ISO26262 automotive safety standard recommends, although not strongly, using formal notations in automotive systems that exhibit high risk in case of failure yet the industry still heavily relies on semi-formal notations such as UML. The use of semi-formal notations makes specification compliance still heavily dependent on manual processes and testing efforts. In this research, we propose a framework where UML finite state machines are compiled into formal notations, specification requirements are mapped into formal model theorems and SAT/SMT solvers are utilized to validate implementation compliance to specification. The framework will allow semi-formal verification of AUTOSAR UML designs via an automated formal framework backbone. This semi-formal verification framework will allow automotive software to comply with ISO-26262 ASIL C and D unit design and implementation formal verification guideline. Semi-formal UML finite state machines are automatically compiled into formal notations based on Symbolic Analysis Laboratory formal notation. Requirements are captured in the UML design and compiled automatically into theorems. Model Checkers are run against the compiled formal model and theorems to detect counterexamples that violate the requirements in the UML model. Semi-formal verification of the design allows us to uncover issues that were previously detected in testing and production stages. The methodology is applied on several automotive systems to show how the framework automates the verification of UML based designs, the de-facto standard for automotive systems design, based on an implicit formal methodology while hiding the cons that discouraged the industry from using it. Additionally, the framework automates ISO-26262 system design verification guideline which would otherwise be verified via human error prone approaches

Resource Bound Guarantees via Programming Languages

We present a programming language in which every well-typed program halts in time polynomial with respect to its input and, more importantly, in which upper bounds on resource requirements can be inferred with certainty. Ensuring that software meets its resource constraints is important in a number of domains, most prominently in hard real-time systems and safety critical systems where failing to meet its time constraints can result in catastrophic failure. The use of test- ing in ensuring resource constraints is of limited use since the testing of every input or environment is impossible in general. Static analysis, whether via the compiler or com- plementary programming tool, can generate proofs of correctness with certainty at the cost that not all programs can be analysed. We describe a programming language, Pola, which provides upper bounds on resource usage for well-typed programs. Further, we describe novel features of Pola that make it more expressive than existing resource-constrained programming languages

Adverse events in veterans affairs inpatient psychiatric units: Staff perspectives on contributing and protective factors.

OBJECTIVES: This study sought to identify risk factors and protective factors in hospital-based mental health settings in the Veterans Health Administration (VHA), with the goal of informing interventions to improve care of persons with serious mental illness. METHODS: Twenty key informants from a stratified sample of 7 VHA inpatient psychiatric units were interviewed to gain their insights on causes of patient safety events and the factors that constrain or facilitate patient safety efforts. RESULTS: Respondents identified threats to patient safety at the system-, provider-, and patient-levels. Protective factors that, when in place, made patient safety events less likely to occur included: promoting a culture of safety; advocating for patient-centeredness; and engaging administrators and organizational leadership to champion these changes. CONCLUSIONS: Findings highlight the impact of systems-level policies and procedures on safety in inpatient mental health care. Engaging all stakeholders, including patients, in patient safety efforts and establishing a culture of safety will help improve the quality of inpatient psychiatric care. Successful implementation of changes require the knowledge of local experts most closely involved in patient care, as well as support and buy-in from organizational leadership

Safety-related challenges and opportunities for GPUs in the automotive domain

GPUs have been shown to cover the computing performance needs of autonomous driving (AD) systems. However, since the GPUs used for AD build on designs for the mainstream market, they may lack fundamental properties for correct operation under automotive's safety regulations. In this paper, we analyze some of the main challenges in hardware and software design to embrace GPUs as the reference computing solution for AD, with the emphasis in ISO 26262 functional safety requirements.Authors would like to thank Guillem Bernat from Rapita Systems for his technical feedback on this work. The research leading to this work has received funding from the European Re-search Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 772773). This work has also been partially supported by the Spanish Ministry of Science and Innovation under grant TIN2015-65316-P and the HiPEAC Network of Excellence. Jaume Abella has been partially supported by the Ministry of Economy and Competitiveness under Ramon y Cajal postdoctoral fellowship number RYC-2013-14717. Carles Hernández is jointly funded by the Spanish Ministry of Economy and Competitiveness and FEDER funds through grant TIN2014-60404-JIN.Peer ReviewedPostprint (author's final draft