A controlled experiment for the empirical evaluation of safety analysis techniques for safety-critical software

Context: Today's safety critical systems are increasingly reliant on software. Software becomes responsible for most of the critical functions of systems. Many different safety analysis techniques have been developed to identify hazards of systems. FTA and FMEA are most commonly used by safety analysts. Recently, STPA has been proposed with the goal to better cope with complex systems including software. Objective: This research aimed at comparing quantitatively these three safety analysis techniques with regard to their effectiveness, applicability, understandability, ease of use and efficiency in identifying software safety requirements at the system level. Method: We conducted a controlled experiment with 21 master and bachelor students applying these three techniques to three safety-critical systems: train door control, anti-lock braking and traffic collision and avoidance. Results: The results showed that there is no statistically significant difference between these techniques in terms of applicability, understandability and ease of use, but a significant difference in terms of effectiveness and efficiency is obtained. Conclusion: We conclude that STPA seems to be an effective method to identify software safety requirements at the system level. In particular, STPA addresses more different software safety requirements than the traditional techniques FTA and FMEA, but STPA needs more time to carry out by safety analysts with little or no prior experience.Comment: 10 pages, 1 figure in Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering (EASE '15). ACM, 201

An evaluation of software fault tolerance techniques in real-time safety-critical applications

The usefulness of three software fault tolerance techniques -- n-version programming, recovery blocks, and exception handling is examined within the context of real-time safety-critical environments. The general requirements of such application systems are presented and the techniques evaluated with regard to how well they satisfy these requirements

Development of a software safety process and a case study of its use

The goal of this research is to continue the development of a comprehensive approach to software safety and to evaluate the approach with a case study. The case study is a major part of the project, and it involves the analysis of a specific safety-critical system from the medical equipment domain. The particular application being used was selected because of the availability of a suitable candidate system. We consider the results to be generally applicable and in no way particularly limited by the domain. The research is concentrating on issues raised by the specification and verification phases of the software lifecycle since they are central to our previously-developed rigorous definitions of software safety. The theoretical research is based on our framework of definitions for software safety. In the area of specification, the main topics being investigated are the development of techniques for building system fault trees that correctly incorporate software issues and the development of rigorous techniques for the preparation of software safety specifications. The research results are documented. Another area of theoretical investigation is the development of verification methods tailored to the characteristics of safety requirements. Verification of the correct implementation of the safety specification is central to the goal of establishing safe software. The empirical component of this research is focusing on a case study in order to provide detailed characterizations of the issues as they appear in practice, and to provide a testbed for the evaluation of various existing and new theoretical results, tools, and techniques. The Magnetic Stereotaxis System is summarized

Towards Formally Verified Optimizing Compilation in Flight Control Software

International audienceThis work presents a preliminary evaluation of the use of the CompCert formally specified and verified optimizing compiler for the development of level A critical flight control software. First, the motivation for choosing CompCert is presented, as well as the requirements and constraints for safety-critical avionics software. The main point is to allow optimized code generation by relying on the formal proof of correctness instead of the current un-optimized generation required to produce assembly code structurally similar to the algorithmic language (and even the initial models) source code. The evaluation of its performance (measured using WCET) is presented and the results are compared to those obtained with the currently used compiler. Finally, the paper discusses verification and certification issues that are raised when one seeks to use CompCert for the development of such critical software

The integration of hazard evaluation procedures and requirements engineering for safety-critical embedded systems

Although much work has been done on assessing safety requirements in programmable systems, one very important aspect, the integration of hazard evaluation procedures and requirements engineering, has been somewhat neglected. This thesis describes the derivation and application of a methodology, HAZAPS (HAZard Assessment in Programmable Systems). The methodology assists at the requirements stage in the development of safety-critical embedded systems. The objectives are to identify hazards in programmable systems, construct and model the associated safety requirements, and, finally, to assess these requirements. HAZAPS integrates safety engineering and software modelling techniques. The analysis of more than 300 computer related incidents provided the criteria used to identify, select and modify safety engineering techniques. [Continues.

INTEGRATION OF SOFTWARE SAFETY ASSURANCE PRINCIPLES WITH AN AGILE DEVELOPMENT METHOD

Agile software development has had success in different domains. However there is one area where the implementation of agile methods still needs significant development – that is in the field of agile and safety-critical system development. In this field, software engineering processes need to be justified against the requirements of software safety assurance standards (such as ISO 26262 in the automotive domain). It is therefore important that agile development processes can be justified to levels of assurance equivalent to that provided by traditional development approaches. While there is existing literature concerning the integration of agile methods with specific safety-critical system development standards and agile methods, the question of how fundamental software safety assurance principles can be addressed within agile methods has received little attention. In this thesis we describe the results of practitioner surveys that highlight the primary concerns regarding the use of agile methods within safety-critical development. In the context of this survey, and of existing work on software safety assurance principles, we then present an initial proposal as to how assurance could be addressed with an existing agile development method – Scrum. This proposal was submitted to practitioners for initial feedback and evaluation. The results of this evaluation are also presented

Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving

Deep Neural Networks (DNNs) are rapidly being adopted by the automotive industry, due to their impressive performance in tasks that are essential for autonomous driving. Object segmentation is one such task: its aim is to precisely locate boundaries of objects and classify the identified objects, helping autonomous cars to recognise the road environment and the traffic situation. Not only is this task safety critical, but developing a DNN based object segmentation module presents a set of challenges that are significantly different from traditional development of safety critical software. The development process in use consists of multiple iterations of data collection, labelling, training, and evaluation. Among these stages, training and evaluation are computation intensive while data collection and labelling are manual labour intensive. This paper shows how development of DNN based object segmentation can be improved by exploiting the correlation between Surprise Adequacy (SA) and model performance. The correlation allows us to predict model performance for inputs without manually labelling them. This, in turn, enables understanding of model performance, more guided data collection, and informed decisions about further training. In our industrial case study the technique allows cost savings of up to 50% with negligible evaluation inaccuracy. Furthermore, engineers can trade off cost savings versus the tolerable level of inaccuracy depending on different development phases and scenarios.Comment: to be published in Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineerin

Some issues in numerical simulation of nonlinear structural response

The development of commercial finite element software is addressed. This software provides practical tools that are used in an astonishingly wide range of engineering applications that include critical aspects of the safety evaluation of nuclear power plants or of heavily loaded offshore structures in the hostile environments of the North Sea or the Arctic, major design activities associated with the development of airframes for high strength and minimum weight, thermal analysis of electronic components, and the design of sports equipment. In the more advanced application areas, the effectiveness of the product depends critically on the quality of the mechanics and mechanics related algorithms that are implemented. Algorithmic robustness is of primary concern. Those methods that should be chosen will maximize reliability with minimal understanding on the part of the user. Computational efficiency is also important because there are always limited resources, and hence problems that are too time consuming or costly. Finally, some areas where research work will provide new methods and improvements is discussed