A controlled experiment for the empirical evaluation of safety analysis techniques for safety-critical software

Context: Today's safety critical systems are increasingly reliant on software. Software becomes responsible for most of the critical functions of systems. Many different safety analysis techniques have been developed to identify hazards of systems. FTA and FMEA are most commonly used by safety analysts. Recently, STPA has been proposed with the goal to better cope with complex systems including software. Objective: This research aimed at comparing quantitatively these three safety analysis techniques with regard to their effectiveness, applicability, understandability, ease of use and efficiency in identifying software safety requirements at the system level. Method: We conducted a controlled experiment with 21 master and bachelor students applying these three techniques to three safety-critical systems: train door control, anti-lock braking and traffic collision and avoidance. Results: The results showed that there is no statistically significant difference between these techniques in terms of applicability, understandability and ease of use, but a significant difference in terms of effectiveness and efficiency is obtained. Conclusion: We conclude that STPA seems to be an effective method to identify software safety requirements at the system level. In particular, STPA addresses more different software safety requirements than the traditional techniques FTA and FMEA, but STPA needs more time to carry out by safety analysts with little or no prior experience.Comment: 10 pages, 1 figure in Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering (EASE '15). ACM, 201

An improved approach for flight readiness assessment

An improved methodology for quantitatively evaluating failure risk for a spaceflight system in order to assess flight readiness is presented. This methodology is of particular value when information relevant to failure prediction, including test experience and knowledge of parameters used in engineering analyses of failure phenomena, is limited. In this approach, engineering analysis models that characterize specific failure modes based on the physics and mechanics of the failure phenomena are used in a prescribed probabilistic structure to generate a failure probability distribution that is modified by test and flight experience in a Bayesian statistical procedure. The probabilistic structure and statistical methodology are generally applicable to any failure mode for which quantitative engineering analysis can be employed to characterize the failure phenomenon and are particularly well suited for use under the constraints on information availability that are typical of such spaceflight systems as the Space Shuttle and planetary spacecraft

A safety analysis approach to clinical workflows : application and evaluation

Clinical workflows are safety critical workflows as they have the potential to cause harm or death to patients. Their safety needs to be considered as early as possible in the development process. Effective safety analysis methods are required to ensure the safety of these high-risk workflows, because errors that may happen through routine workflow could propagate within the workflow to result in harmful failures of the system’s output. This paper shows how to apply an approach for safety analysis of clinic al workflows to analyse the safety of the workflow within a radiology department and evaluates the approach in terms of usability and benefits. The outcomes of using this approach include identification of the root causes of hazardous workflow failures that may put patients’ lives at risk. We show that the approach is applicable to this area of healthcare and is able to present added value through the detailed information on possible failures, of both their causes and effects; therefore, it has the potential to improve the safety of radiology and other clinical workflows

The practice of risk management by cost consultants in Northern Ireland

This research endeavoured to explore the practice of risk management by cost consultants in Northern Ireland. It attempted to subjectively investigate the cost consultant’s appreciation of risk management practices and then further appraise the cost consultant’s understanding and usage of the theories and techniques available to manage risk under the risk management framework. A case study based approach involving five consultancy practices was adopted. A series of semi structured interviews (one per each case study) was carried out. The data collected was analysed using the Delphi technique. The practice of risk management for each organisation was documented using an analysis and evaluation of project documentation substantiated with interviews. The research indicated that consultants have a broad awareness of risk management but disparity exists on considering it as a core service. All consultants were unequivocal in identifying the need for an improved risk management framework. It was evident that there was a lack of knowledge of the array of risk identification and analysis techniques available. The research has established that there is a severe need to bridge the void between the theories and techniques used to manage risk and those which are implemented in practice. There is a necessity to train consultants in the practice of risk management and educate clients in the benefits of enforcing risk management practices as an integral part of project delivery

Fault Insertions into Hardware-in-the-Loop Simulation

The Ohio State EcoCAR Mobility challenge is an intercollegiate team that designs, builds, and tests a hybrid electric vehicle. One of the main goals of this team is to build a hybrid supervisory controls strategy that tests the potential failure mechanisms derived from fault analysis. Currently, Automotive companies are focused on integrating model-based designs enabling simulations for low-cost, rapid experimentation that assess a vehicle's performance. Model-based designs allow engineers to simulate specific tests within controlled environmental conditions. Through the use of model-based design, engineers can test vehicle and component faults inside a simulation model to assess how the vehicle behaves during various failures without incurring the cost of destructive testing. This thesis, in partner with the EcoCAR Mobility Challenge, aims to incorporate modern industrial fault diagnostics into a hardware-in-the-loop (HIL) simulation and analyze the performance of the model-based design. Fault Tree Analysis (FTA) and Failure Mode and Effect Analysis (FMEA) were used to develop the necessary requirements for the vehicle system. Different faults were intended to be tested for each major component, including, but not limited to, the energy storage system (ESS), rear electric motor, belted alternator starter, DC-DC converter, and the multiplexed vehicle electrical center. The ESS was the only component demonstrated as an example for integrating the fault insertion method. The research details how a standard method was constructed for developing and inserting faults in the HIL test environment. The process is used for testing and designing the control algorithm for a hybrid supervisor controller.No embargoAcademic Major: Mechanical Engineerin

A systems approach to the development and use of FMEA in complex automotive applications

YesThe effective deployment of FMEAs within complex automotive applications faces a number of challenges, including the complexity of the system being analysed, the need to develop a series of coherently linked FMEAs at different levels within the systems hierarchy and across intrinsically interlinked engineering disciplines, and the need for coherent linkage between critical design characteristics cascaded through the systems levels with their counterparts in manufacturing. The approach presented in this paper to address these challenges is based on a structured Failure Mode Avoidance (FMA) framework which promotes the development of FMEAs within an integrated Systems Engineering approach. The effectiveness of the framework is illustrated through a case study, centred on the development of a diesel exhaust aftertreatment system. This case study demonstrates that the structured FMA framework for function analysis supports an effective decomposition of complex interdisciplinary systems facilitating the DFMEA deployment through a series of containable, structured DFMEAs developed at successive system levels, with clear vertical integration of functional requirements and critical parameters cascade. The paper also discusses the way in which the approach supports deployment across engineering disciplines and domains, ensuring the integrity of information flow between the design and manufacturing activities

Proposing the Use of Hazard Analysis for Machine Learning Data Sets

There is no debating the importance of data for artificial intelligence. The behavior of data-driven machine learning models is determined by the data set, or as the old adage states: “garbage in, garbage out (GIGO).” While the machine learning community is still debating which techniques are necessary and sufficient to assess the adequacy of data sets, they agree some techniques are necessary. In general, most of the techniques being considered focus on evaluating the volumes of attributes. Those attributes are evaluated with respect to anticipated counts of attributes without considering the safety concerns associated with those attributes. This paper explores those techniques to identify instances of too little data and incorrect attributes. Those techniques are important; however, for safety critical applications, the assurance analyst also needs to understand the safety impact of not having specific attributes present in the machine learning data sets. To provide that information, this paper proposes a new technique the authors call data hazard analysis. The data hazard analysis provides an approach to qualitatively analyze the training data set to reduce the risk associated with the GIGO