On Space Exploration and Human Error: A Paper on Reliability and Safety

NASA space exploration should largely address a problem class in reliability and risk management stemming primarily from human error, system risk and multi-objective trade-off analysis, by conducting research into system complexity, risk characterization and modeling, and system reasoning. In general, in every mission we can distinguish risk in three possible ways: a) known-known, b) known-unknown, and c) unknown-unknown. It is probably almost certain that space exploration will partially experience similar known or unknown risks embedded in the Apollo missions, Shuttle or Station unless something alters how NASA will perceive and manage safety and reliabilit

A voyage to Mars: A challenge to collaboration between man and machines

A speech addressing the design of man machine systems for exploration of space beyond Earth orbit from the human factors perspective is presented. Concerns relative to the design of automated and intelligent systems for the NASA Space Exploration Initiative (SEI) missions are largely based on experiences with integrating humans and comparable systems in aviation. The history, present status, and future prospect, of human factors in machine design are discussed in relation to a manned voyage to Mars. Three different cases for design philosophy are presented. The use of simulation is discussed. Recommendations for required research are given

Safety Engineering with COTS components

Safety-critical systems are becoming more widespread, complex and reliant on software. Increasingly they are engineered through Commercial Off The Shelf (COTS) (Commercial Off The Shelf) components to alleviate the spiralling costs and development time, often in the context of complex supply chains. A parallel increased concern for safety has resulted in a variety of safety standards, with a growing consensus that a safety life cycle is needed which is fully integrated with the design and development life cycle, to ensure that safety has appropriate influence on the design decisions as system development progresses. In this article we explore the application of an integrated approach to safety engineering in which assurance drives the engineering process. The paper re- ports on the outcome of a case study on a live industrial project with a view to evaluate: its suitability for application in a real-world safety engineering setting; its benefits and limitations in counteracting some of the difficulties of safety en- gineering with COTS components across supply chains; and, its effectiveness in generating evidence which can contribute directly to the construction of safety cases

"Making Safety Happen" Through Probabilistic Risk Assessment at NASA

NASA is using Probabilistic Risk Assessment (PRA) as one of the tools in its Safety & Mission Assurance (S&MA) tool belt to identify and quantify risks associated with human spaceflight. This paper discusses some of the challenges and benefits associated with developing and using PRA for NASA human space programs. Some programs have entered operation prior to developing a PRA, while some have implemented PRA from the start of the program. It has been observed that the earlier a design change is made in the concept or design phase, the less impact it has on cost and schedule. Not finding risks until the operation phase yields much costlier design changes and major delays, which can result in discussions of just accepting the risk. Risk contributors identified by PRA are not just associated with hardware failures. They include but are not limited to crew fatality due to medical causes, the environment the vehicle and crew are exposed to, the software being used, and the reliability of the crew performing required actions. Some programs have entered operation prior to developing a PRA, and while PRA can still provide a benefit for operations and future design trades, the benefit of implementing PRA from the start of the program provides the added benefit of informing design and reducing risk early in program development. Currently, NASAs International Space Station (ISS) program is in its 20th year of on-orbit operations around the Earth and has several new programs in the design phase preparing to enter the operation phase all of which have active (or living) PRAs. These programs incorporate PRA as part of their Risk-Informed, Decision-Making (RIDM) process. For new NASA human spaceflight programs discussion begins with mission concept, establishing requirements, forming the PRA team, and continues through the design cycles into the operational phase. Several examples of PRA related applications and observed lessons are included

A synthesis of logic and biology in the design of dependable systems

The technologies of model-based design and dependability analysis in the design of dependable systems, including software intensive systems, have advanced in recent years. Much of this development can be attributed to the application of advances in formal logic and its application to fault forecasting and verification of systems. In parallel, work on bio-inspired technologies has shown potential for the evolutionary design of engineering systems via automated exploration of potentially large design spaces. We have not yet seen the emergence of a design paradigm that combines effectively and throughout the design lifecycle these two techniques which are schematically founded on the two pillars of formal logic and biology. Such a design paradigm would apply these techniques synergistically and systematically from the early stages of design to enable optimal refinement of new designs which can be driven effectively by dependability requirements. The paper sketches such a model-centric paradigm for the design of dependable systems that brings these technologies together to realise their combined potential benefits

Trial without Error: Towards Safe Reinforcement Learning via Human Intervention

AI systems are increasingly applied to complex tasks that involve interaction with humans. During training, such systems are potentially dangerous, as they haven't yet learned to avoid actions that could cause serious harm. How can an AI system explore and learn without making a single mistake that harms humans or otherwise causes serious damage? For model-free reinforcement learning, having a human "in the loop" and ready to intervene is currently the only way to prevent all catastrophes. We formalize human intervention for RL and show how to reduce the human labor required by training a supervised learner to imitate the human's intervention decisions. We evaluate this scheme on Atari games, with a Deep RL agent being overseen by a human for four hours. When the class of catastrophes is simple, we are able to prevent all catastrophes without affecting the agent's learning (whereas an RL baseline fails due to catastrophic forgetting). However, this scheme is less successful when catastrophes are more complex: it reduces but does not eliminate catastrophes and the supervised learner fails on adversarial examples found by the agent. Extrapolating to more challenging environments, we show that our implementation would not scale (due to the infeasible amount of human labor required). We outline extensions of the scheme that are necessary if we are to train model-free agents without a single catastrophe

Safety in maritime oil sector: Content analysis of machinery space fire hazards

An in-depth study of the practice within the maritime oil industry was undertaken to ascertain safety issues in seafaring vessels. It was more concentrated on the type of accidents that occur in machine spaces of seafaring vessels in this industry. The main focus of the research was streamlined to fire in machinery spaces. The literature review later concentrated on two of such incidences, they are oil spill and fire events. An investigation was done to assess those factors which actually contribute or are in association to fire outbreak. A content analysis methodology was used to investigate the associative relationships to fire outbreak with the aid of NVivo 9.0 software. The investigation focused on 15 key in-depth reports on machinery space incidences which were uploaded into the software. The results indicate that leakages on hot surfaces were the major causes of fire hazards in seafaring vessels. The results from using this methodology also highlighted two more fire hazards that were not so apparent in previous studies. They are generator fire and compressors fire. The results supported other studies about leakages on hot surfaces as a major contributor, but also clearly show that there are other hazardous factors of fire in machinery spaces that require further investigation

A synthesis of logic and bio-inspired techniques in the design of dependable systems

Much of the development of model-based design and dependability analysis in the design of dependable systems, including software intensive systems, can be attributed to the application of advances in formal logic and its application to fault forecasting and verification of systems. In parallel, work on bio-inspired technologies has shown potential for the evolutionary design of engineering systems via automated exploration of potentially large design spaces. We have not yet seen the emergence of a design paradigm that effectively combines these two techniques, schematically founded on the two pillars of formal logic and biology, from the early stages of, and throughout, the design lifecycle. Such a design paradigm would apply these techniques synergistically and systematically to enable optimal refinement of new designs which can be driven effectively by dependability requirements. The paper sketches such a model-centric paradigm for the design of dependable systems, presented in the scope of the HiP-HOPS tool and technique, that brings these technologies together to realise their combined potential benefits. The paper begins by identifying current challenges in model-based safety assessment and then overviews the use of meta-heuristics at various stages of the design lifecycle covering topics that span from allocation of dependability requirements, through dependability analysis, to multi-objective optimisation of system architectures and maintenance schedules