How much do organizations plan for a positive safety culture? Introducing the Aviation Academy Safety Culture Prerequisites (AVAC-SCP) Tool

Safety culture has been a topic of discussion in safety literature in the past three decades. Since its first mentioning after the Chernobyl accident much have been written about what fosters a positive safety culture within organizations. The Aviation Academy of the Amsterdam University of Applied Sciences conducted a literature review into safety culture development guidelines and identified a list of 37 prerequisites for safety culture development. Existing safety culture assessment tools target to measure the subjective perceptions of the workforce without examining the parameters affecting safety culture. Thus, they cannot be used to provide the organisations with guidance on the action points for improving their safety culture. To accommodate the need for practical guidance to companies, and as part of four-year research into Aviation Safety Metrics, the Aviation Academy has developed the AVAC-SCP tool, which aims to shed light on the planning and implementation for developing a safety culture. Knowledge experts and companies have reviewed the tool as a means to meet various criteria referred to literature with regards to safety metrics. The tool will enable organisations to derive actionable points from the results. A scoring method is included in the AVAC-SCP in order provide organizations with the ability to monitor their performance over time, compare between departments, and prioritize changes. The concept, academic background, reviews and other characteristics of the AVAC-SCP are discussed

Five novel metrics to support safety improvements in current and future air transport

In the frame of the 4-years research project "Measuring Safety in Aviation - Developing Metrics for Safety Management Systems", an initial literature review concluded that current safety metrics are not standardized and their validity lies more on reasoning rather than empirical evidence. Next, surveys across aviation companies showed that different metrics are used by various organizations, and indicated a lack of consistent monotonic relationships of SMS processes, operational activity and demographic data with safety outcomes. The researchers with the support and reviews from academic and research institutions, subject matter experts and through pilot studies in aviation companies have developed a set of five new safety metrics. These metrics are based on the premise that the greater the gap between system design (Work-as-Imagined; Wal) and operation (Wark-as-Done; WaD), the lower the safety/system performance. The metrics regard: (1) assessment of Safety Management Systems by addressing their design, operation, quality of delivery and internal and external interfaces and dependencies, (2) planning and implementation of safety culture prerequisites, (3) system complexity and coupling, (4) measurement of risk control effectiveness, and (5) utilization of resources. The metrics mentioned above operationalize and correspond to suggestions from academic literature, challenges in professional practice, depictions of system structures, and consideration of "soft" organizational aspects. The particular metrics meet the requirements for accuracy, construct, content and face validity and they can be used jointly or separately to offer to organizations insights into various aspects of their systems

Complexity of socio-technical systems: Concept for a uniform metric

For five decades complex socio-technical systems have been studied in an attempt to understand and prevent the occurrence of accidents. In this paper, the authors define a concept for a System Complexity metric, comprising the total of all direct interactions between the system elements and the tools the controller has to control the system. Subsequently, the human performance of the operator is taken into account to arrive at the Perceived System Complexity. Finally a hypothesis for a relation between the dynamic actually perceived system complexity, and the occurrence of incidents is postulated, which are still to be proven in practice

Safety metrics based on utilisation of resources

The objective of the study described in this paper is to define safety metrics that are based on the utilisation of resources. The background of this research is a specific need of the aviation industry where small and medium-sized enterprises lack large amounts of safety-related data to measure and demonstrate their safety performance proactively. The research department of the Aviation Academy has initiated a 4-year study, which will test the possibility to develop new safety indicators that will be able to represent safety levels proactively without the benefit of large data sets. The research team has reviewed the academic and professional literature about safety performance indicators and has performed surveys into 13 companies in order to explore what, how, and why safety performance indicators are used and whether there is a statistically linear relation between SMS process metrics and safety outcomes. The preliminary results showed that companies do not use data from all SMS processes in the development of safety performance indicators, they do not ground the selection of indicators on specific criteria, they implement SMS process in different ways, but they are eager to use alternative metrics, including ones potentially to be derived on the basis of contemporary safety models and views. As part of the development of alternative safety metrics, safety performance indicators were defined that are based on the difference between required resources and available resources. Resources are people, time, equipment and budget. This work is inspired by the general notion that a large gap between ‘work as imagined’ and ‘work as done’ has a negative influence on the level of safety. Work as imagined in this context is represented by available resources and work as done by required resources. The metrics were defined by a combination of literature research and semi-structured interviews with operational practitioners in the aviation industry. The suitability of the metrics will subsequently be tested in pilot studies within the aviation industry

Effectiveness of risk controls as indicator of safety performance

The objective of the study described in this paper is to define safety metrics that are based on the effectiveness of risk controls. Service providers define and implement such risk controls in order to prevent hazards developing into an accident. The background of this research is a specific need of the aviation industry where small and medium-sized enterprises lack large amounts of safety-related data to measure and demonstrate their safety performance proactively. The research department of the Aviation Academy has initiated a 4-year study, which will test the possibility to develop new safety indicators that will be able to represent safety levels proactively without the benefit of large data sets. As part of the development of alternative safety metrics, safety performance indicators were defined that are based on the effectiveness of risk controls. ICAO (2013) defines a risk control as “a defence with specific mitigation actions, preventive controls or recovery measures put in place to prevent the realization of a hazard or its escalation into an undesirable consequence”. Examples of risk controls are procedures, education and training, a piece of equipment etc. It is crucial for service providers to determine whether the introduced risk controls are indeed effective in reducing the targeted risk. ICAO (2013) describes the effectiveness of risk control as "the extent to which the risk control reduces or eliminates the safety risks”, but does not provide guidance on how to measure the effectiveness of risk control. In this study, a generic metrics for the effectiveness of risk controls based on their effectiveness was developed. The definition of the indicators allows, for each risk control, derivation of specific indicators based on the generic metrics. The suitability of the metrics will subsequently be tested in pilot studies within the aviation industry

The AVAC-SMS metric for the self-assessment of maturity of aviation safety management systems

This paper introduces the AVAC-SMS maturity metric and its accompanying tool which were developed in the frame of a research project with the aim to suggest new safety metrics, especially for Small-Medium Enterprises (SMEs). The metric is based on the ICAO Safety Management Manual, it was designed by applying the Systems-Theoretic Process Analysis (STPA) technique and it was reviewed by companies, authorities and field experts. It can be used to assess the institutionalisation, capability and effectiveness of an aviation SMS by following a systematic approach that employs the use of information from the safety department, managers and employees of an organisation. The AVAC-SMS maturity metric is uniform for the aviation sector, customisable to the size and complexity of the organisation, and results in numerical scores that can be used to monitor SMS maturity levels over time or perform benchmarking among companies

Exploring the diversity in safety measurement practices: Empirical results from aviation

A literature review, which was conducted during the research project “Measuring Safety in Aviation – Developing Metrics for Safety Management Systems”, identified several problems and challenges regarding safety performance metrics in aviation. The findings from this review were used to create a framework for interviewing 13 companies in order to explore how safety performance is measured in the industry. The results from the surveys showed a wide variety of approaches for assessing the level of safety. The companies encounter and/or recognise problematic areas in practice when implementing their safety management. The findings from the literature review are partially confirmed and it seems that the current ways of measuring safety performance are not as straight forward as it might be assumed. Further research is recommended to explore alternative methods for measuring aviation safety performance

Results from Surveys about Existing Aviation Safety Metrics

In September 2015, the Aviation Academy of the Amsterdam University of Applied Sciences initiated the research project entitled “Measuring Safety in Aviation – Developing Metrics for Safety Management Systems”. The project responds to specific needs of the aviation industry: Small and Medium Enterprises (SME) lack large amounts of safety related data in order to measure and demonstrate their safety performance; large companies might obtain abundance of data, but they need safety metrics of better quality. Therefore, the aim of the project is to identify ways to measure safety in scientifically rigorous, meaningful and practical ways without the benefit of large amounts of data (Aviation Academy, 2014). The research phases are: examination of validity of current safety metrics, exploration of new suitable safety metrics based on existing and alternative models and approaches to safety, generation and validation of a short list of suitable safety metrics, and translation of this knowledge into a web-based dashboard for the industry. The project will last until August 2019, is co-funded by the Nationaal Regieorgaan Praktijkgericht Onderzoek SIA (SIA, 2015), and is executed by a team of researchers from the Aviation Academy in collaboration with a consortium of industry, academia and authorities’ representatives..

RAAK PRO Project: Measuring Safety in Aviation: Deliverable: Concept for the Design of New Metrics

Following the completion of the 1st phase of the RAAK PRO project Aviation Safety Metrics, during which the researchers mapped the current practice in safety metrics and explored the validity of monotonic relationships of SMS, activity and demographic metrics with safety outcomes, this report presents the concept for the design of new metrics. Those metrics will be based on the hypothesis that the greater the gap between Work-As-Imagined and Work-As-Done the lower the safety performance, and they correspond to a set of references from academic literature, challenges in professional practice, depiction of system structure, and consideration of “soft” organizational aspects. Along with the design of the alternative metrics, this report explains the respective concepts referred in the literature but excluded from the current research, as well as the process and possible difficulties in ensuring various validity types of the new metrics