Active Learning with Rationales for Identifying Operationally Significant Anomalies in Aviation

A major focus of the commercial aviation community is discovery of unknown safety events in flight operations data. Data-driven unsupervised anomaly detection methods are better at capturing unknown safety events compared to rule-based methods which only look for known violations. However, not all statistical anomalies that are discovered by these unsupervised anomaly detection methods are operationally significant (e.g., represent a safety concern). Subject Matter Experts (SMEs) have to spend significant time reviewing these statistical anomalies individually to identify a few operationally significant ones. In this paper we propose an active learning algorithm that incorporates SME feedback in the form of rationales to build a classifier that can distinguish between uninteresting and operationally significant anomalies. Experimental evaluation on real aviation data shows that our approach improves detection of operationally significant events by as much as 75% compared to the state-of-the-art. The learnt classifier also generalizes well to additional validation data sets

Ask-the-expert: Active Learning Based Knowledge Discovery Using the Expert

Often the manual review of large data sets, either for purposes of labeling unlabeled instances or for classifying meaningful results from uninteresting (but statistically significant) ones is extremely resource intensive, especially in terms of subject matter expert (SME) time. Use of active learning has been shown to diminish this review time significantly. However, since active learning is an iterative process of learning a classifier based on a small number of SME-provided labels at each iteration, the lack of an enabling tool can hinder the process of adoption of these technologies in real-life, in spite of their labor-saving potential. In this demo we present ASK-the-Expert, an interactive tool that allows SMEs to review instances from a data set and provide labels within a single framework. ASK-the-Expert is powered by an active learning algorithm for training a classifier in the backend. We demonstrate this system in the context of an aviation safety application, but the tool can be adopted to work as a simple review and labeling tool as well, without the use of active learning

Toward Justifiable Trust in Autonomous Systems Incorporating Human Knowledge in Autonomous Systems through Machine Learning

Trust in Autonomous Systems is largely about humans trusting the decisions made by autonomous systems. This trust can be increased through learning from domain experts. In particular, autonomous systems can learn offline from past mission operations before conducting any operations of its own. Additionally, autonomous systems can learn online by obtaining human feedback during operations. We will discuss several classes of machine learning methods and our application of them to autonomous systems. The first class of methods is anomaly detection, which uses operations data to identify examples of anomalous operations. The second class of methods is inverse reinforcement learning, also known as apprenticeship learning, that takes past operations data as input and yields a controller that is able to duplicate the operations described by the data. The third class is active learning, which identifies examples on which the model is most uncertain and requests domain expert feedback

Handbook for New Actors in Space

Driven by Cold War tensions between the US and the Soviet Union, the space race began almost 60 years ago. Each power was racing to accomplish new feats in space and demonstrate its superiority. In 2017, while much remains the same, much has changed. Space actors comprise a wide variety of national and non-governmental entities comprising diverse rationales, goals, and activities. More than 70 states, commercial companies, and international organizations currently operate more than 1,500 satellites in Earth orbit. Driven largely by the commoditization of space technology and the lowering of barriers to participation, the number of space actors is growing. This broadening of space has both advantages and disadvantages. On the positive side, it is leading to greatly increased technological innovations, lower costs, and greater access to the beneficial capabilities and services offered by satellites. However, the accelerated growth in space activities and the influx of new actors has the potential to exacerbate many of the current threats to the long-term sustainable use of space. These threats include on-orbit crowding, radio-frequency interference, and the chances of an incident in space sparking or escalating geopolitical tensions on Earth. Michael K. Simpson, Ph.D. - Executive Director, Secure World Foundatio

The organisational precursors to human automation interaction issues in safety-critical domains: the case of an automated alarm system from the air traffic management domain

Much has been written about the side effects of automation in complex safety-critical domains, such as air traffic management, aviation, nuclear power generation, and healthcare. Here, human factors and safety researchers have long acknowledged that the potential of automation to increase cost-effectiveness, quality of service and safety, is accompanied by undesired side effects or issues in human automation interaction (HAI). Such HAI issues may introduce the potential for increased confusion, uncertainty, and frustration amongst sharp end operators, i.e. the users of automation. These conditions may result in operators to refuse to use the automation, in impaired ability of operators to control the hazardous processes for which they are responsible, and in new, unintended paths to safety failure. The present thesis develops a qualitative framework of the organisational precursors to HAI issues (OPHAII) that can be found in safety-critical domains. Organisational precursors denote those organisational and managerial conditions that, although distant in time and space from the operational environment, may actually influence the quality of HAI found there. Such precursors have been extensively investigated by organisational safety (OS) scholars in relation to the occurrence of accidents and disasters—although not HAI issues. Thus, the framework’s development is motivated by the intent to explore the theoretical gap lying at the intersection between the OS area and the current perspectives on the problem—the human computer interaction (HCI) and the system lifecycle ones. While considering HAI issues as a design problem or a failure in human factors integration and/or safety assurance respectively, both perspectives, in fact, ignore, the organisational roots of the problem. The OPHAII framework was incrementally developed based on three qualitative studies: two successive, historical, case studies coupled with a third corroboratory expert study. The first two studies explored the organisational precursors to a known HAI issue: the nuisance alert problem relative to an automated alarm system from the air traffic management domain. In particular, the first case study investigated retrospectively the organisational response to the nuisance alert problem in the context of the alarm’s implementation and improvement in the US between 1977 and 2006. The second case study has a more contemporary focus, and examined at the organisational response to the same problem within two European Air Navigation Service Providers between 1990 and 2010. The first two studies produced a preliminary version of the framework. The third study corroborated and refined this version by subjecting it to the criticism from a panel of 11 subject matter experts. The resulting framework identifies three classes of organisational precursors: (i) the organisational assumptions driving automation adoption and improvement; (2) the availability of specific organisational capabilities for handling HAI issues; and (3) the control of implementation quality at the boundary between the service provider and the software manufacturer. These precursors advance current understanding of the organisational factors involved in the (successful and problematic) handling of HAI issues within safety-critical service provider organisations. Its dimensions support the view that HAI issues can be seen as and organisational phenomenon—an organisational problem that can be the target of analysis and improvements complementary to those identified by the HCI and the system lifecycle perspectives

Handbook for New Actors in Space

Driven by Cold War tensions between the US and the Soviet Union, the space race began almost 60 years ago. Each power was racing to accomplish new feats in space and demonstrate its superiority. In 2017, while much remains the same, much has changed. Space actors comprise a wide variety of national and non-governmental entities comprising diverse rationales, goals, and activities. More than 70 states, commercial companies, and international organizations currently operate more than 1,500 satellites in Earth orbit. Driven largely by the commoditization of space technology and the lowering of barriers to participation, the number of space actors is growing. This broadening of space has both advantages and disadvantages. On the positive side, it is leading to greatly increased technological innovations, lower costs, and greater access to the beneficial capabilities and services offered by satellites. However, the accelerated growth in space activities and the influx of new actors has the potential to exacerbate many of the current threats to the long-term sustainable use of space. These threats include on-orbit crowding, radio-frequency interference, and the chances of an incident in space sparking or escalating geopolitical tensions on Earth. Michael K. Simpson, Ph.D. - Executive Director, Secure World Foundatio

A DATA-DRIVEN METHODOLOGY TO ANALYZE AIR TRAFFIC MANAGEMENT SYSTEM OPERATIONS WITHIN THE TERMINAL AIRSPACE

Air Traffic Management (ATM) systems are the systems responsible for managing the operations of all aircraft within an airspace. In the past two decades, global modernization efforts have been underway to increase ATM system capacity and efficiency, while maintaining safety. Gaining a comprehensive understanding of both flight-level and airspace-level operations enables ATM system operators, planners, and decision-makers to make better-informed and more robust decisions related to the implementation of future operational concepts. The increased availability of operational data, including widely-accessible ADS-B trajectory data, and advances in modern machine learning techniques provide the basis for offline data-driven methods to be applied to analyze ATM system operations. Further, analysis of ATM system operations of arriving aircraft within the terminal airspace has the highest potential to impact safety, capacity, and efficiency levels due to the highest rate of accidents and incidents occurring during the arrival flight phases. Therefore, motivating this research is the question of how offline data-driven methods may be applied to ADS-B trajectory data to analyze ATM system operations at both the flight and airspace levels for arriving aircraft within the terminal airspace to extract novel insights relevant to ATM system operators, planners, and decision-makers. An offline data-driven methodology to analyze ATM system operations is proposed involving the following three steps: (i) Air Traffic Flow Identification, (ii) Anomaly Detection, and (iii) Airspace-Level Analysis. The proposed methodology is implemented considering ADS-B trajectory data that was extracted, cleaned, processed, and augmented for aircraft arriving at San Francisco International Airport (KSFO) during the full year of 2019 as well as the corresponding extracted and processed ASOS weather data. The Air Traffic Flow Identification step contributes a method to more reliably identify air traffic flows for arriving aircraft trajectories through a novel implementation of the HDBSCAN clustering algorithm with a weighted Euclidean distance function. The Anomaly Detection step contributes the novel distinction between spatial and energy anomalies in ADS-B trajectory data and provides key insights into the relationship between the two types of anomalies. Spatial anomalies are detected leveraging the aforementioned air traffic flow identification method, whereas energy anomalies are detected leveraging the DBSCAN clustering algorithm. Finally, the Airspace-Level Analysis step contributes a novel method to identify operational patterns and characterize operational states of aircraft arriving within the terminal airspace during specified time intervals leveraging the UMAP dimensionality reduction technique and DBSCAN clustering algorithm. Additionally, the ability to predict, in advance, a time interval’s operational pattern using metrics derived from the ASOS weather data as input and training a gradient-boosted decision tree (XGBoost) algorithm is provided.Ph.D