Predicting Pilot Misperception of Runway Excursion Risk Through Machine Learning Algorithms of Recorded Flight Data

The research used predictive models to determine pilot misperception of runway excursion risk associated with unstable approaches. The Federal Aviation Administration defined runway excursion as a veer-off or overrun of the runway surface. The Federal Aviation Administration also defined a stable approach as an aircraft meeting the following criteria: (a) on target approach airspeed, (b) correct attitude, (c) landing configuration, (d) nominal descent angle/rate, and (e) on a straight flight path to the runway touchdown zone. Continuing an unstable approach to landing was defined as Unstable Approach Risk Misperception in this research. A review of the literature revealed that an unstable approach followed by the failure to execute a rejected landing was a common contributing factor in runway excursions. Flight Data Recorder data were archived and made available by the National Aeronautics and Space Administration for public use. These data were collected over a four-year period from the flight data recorders of a fleet of 35 regional jets operating in the National Airspace System. The archived data were processed and explored for evidence of unstable approaches and to determine whether or not a rejected landing was executed. Once identified, those data revealing evidence of unstable approaches were processed for the purposes of building predictive models. SAS™ Enterprise MinerR was used to explore the data, as well as to build and assess predictive models. The advanced machine learning algorithms utilized included: (a) support vector machine, (b) random forest, (c) gradient boosting, (d) decision tree, (e) logistic regression, and (f) neural network. The models were evaluated and compared to determine the best prediction model. Based on the model comparison, the decision tree model was determined to have the highest predictive value. The Flight Data Recorder data were then analyzed to determine predictive accuracy of the target variable and to determine important predictors of the target variable, Unstable Approach Risk Misperception. Results of the study indicated that the predictive accuracy of the best performing model, decision tree, was 99%. Findings indicated that six variables stood out in the prediction of Unstable Approach Risk Misperception: (1) glideslope deviation, (2) selected approach speed deviation (3) localizer deviation, (4) flaps not extended, (5) drift angle, and (6) approach speed deviation. These variables were listed in order of importance based on results of the decision tree predictive model analysis. The results of the study are of interest to aviation researchers as well as airline pilot training managers. It is suggested that the ability to predict the probability of pilot misperception of runway excursion risk could influence the development of new pilot simulator training scenarios and strategies. The research aids avionics providers in the development of predictive runway excursion alerting display technologies

Identifying Opportunities of Tracking Major Human Factors Risks through Flight Data Monitoring

It is widely believed that human factors risks contribute to more than half of the aviation accidents (Shappell et al., 2007). Thus, aviation safety risk identification, and in particular human factor risk identification, is one of the crucial components in today’s aviation safety management systems. There is a need to identify examples of major human factors risks in recent years in the industry and track the exposure of these risks in an individual airline’s own operation routinely. Flight Data Monitoring (FDM) is a systematic and proactive program (Civil Aviation Authority, 2013), which aims to improve aviation safety by collecting and analyzing digital flight data. Since the flight data is able to provide objective and up-to-date information of routine flight performance, this program has the potential to contribute to the identification of the existence and status of the some major human factors risks in airlines’ routine operations. However, current FDM data is not widely used to proactively monitor and track human factors issues. This thesis presents an initial analysis of the potential of using FDM data for identifying and tracking human factors risks. As a first step, in order to obtain insights into the current key human factors risks in the North American commercial aviation operations, the Human Factors Analysis and Classification System (HFACS) was used to categorize 267 accident and incident final reports from 2006 to 2010. Semi-structured interviews have also been conducted to identify and understand major and projected human factors issues from the airline operators’ perspectives. By combining the results obtained from two methods, examples of perceived major human factors risks in current operations are determined. The current top risks of concern include Standard Operational Procedures (SOPs) noncompliance, fatigue, distraction, communication issues, inadequate situation awareness, training issues, pressure, and high workload. In order to assess the potential opportunities of tracking these top human factors risks in airline operations through FDM, current FDM process, applications, best practices and recorded flight parameters were studied. A literature review, field observations, and interviews with experienced safety investigators and flight data analysts were conducted. Models of general FDM process, event setting process, and daily review workflow are presented and human performance related flight parameters are categorized into seven classes. Finally, opportunities and two potential approaches of using FDM to track some major human factors risks have been identified. These two approaches have the potential of being embedded into current FDM processes are 1) setting up new human factors events (HF events) and 2) conducting specific human factors focused studies (HF studies). Implementation examples demonstrating how these two approaches can be applied to track some major human factors, including automation confusion, high workload, and on time pressure are provided. For example, a proposed “automation mode confusion event” is recommended especially for new type of aircrafts (e.g., the Boeing 787), where new pilots are interacting with new operational environments. Applications of the potential approaches, recommendations to commercial airlines, and future work of this study are also discussed

Anomaly detection in airline routine operations using flight data recorder data

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Aeronautics and Astronautics, 2013.This thesis was scanned as part of an electronic thesis pilot project.Cataloged from PDF version of thesis.Includes bibliographical references (p. 141-145).In order to improve safety in current air carrier operations, there is a growing emphasis on proactive safety management systems. These systems identify and mitigate risks before accidents occur. This thesis develops a new anomaly detection approach using routine operational data to support proactive safety management. The research applies cluster analysis to detect abnormal flights based on Flight Data Recorder (FDR) data. Results from cluster analysis are provided to domain experts to verify operational significance of such anomalies and associated safety hazards. Compared with existing methods, the cluster-based approach is capable of identifying new types of anomalies that were previously unaccounted for. It can help airlines detect early signs of performance deviation, identify safety degradation, deploy predictive maintenance, and train staff accordingly. The first part of the detection approach employs data-mining algorithms to identify flights of interest from FDR data. These data are transformed into a high-dimensional space for cluster analysis, where normal patterns are identified in clusters while anomalies are detected as outliers. Two cluster-based anomaly detection algorithms were developed to explore different transformation techniques: ClusterAD-Flight and ClusterAD-Data Sample. The second part of the detection approach is domain expert review. The review process is to determine whether detected anomalies are operationally significant and whether they represent safety risks. Several data visualization tools were developed to support the review process which can be otherwise labor-intensive: the Flight Parameter Plots can present raw FDR data in informative graphics; The Flight Abnormality Visualization can help domain experts quickly locate the source of such anomalies. A number of evaluation studies were conducted using airline FDR data. ClusterAD-Flight and ClusterAD-Data Sample were compared with Exceedance Detection, the current method in use by airlines, and MKAD, another anomaly detection algorithm developed at NASA, using a dataset of 25519 A320 flights. An evaluation of the entire detection approach was conducted with domain experts using a dataset of 10,528 A320 flights. Results showed that both cluster-based detection algorithms were able to identify operationally significant anomalies that beyond the capacities of current methods. Also, domain experts confirmed that the data visualization tools were effective in supporting the review process.by Lishuai Li.Ph.D

Using flight data in Bayesian networks and other methods to quantify airline operational risks.

The risk assessment methods used in airline operations are usually qualitative rather than quantitative, despite the routine collection of vast amounts of safety data through programmes such as flight data monitoring (FDM). The overall objective of this research is to exploit airborne recorded flight data to provide enhanced operational safety knowledge and quantitative risk assessments. Runway veer-off at landing, accounting for over 10% of air transport incidents and accidents, is used as an example risk. Literature on FDM, risk assessment and veer-off accidents is reviewed, leading to the identification of three potential areas for further examination: variability in operational parameters as a measure of risk; measures of workload derived from flight data as a measure of risk; and Bayesian networks. Methods relating to variability and workload are briefly explored and preliminary results are presented, before the main methods of the thesis relating to Bayesian networks are introduced. The literature shows that Bayesian networks are a suitable method for quantifying risk and a causal network for lateral deviation at landing is developed based on accident investigation data. Flight data from over 300,000 flights is used to provide empirical probabilities for causal factors and data for some causal factors is modelled to estimate the probabilities of extreme events. As an alternative to predefining the Bayesian network structure from accident data, a series of networks are learnt from flight data and an assessment is made of the performance of different learning algorithms, such as Bayesian Search and Greedy Thick Thinning. Finally, a network with parameters and structure learnt from flight data is adapted to incorporate causal knowledge from accident data, and the performance of the resulting “combined” network is assessed. All three types of network were able to make use of flight data to calculate relative probabilities of a lateral deviation event, given different scenarios of causal factors present, and for different airports, however the “combined” approach is preferred due to the relative ease of running scenarios for different airports and the avoidance of the lengthy process of modelling data for causal factor nodes. The preferred method provides airlines with a practicable way to use their existing flight data to quantify operational risks. The resulting quantitative risk assessments could be used to provide pilots with enhanced pre-flight briefings and provide airlines with up-to-date risk information of operations to different airports, and enhanced safety oversight.PhD in Transport System

Machine Learning Approach for the Evaluation of Degraded Wheel Braking Performance on Contaminated Runways

One of the most challenging issues in the airline community has been the safe operations of aircraft during the landing phase of flight. Several entities agree that runway excursion is the most frequent type of landing incident or accident, and studies on this topic have shown that a significant contributor is the condition of the runway surface at the time of landing, as well as the potential for degraded braking performance

12th EASN International Conference on "Innovation in Aviation & Space for opening New Horizons"

Epoxy resins show a combination of thermal stability, good mechanical performance, and durability, which make these materials suitable for many applications in the Aerospace industry. Different types of curing agents can be utilized for curing epoxy systems. The use of aliphatic amines as curing agent is preferable over the toxic aromatic ones, though their incorporation increases the flammability of the resin. Recently, we have developed different hybrid strategies, where the sol-gel technique has been exploited in combination with two DOPO-based flame retardants and other synergists or the use of humic acid and ammonium polyphosphate to achieve non-dripping V-0 classification in UL 94 vertical flame spread tests, with low phosphorous loadings (e.g., 1-2 wt%). These strategies improved the flame retardancy of the epoxy matrix, without any detrimental impact on the mechanical and thermal properties of the composites. Finally, the formation of a hybrid silica-epoxy network accounted for the establishment of tailored interphases, due to a better dispersion of more polar additives in the hydrophobic resin

Runway Safety Improvements Through a Data Driven Approach for Risk Flight Prediction and Simulation

Runway overrun is one of the most frequently occurring flight accident types threatening the safety of aviation. Sensors have been improved with recent technological advancements and allow data collection during flights. The recorded data helps to better identify the characteristics of runway overruns. The improved technological capabilities and the growing air traffic led to increased momentum for reducing flight risk using artificial intelligence. Discussions on incorporating artificial intelligence to enhance flight safety are timely and critical. Using artificial intelligence, we may be able to develop the tools we need to better identify runway overrun risk and increase awareness of runway overruns. This work seeks to increase attitude, skill, and knowledge (ASK) of runway overrun risks by predicting the flight states near touchdown and simulating the flight exposed to runway overrun precursors. To achieve this, the methodology develops a prediction model and a simulation model. During the flight training process, the prediction model is used in flight to identify potential risks and the simulation model is used post-flight to review the flight behavior. The prediction model identifies potential risks by predicting flight parameters that best characterize the landing performance during the final approach phase. The predicted flight parameters are used to alert the pilots for any runway overrun precursors that may pose a threat. The predictions and alerts are made when thresholds of various flight parameters are exceeded. The flight simulation model simulates the final approach trajectory with an emphasis on capturing the effect wind has on the aircraft. The focus is on the wind since the wind is a relatively significant factor during the final approach; typically, the aircraft is stabilized during the final approach. The flight simulation is used to quickly assess the differences between fight patterns that have triggered overrun precursors and normal flights with no abnormalities. The differences are crucial in learning how to mitigate adverse flight conditions. Both of the models are created with neural network models. The main challenges of developing a neural network model are the unique assignment of each model design space and the size of a model design space. A model design space is unique to each problem and cannot accommodate multiple problems. A model design space can also be significantly large depending on the depth of the model. Therefore, a hyperparameter optimization algorithm is investigated and used to design the data and model structures to best characterize the aircraft behavior during the final approach. A series of experiments are performed to observe how the model accuracy change with different data pre-processing methods for the prediction model and different neural network models for the simulation model. The data pre-processing methods include indexing the data by different frequencies, by different window sizes, and data clustering. The neural network models include simple Recurrent Neural Networks, Gated Recurrent Units, Long Short Term Memory, and Neural Network Autoregressive with Exogenous Input. Another series of experiments are performed to evaluate the robustness of these models to adverse wind and flare. This is because different wind conditions and flares represent controls that the models need to map to the predicted flight states. The most robust models are then used to identify significant features for the prediction model and the feasible control space for the simulation model. The outcomes of the most robust models are also mapped to the required landing distance metric so that the results of the prediction and simulation are easily read. Then, the methodology is demonstrated with a sample flight exposed to an overrun precursor, and high approach speed, to show how the models can potentially increase attitude, skill, and knowledge of runway overrun risk. The main contribution of this work is on evaluating the accuracy and robustness of prediction and simulation models trained using Flight Operational Quality Assurance (FOQA) data. Unlike many studies that focused on optimizing the model structures to create the two models, this work optimized both data and model structures to ensure that the data well capture the dynamics of the aircraft it represents. To achieve this, this work introduced a hybrid genetic algorithm that combines the benefits of conventional and quantum-inspired genetic algorithms to quickly converge to an optimal configuration while exploring the design space. With the optimized model, this work identified the data features, from the final approach, with a higher contribution to predicting airspeed, vertical speed, and pitch angle near touchdown. The top contributing features are altitude, angle of attack, core rpm, and air speeds. For both the prediction and the simulation models, this study goes through the impact of various data preprocessing methods on the accuracy of the two models. The results may help future studies identify the right data preprocessing methods for their work. Another contribution from this work is on evaluating how flight control and wind affect both the prediction and the simulation models. This is achieved by mapping the model accuracy at various levels of control surface deflection, wind speeds, and wind direction change. The results saw fairly consistent prediction and simulation accuracy at different levels of control surface deflection and wind conditions. This showed that the neural network-based models are effective in creating robust prediction and simulation models of aircraft during the final approach. The results also showed that data frequency has a significant impact on the prediction and simulation accuracy so it is important to have sufficient data to train the models in the condition that the models will be used. The final contribution of this work is on demonstrating how the prediction and the simulation models can be used to increase awareness of runway overrun.Ph.D

A multiple objective optimization approach to quality control

The use of product quality as the performance criteria for manufacturing system control is explored. The goal in manufacturing, for economic reasons, is to optimize product quality. The problem is that since quality is a rather nebulous product characteristic, there is seldom an analytic function that can be used as a measure. Therefore standard control approaches, such as optimal control, cannot readily be applied. A second problem with optimizing product quality is that it is typically measured along many dimensions: there are many apsects of quality which must be optimized simultaneously. Very often these different aspects are incommensurate and competing. The concept of optimality must now include accepting tradeoffs among the different quality characteristics. These problems are addressed using multiple objective optimization. It is shown that the quality control problem can be defined as a multiple objective optimization problem. A controller structure is defined using this as the basis. Then, an algorithm is presented which can be used by an operator to interactively find the best operating point. Essentially, the algorithm uses process data to provide the operator with two pieces of information: (1) if it is possible to simultaneously improve all quality criteria, then determine what changes to the process input or controller parameters should be made to do this; and (2) if it is not possible to improve all criteria, and the current operating point is not a desirable one, select a criteria in which a tradeoff should be made, and make input changes to improve all other criteria. The process is not operating at an optimal point in any sense if no tradeoff has to be made to move to a new operating point. This algorithm ensures that operating points are optimal in some sense and provides the operator with information about tradeoffs when seeking the best operating point. The multiobjective algorithm was implemented in two different injection molding scenarios: tuning of process controllers to meet specified performance objectives and tuning of process inputs to meet specified quality objectives. Five case studies are presented

Naval Research Program 2019 Annual Report

NPS NRP Annual ReportThe Naval Postgraduate School (NPS) Naval Research Program (NRP) is funded by the Chief of Naval Operations and supports research projects for the Navy and Marine Corps. The NPS NRP serves as a launch-point for new initiatives which posture naval forces to meet current and future operational warfighter challenges. NRP research projects are led by individual research teams that conduct research and through which NPS expertise is developed and maintained. The primary mechanism for obtaining NPS NRP support is through participation at NPS Naval Research Working Group (NRWG) meetings that bring together fleet topic sponsors, NPS faculty members, and students to discuss potential research topics and initiatives.Chief of Naval Operations (CNO)This research is supported by funding from the Naval Postgraduate School, Naval Research Program (PE 0605853N/2098). https://nps.edu/nrpChief of Naval Operations (CNO)Approved for public release. Distribution is unlimited.