Intelligent cockpit: eye tracking integration to enhance the pilot-aircraft interaction

In this research, we use eye tracking to monitor the attentional behavior of pilots in the cockpit. We built a cockpit monitoring database that serves as a reference for real-time assessment of the pilot’s monitoring strategies, based on numerous flight simulator sessions with eye-tracking recordings. Eye tracking may also be employed as a passive input for assistive system, future studies will also explore the possibility to adapt the notifications’ modality using gaze

Speech Interfaces and Pilot Performance: A Meta-Analysis

As the aviation industry modernizes, new technology and interfaces must support growing aircraft complexity without increasing pilot workload. Natural language processing presents just such a simple and intuitive interface, yet the performance implications for use by pilots remain unknown. A meta-analysis was conducted to understand performance effects of using speech and voice interfaces in a series of pilot task analogs. The inclusion criteria selected studies that involved participants performing a demanding primary task, such as driving, while interacting with a vehicle system to enter numbers, dial radios, or enter a navigation destination. Compared to manual system interfaces, voice interfaces reduced subjective workload and inattention. Time to complete interactions with the interface appear to vary based upon the individual system, with systems that require prompts or provide confirmation taking the longest. Implications of introducing natural language systems in aviation are discussed and recommendations for future study are provided

Feature selection model based on EEG signals for assessing the cognitive workload in drivers

In recent years, research has focused on generating mechanisms to assess the levels of subjects’ cognitive workload when performing various activities that demand high concentration levels, such as driving a vehicle. These mechanisms have implemented several tools for analyzing the cognitive workload, and electroencephalographic (EEG) signals have been most frequently used due to their high precision. However, one of the main challenges in implementing the EEG signals is finding appropriate information for identifying cognitive states. Here, we present a new feature selection model for pattern recognition using information from EEG signals based on machine learning techniques called GALoRIS. GALoRIS combines Genetic Algorithms and Logistic Regression to create a new fitness function that identifies and selects the critical EEG features that contribute to recognizing high and low cognitive workloads and structures a new dataset capable of optimizing the model’s predictive process. We found that GALoRIS identifies data related to high and low cognitive workloads of subjects while driving a vehicle using information extracted from multiple EEG signals, reducing the original dataset by more than 50% and maximizing the model’s predictive capacity, achieving a precision rate greater than 90%.This work has been funded by the Ministry of Science, Innovation and Universities of Spain under grant number TRA2016-77012-RPeer ReviewedPostprint (published version

Tracing Pilots’ Situation Assessment by Neuroadaptive Cognitive Modeling

This study presents the integration of a passive brain-computer interface (pBCI) and cognitive modeling as a method to trace pilots’ perception and processing of auditory alerts and messages during operations. Missing alerts on the flight deck can result in out-of-the-loop problems that can lead to accidents. By tracing pilots’ perception and responses to alerts, cognitive assistance can be provided based on individual needs to ensure they maintain adequate situation awareness. Data from 24 participating aircrew in a simulated flight study that included multiple alerts and air traffic control messages in single pilot setup are presented. A classifier was trained to identify pilots’ neurophysiological reactions to alerts and messages from participants’ electroencephalogram (EEG). A neuroadaptive ACT-R model using EEG data was compared to a conventional normative model regarding accuracy in representing individual pilots. Results show that passive BCI can distinguish between alerts that are processed by the pilot as task-relevant or irrelevant in the cockpit based on the recorded EEG. The neuroadaptive model’s integration of this data resulted in significantly higher performance of 87% overall accuracy in representing individual pilots’ responses to alerts and messages compared to 72% accuracy of a normative model that did not consider EEG data. We conclude that neuroadaptive technology allows for implicit measurement and tracing of pilots’ perception and processing of alerts on the flight deck. Careful handling of uncertainties inherent to passive BCI and cognitive modeling shows how the representation of pilot cognitive states can be improved iteratively for providing assistance.TU Berlin, Open-Access-Mittel – 202

Development of cognitive workload models to detect driving impairment

Tesi redactada en castellàDriving a vehicle is a complex activity exposed to continuous changes such as speed limits and vehicular traffic. Drivers require a high degree of concentration when performing this activity, increasing the amount of mental demand known as cognitive workload, causing vehicular accidents to the minimum negligence. In fact, human error is the leading contributing factor in over 90% of road accidents. In recent years, the subjects' cognitive workload levels while driving a vehicle have been predicted using subjective and vehicle performance tools. Other research has emphasized the use and analysis of physiological information, where electroencephalographic (EEG) signals are the most used to identify cognitive states due to their high precision. Although significant progress has been made in this area, these investigations have been based on traditional techniques or data analysis from a specific source due to the information's complexity. A new trend has been opened in the study of the internal behavior of subjects by implementing machine learning techniques to analyze information from various sources. However, there are still several challenges to face in this new line of research. This doctoral thesis presents a new model to predict the states of low and high cognitive workload of subjects when facing scenarios of driving a vehicle called GALoRSI-SVMRBF (Genetic Algorithms and Logistic Regression for the Structuring of Information-Support Vector Machine with Radial Basis Function Kernel). GALoRSI-SVMRBF is developed using machine learning algorithms based on information from EEG signals. Also, the information collected from NASA-TLX, instant online self-assessment and the error rate measure are implemented in the model. First, GALoRSI-SVMRBF proposes a new method for pattern recognition based on feature selection that combines statistical tests, genetic algorithms, and logistic regression. This method consists mainly of selecting an EEG dataset and exploring the information to identify the key features that recognize cognitive states. The selected data are defined as an index for pattern recognition and used to structure a new dataset capable of optimizing the model's learning and classification process. Second, the methodology and development of a classifier for the prediction model are presented, implementing machine learning algorithms. The classifier is developed mainly in two phases, defined as training and testing. Once the prediction model has been developed, this thesis presents the validation phase of GALoRSI-SVMRBF. The validation consists of evaluating the model's adaptability to new datasets, maintaining a high prediction rate. Finally, an analysis of the performance of GALoRSI-SVMRBF is presented. The objective is to know the model's scope and limitations, evaluating various performance metrics to find the optimal configuration for GALoRSI-SVMRBF. We found that GALoRSI-SVMRBF successfully predicts low and high cognitive workload of subjects while driving a vehicle. In general, it is observed that the model uses the information extracted from multiple EEG signals, reducing the original dataset by more than 50%, maximizing its predictive capacity, achieving a precision rate of >90% in the classification of the information. During this thesis, the experiments showed that obtaining a high percentage of prediction depends on several factors, from applying a useful collection technique data until the last step of the prediction model.La conducción de un vehículo es una actividad compleja que está expuesta a demandas que cambian continuamente por diferentes factores, tales como, el límite de velocidad, obstáculos en la vía, tráfico vehicular, entre otros. Al desempeñar esta actividad, los conductores requieren un alto grado de concentración incrementando la cantidad de demanda mental conocida como carga. En los últimos años, se han propuesto mecanismos para monitorear y/o predecir los niveles de carga cognitiva de los sujetos al conducir un vehículo, centrándose en el uso de herramientas subjetivas y de rendimiento vehicular. Otras investigaciones, han enfatizado en el uso y análisis de la información fisiológica, siendo las señales electroencefalográficas (EEG) las más utilizadas para identificar los estados cognitivos por su alta precisión. A pesar del gran avance realizado, estas investigaciones se han basado en técnicas tradicionales o en el análisis de la información proveniente de fuentes específicas para identificar el estado interno del sujeto, obteniendo modelos sobreentrenados o robustos, incrementando el tiempo de análisis afectando el desempeño del modelo. En esta tesis doctoral se presenta un nuevo modelo para predecir los estados de baja y alta carga cognitiva de los sujetos al enfrentarse a escenarios de la conducción de un vehículo denominado GALoRSI-SVMRBF (Genetic Algorithms and Logistic Regression for the Structuring of Information-Support Vector Machine with Radial Basis Function Kernel). GALoRSI-SVMRBF fue desarrollado utilizando los algoritmos de aprendizaje automático y técnicas estadísticas basado en la información proveniente de las señales EEG. Primero, GALoRSI-SVMRBF crea una base de datos extrayendo las características que serán utilizadas en el modelo a través de técnicas estadísticas. Posteriormente, propone un nuevo método para el reconocimiento de patrones basado en la selección de características que combina pruebas estadísticas, algoritmos genéticos y regresión logística. Este método consiste principalmente en seleccionar un conjunto de datos EEG y explorar la combinación de la información para identificar las características claves que contribuyan al reconocimiento de dos estados cognitivos. Después, la información seleccionada es definida como un índice para el reconocimiento de patrones y utilizada para estructurar un nuevo conjunto de datos que soporta información de uno o múltiples canales para optimizar el proceso de aprendizaje y clasificación del modelo. Por último, es desarrollado el clasificador del modelo de predicciones el cual consiste en dos etapas definidas como entrenamiento y prueba. Nosotros encontramos que GALoRSI-SVMRBF predice de manera exitosa la carga cognitiva baja y alta de los sujetos durante la conducción de un vehículo. En general, se observó que el modelo utiliza la información extraída de una o múltiples señales EEG y logrando una tasa de precisión >90% en la clasificación de la informaciónPostprint (published version

Next generation flight management systems for manned and unmanned aircraft operations - automated separation assurance and collision avoidance functionalities

The demand for improved safety, efficiency and dynamic demand-capacity balancing due to the rapid growth of the aviation sector and the increasing proliferation of Unmanned Aircraft Systems (UAS) in different classes of airspace pose significant challenges to avionics system developers. The design of Next Generation Flight Management Systems (NG-FMS) for manned and unmanned aircraft operations is performed by addressing the challenges identified by various Air Traffic Management (ATM) modernisation programmes and UAS Traffic Management (UTM) system initiatives. In particular, this research focusses on introducing automated Separation Assurance and Collision Avoidance (SA&CA) functionalities (mathematical models) in the NG-FMS. The innovative NG-FMS is also capable of supporting automated negotiation and validation of 4-Dimensional Trajectory (4DT) intents in coordination with novel ground-based Next Generation Air Traffic Management (NG-ATM) systems. One of the key research contributions is the development of a unified method for cooperative and non-cooperative SA&CA, addressing the technical and regulatory challenges of manned and unmanned aircraft coexistence in all classes of airspace. Analytical models are presented and validated to compute the overall avoidance volume in the airspace surrounding a tracked object, supporting automated SA&CA functionalities. The scientific basis of this approach is to assess real-time measurements and associated uncertainties affecting navigation states (of the host aircraft platform), tracking observables (of the static or moving object) and platform dynamics, and translate them to unified range and bearing uncertainty descriptors. The SA&CA unified approach provides an innovative analytical framework to generate high-fidelity dynamic geo-fences suitable for integration in the NG-FMS and in the ATM/UTM/defence decision support tools