An EEG-based perceptual function integration network for application to drowsy driving

© 2015 Elsevier B.V. All rights reserved. Drowsy driving is among the most critical causes of fatal crashes. Thus, the development of an effective algorithm for detecting a driver's cognitive state demands immediate attention. For decades, studies have observed clear evidence using electroencephalography that the brain's rhythmic activities fluctuate from alertness to drowsiness. Recognition of this physiological signal is the major consideration of neural engineering for designing a feasible countermeasure. This study proposed a perceptual function integration system which used spectral features from multiple independent brain sources for application to recognize the driver's vigilance state. The analysis of brain spectral dynamics demonstrated physiological evidenced that the activities of the multiple cortical sources were highly related to the changes of the vigilance state. The system performances showed a robust and improved accuracy as much as 88% higher than any of results performed by a single-source approach

Electro-Encephalography and Electro-Oculography in Aeronautics: A Review Over the Last Decade (2010–2020)

Electro-encephalography (EEG) and electro-oculography (EOG) are methods of electrophysiological monitoring that have potentially fruitful applications in neuroscience, clinical exploration, the aeronautical industry, and other sectors. These methods are often the most straightforward way of evaluating brain oscillations and eye movements, as they use standard laboratory or mobile techniques. This review describes the potential of EEG and EOG systems and the application of these methods in aeronautics. For example, EEG and EOG signals can be used to design brain-computer interfaces (BCI) and to interpret brain activity, such as monitoring the mental state of a pilot in determining their workload. The main objectives of this review are to, (i) offer an in-depth review of literature on the basics of EEG and EOG and their application in aeronautics; (ii) to explore the methodology and trends of research in combined EEG-EOG studies over the last decade; and (iii) to provide methodological guidelines for beginners and experts when applying these methods in environments outside the laboratory, with a particular focus on human factors and aeronautics. The study used databases from scientific, clinical, and neural engineering fields. The review first introduces the characteristics and the application of both EEG and EOG in aeronautics, undertaking a large review of relevant literature, from early to more recent studies. We then built a novel taxonomy model that includes 150 combined EEG-EOG papers published in peer-reviewed scientific journals and conferences from January 2010 to March 2020. Several data elements were reviewed for each study (e.g., pre-processing, extracted features and performance metrics), which were then examined to uncover trends in aeronautics and summarize interesting methods from this important body of literature. Finally, the review considers the advantages and limitations of these methods as well as future challenges

A Physiological Signal Processing System for Optimal Engagement and Attention Detection.

In today’s high paced, hi-tech and high stress environment, with extended work hours, long to-do lists and neglected personal health, sleep deprivation has become common in modern culture. Coupled with these factors is the inherent repetitious and tedious nature of certain occupations and daily routines, which all add up to an undesirable fluctuation in individuals’ cognitive attention and capacity. Given certain critical professions, a momentary or prolonged lapse in attention level can be catastrophic and sometimes deadly. This research proposes to develop a real-time monitoring system which uses fundamental physiological signals such as the Electrocardiograph (ECG), to analyze and predict the presence or lack of cognitive attention in individuals during task execution. The primary focus of this study is to identify the correlation between fluctuating level of attention and its implications on the physiological parameters of the body. The system is designed using only those physiological signals that can be collected easily with small, wearable, portable and non-invasive monitors and thereby being able to predict well in advance, an individual’s potential loss of attention and ingression of sleepiness. Several advanced signal processing techniques have been implemented and investigated to derive multiple clandestine and informative features. These features are then applied to machine learning algorithms to produce classification models that are capable of differentiating between the cases of a person being attentive and the person not being attentive. Furthermore, Electroencephalograph (EEG) signals are also analyzed and classified for use as a benchmark for comparison with ECG analysis. For the study, ECG signals and EEG signals of volunteer subjects are acquired in a controlled experiment. The experiment is designed to inculcate and sustain cognitive attention for a period of time following which an attempt is made to reduce cognitive attention of volunteer subjects. The data acquired during the experiment is decomposed and analyzed for feature extraction and classification. The presented results show that to a fairly reasonable accuracy it is possible to detect the presence or lack of attention in individuals with just their ECG signal, especially in comparison with analysis done on EEG signals. The continual work of this research includes other physiological signals such as Galvanic Skin Response, Heat Flux, Skin Temperature and video based facial feature analysis

Sitting behaviour-based pattern recognition for predicting driver fatigue

The proposed approach based on physiological characteristics of sitting behaviours and sophisticated machine learning techniques would enable an effective and practical solution to driver fatigue prognosis since it is insensitive to the illumination of driving environment, non-obtrusive to driver, without violating driver’s privacy, more acceptable by drivers

Heterogeneous data fusion for brain psychology applications

This thesis aims to apply Empirical Mode Decomposition (EMD), Multiscale Entropy (MSE), and collaborative adaptive filters for the monitoring of different brain consciousness states. Both block based and online approaches are investigated, and a possible extension to the monitoring and identification of Electromyograph (EMG) states is provided. Firstly, EMD is employed as a multiscale time-frequency data driven tool to decompose a signal into a number of band-limited oscillatory components; its data driven nature makes EMD an ideal candidate for the analysis of nonlinear and non-stationary data. This methodology is further extended to process multichannel real world data, by making use of recent theoretical advances in complex and multivariate EMD. It is shown that this can be used to robustly measure higher order features in multichannel recordings to robustly indicate ‘QBD’. In the next stage, analysis is performed in an information theory setting on multiple scales in time, using MSE. This enables an insight into the complexity of real world recordings. The results of the MSE analysis and the corresponding statistical analysis show a clear difference in MSE between the patients in different brain consciousness states. Finally, an online method for the assessment of the underlying signal nature is studied. This method is based on a collaborative adaptive filtering approach, and is shown to be able to approximately quantify the degree of signal nonlinearity, sparsity, and non-circularity relative to the constituent subfilters. To further illustrate the usefulness of the proposed data driven multiscale signal processing methodology, the final case study considers a human-robot interface based on a multichannel EMG analysis. A preliminary analysis shows that the same methodology as that applied to the analysis of brain cognitive states gives robust and accurate results. The analysis, simulations, and the scope of applications presented suggest great potential of the proposed multiscale data processing framework for feature extraction in multichannel data analysis. Directions for future work include further development of real-time feature map approaches and their use across brain-computer and brain-machine interface applications

Enhancing brain-computer interfacing through advanced independent component analysis techniques

A Brain-computer interface (BCI) is a direct communication system between a brain and an external device in which messages or commands sent by an individual do not pass through the brain’s normal output pathways but is detected through brain signals. Some severe motor impairments, such as Amyothrophic Lateral Sclerosis, head trauma, spinal injuries and other diseases may cause the patients to lose their muscle control and become unable to communicate with the outside environment. Currently no effective cure or treatment has yet been found for these diseases. Therefore using a BCI system to rebuild the communication pathway becomes a possible alternative solution. Among different types of BCIs, an electroencephalogram (EEG) based BCI is becoming a popular system due to EEG’s fine temporal resolution, ease of use, portability and low set-up cost. However EEG’s susceptibility to noise is a major issue to develop a robust BCI. Signal processing techniques such as coherent averaging, filtering, FFT and AR modelling, etc. are used to reduce the noise and extract components of interest. However these methods process the data on the observed mixture domain which mixes components of interest and noise. Such a limitation means that extracted EEG signals possibly still contain the noise residue or coarsely that the removed noise also contains part of EEG signals embedded. Independent Component Analysis (ICA), a Blind Source Separation (BSS) technique, is able to extract relevant information within noisy signals and separate the fundamental sources into the independent components (ICs). The most common assumption of ICA method is that the source signals are unknown and statistically independent. Through this assumption, ICA is able to recover the source signals. Since the ICA concepts appeared in the fields of neural networks and signal processing in the 1980s, many ICA applications in telecommunications, biomedical data analysis, feature extraction, speech separation, time-series analysis and data mining have been reported in the literature. In this thesis several ICA techniques are proposed to optimize two major issues for BCI applications: reducing the recording time needed in order to speed up the signal processing and reducing the number of recording channels whilst improving the final classification performance or at least with it remaining the same as the current performance. These will make BCI a more practical prospect for everyday use. This thesis first defines BCI and the diverse BCI models based on different control patterns. After the general idea of ICA is introduced along with some modifications to ICA, several new ICA approaches are proposed. The practical work in this thesis starts with the preliminary analyses on the Southampton BCI pilot datasets starting with basic and then advanced signal processing techniques. The proposed ICA techniques are then presented using a multi-channel event related potential (ERP) based BCI. Next, the ICA algorithm is applied to a multi-channel spontaneous activity based BCI. The final ICA approach aims to examine the possibility of using ICA based on just one or a few channel recordings on an ERP based BCI. The novel ICA approaches for BCI systems presented in this thesis show that ICA is able to accurately and repeatedly extract the relevant information buried within noisy signals and the signal quality is enhanced so that even a simple classifier can achieve good classification accuracy. In the ERP based BCI application, after multichannel ICA the data just applied to eight averages/epochs can achieve 83.9% classification accuracy whilst the data by coherent averaging can reach only 32.3% accuracy. In the spontaneous activity based BCI, the use of the multi-channel ICA algorithm can effectively extract discriminatory information from two types of singletrial EEG data. The classification accuracy is improved by about 25%, on average, compared to the performance on the unpreprocessed data. The single channel ICA technique on the ERP based BCI produces much better results than results using the lowpass filter. Whereas the appropriate number of averages improves the signal to noise rate of P300 activities which helps to achieve a better classification. These advantages will lead to a reliable and practical BCI for use outside of the clinical laboratory

Detecting Flow Experiences in Cognitive Tasks - A Neurophysiological Approach

Das Flow-Erlebnis beschreibt einen Zustand vollständiger Aufgabenvertiefung und mühelosen Handelns, der mit Höchstleistungen, persönlichem Wachstum, sowie allgemeinem Wohlbefinden verbunden ist. Für Unternehmen stellen häufigere Flow-Erlebnisse der ArbeitnehmerInnen daher auch eine produktivitäts- und zufriedenheitsfördernde Basis dar. Vor allem da sich aktuell globale Phänomene wie die steigende Nachfrage nach Wissensarbeit und das niedrige Arbeitsengagement zuspitzen, können Unternehmen von einer Förderung von Flow profitieren. Die Unterstützung von Flow stellt allerdings aufgrund der Vielfalt von Arbeitnehmerfertigkeiten, -aufgaben, und -arbeitsplätzen eine komplexe Herausforderung dar. WissensarbeiterInnen stehen dynamischen Aufgaben gegenüber, die diverse Kompetenzen und die Kooperation mit anderen erfordern. Arbeitsplätze werden vielseitiger, indem die Grenzen zwischen ko-präsenten und virtuellen Interaktionen verschwinden. Diese Vielfalt bedeutet, dass eine solide Flow-Förderung nur durch personen-, aufgaben- und situationsunabhängige Ansätze erfolgen kann. Aus diesem Grund werden zunehmend die neurophysiologischen Grundlagen des Flow-Erlebens untersucht. Auf deren Basis könnten adaptive Neuro-Informationssysteme entwickelt werden, die mittels tragbarer Sensorik Flow kontinuierlich erkennen und fördern können. Diese Wissensbasis ist bislang jedoch nur spärlich und in stark fragmentierter Form vorhanden. Für das Individuum existieren lediglich konkurrierende Vorschläge, die noch nicht durch situations- und sensorübergreifende Studien konsolidiert wurden. Für Gruppen existiert noch fast keine Forschung zu neurophysiologischen Flow-Korrelaten, insbesondere keine im Kontext digital-mediierter Interaktionen. In dieser Dissertation werden genau diese Forschungslücken durch die situationsübergreifende Beobachtung von Flow mit tragbaren EKG und EEG Sensoren adressiert. Dabei werden zentrale Grenzen der experimentellen Flow-Forschung berücksichtigt, vor allem die Defizite etablierter Paradigmen zum kontrollierten Hervorrufen von Flow. Indem Erlebnisse in zwei kognitiven Aufgaben und mehreren Manipulationen (von Schwierigkeit, Natürlichkeit, Autonomie und sozialer Interaktion) variiert werden, wird untersucht, wie Flow intensiver hervorgerufen und wie das Erlebnis stabiler über Situationen hinweg beobachtet werden kann. Die Studienergebnisse deuten dabei insgesamt auf ein Flow-Muster von moderater physiologischer Aktivierung und mentaler Arbeitslast, von erhöhter, aufgabenorientierter Aufmerksamkeit und von affektiver Neutralität hin. Vor allem die EEG Daten zeigen ein diagnostisches Potenzial, schwächere von stärkeren Flow-Zuständen unterscheiden zu können, indem optimale und nicht-optimale Aufgabenschwierigkeiten (für Individuen und Gruppen) erkannt werden. Um das Flow-Erleben weiter zu fördern, werden geeignete Wege für zukünftige Forschung abschließend diskutiert

Development of nonlinear techniques based on time-frequency representation and information theory for the analysis of EEG signals to assess different states of consciousness

Electroencephalogram (EEG) recordings provide insight into the changes in brain activity associated with various states of anesthesia, epilepsy, brain attentiveness, sleep disorders, brain disorders, etc. EEG's are complex signals whose statistical properties depend on both space and time. Their randomness and non-stationary characteristics make them impossible to be described in an accurate way with a simple technique, requiring analysis and characterization involves techniques that take into account their non-stationarity. For that, new advanced techniques in order to improve the efficiency of the EEG based methods used in the clinical practice have to be developed. The main objective of this thesis was to investigate and implement different methods based on nonlinear techniques in order to develop indexes able to characterize the frequency spectrum, the nonlinear dynamics and the complexity of the EEG signals recorded in different state of consciousness. Firstly, a new method for removing peak and spike in biological signal based on the signal envelope was successfully designed and applied to simulated and real EEG signals, obtaining performances significantly better than the traditional adaptive filters. Then, several studies were carried out in order to extract and evaluate EEG measures based on nonlinear techniques in different contexts such as the automatic detection of sleepiness and the characterization and prediction of the nociceptive stimuli and the assessment of the sedation level. Four novel indexes were defined by calculating entropy of the Choi-Williams distribution (CWD) with respect to time or frequency, by using the probability mass function at each time instant taken independently or by using the probability mass function of the entire CWD. The values of these indexes tend to decrease, with different proportion, when the behavior of the signals evolved from chaos or randomness to periodicity and present differences when comparing EEG recorded in eyes-open and eyes-closed states and in ictal and non-ictal states. Measures obtained with time-frequency representation, mutual information function and correntropy, were applied to EEG signals for the automatic sleepiness detection in patients suffering sleep disorders. The group of patients with excessive daytime sleepiness presented more power in ¿ band than the group without sleepiness, which presented higher spectral and cross-spectral entropy in the frontal zone in d band. More complexity in the occipital zone was found in the group of patients without sleepiness in ß band, while a stronger nonlinear coupling between the occipital and frontal regions was detected in patients with excessive daytime sleepiness, in ß band. Time-frequency representation and non-linear measures were also used in order to study how adaptation and fatigue affect the event-related brain potentials to stimuli of different modalities. Differences between the responses to infrequent and frequent stimulation in different recording periods were found in series of averaged EEG epochs recorded after thermal, electrical and auditory stimulation. Nonlinear measures calculated on EEG filtered in the traditional frequency bands and in higher frequency bands improved the assessment of the sedation level. These measures were obtained by applying all the developed techniques on signals recorded from patients sedated, in order to predict the responses to pain stimulation such as nail bad compression and endoscopy tube insertion. The proposed measures exhibit better performances than the bispectral index (BIS), a traditional indexes used for hypnosis assessment. In conclusion, nonlinear measures based on time-frequency representation, mutual information functions and correntropy provided additional information that helped to improve the automatic sleepiness detection, the characterization and prediction of the nociceptive responses and thus the assessment of the sedation level.El registro de la señal Electroencefalografíca (EEG) proporciona información sobre los cambios en la actividad cerebral asociados con varios estados de la anestesia, la epilepsia, la atención cerebral, los trastornos del sueño, los trastornos cerebrales, etc. Los EEG son señales complejas cuyas propiedades estadísticas dependen del espacio y del tiempo. Sus características aleatorias y no estacionarias hacen imposible que el EEG se describa de forma precisa con una técnica sencilla requiriendo un análisis y una caracterización que implica técnicas que tengan en cuenta su no estacionariedad. Todo esto aumenta la necesidad de desarrollar nuevas técnicas avanzadas con el fin de mejorar la eficiencia de los métodos utilizados en la práctica clínica que son basados en el análisis de EEG. En esta tesis se han investigado y aplicado diferentes métodos utilizando técnicas no lineales con el fin de desarrollar índices capaces de caracterizar el espectro de frecuencias, la dinámica no lineal y la complejidad de las señales EEG registradas en diferentes estados de conciencia. En primer lugar, se ha desarrollado un nuevo algoritmo basado en la envolvente de la señal para la eliminación de ruido de picos en las señales biológicas. Este algoritmo ha sido aplicado a señales simuladas y reales obteniendo resultados significativamente mejores comparados con los filtros adaptativos tradicionales. Seguidamente, se han llevado a cabo varios estudios con el fin de extraer y evaluar las medidas de EEG basadas en técnicas no lineales en diferentes contextos. Se han definido nuevos índices mediante el cálculo de la entropía de la distribución de Choi-Williams (DCW) con respecto al tiempo o la frecuencia. Se ha observado que los valores de estos índices tienden a disminuir, en diferentes proporciones, cuando el comportamiento de las señales evoluciona de caótico o aleatorio a periódico. Además, se han encontrado valores diferentes de estos índices aplicados a la señal EEG registrada en diferentes estados. Diferentes medidas basadas en la representación tiempo-frecuencia, la función de información mutua y la correntropia se han aplicado al EEG para la detección automática de la somnolencia en pacientes que sufren trastornos del sueño. Se ha observado en la zona frontal que la potencia en la banda θ es mayor en los pacientes con somnolencia diurna excesiva, mientras que la entropía espectral y la entropía espectral cruzada en la banda δ es mayor en los pacientes sin somnolencia. En el grupo sin somnolencia se ha encontrado más complejidad en la zona occipital, mientras que el acoplamiento no lineal entre las regiones occipital y frontal ha resultado más fuerte en pacientes con somnolencia diurna excesiva, en la banda β. La representación tiempo-frecuencia y las medidas no lineales se han utilizado para estudiar cómo la adaptación y la fatiga afectan a los potenciales cerebrales relacionados con estímulos térmicos, eléctricos y auditivos. Analizando el promedio de varias épocas de EEG grabadas después de la estimulación, se han encontrado diferencias entre las respuestas a la estimulación frecuente e infrecuente en diferentes períodos de registro. Todas las técnicas que se han desarrollado, se han aplicado a señales EEG registradas en pacientes sedados, con el fin de predecir las respuestas a la estimulación del dolor. Un conjunto de medidas calculadas en señales EEG filtradas en diferentes bandas de frecuencia ha permitido mejorar la evaluación del nivel de sedación. Las medidas propuestas han presentado un mejor rendimiento comparado con el índice bispectral, un indicador de hipnosis tradicional. En conclusión, las medidas no lineales basadas en la representación tiempofrecuencia, funciones de información mutua y correntropia han proporcionado informaciones adicionales que contribuyeron a mejorar la detección automática de la somnolencia, la caracterización y predicción de las respuestas nociceptivas y por lo tanto la evaluación del nivel de sedación

Novel technologies for the detection and mitigation of drowsy driving

In the human control of motor vehicles, there are situations regularly encountered wherein the vehicle operator becomes drowsy and fatigued due to the influence of long work days, long driving hours, or low amounts of sleep. Although various methods are currently proposed to detect drowsiness in the operator, they are either obtrusive, expensive, or otherwise impractical. The method of drowsy driving detection through the collection of Steering Wheel Movement (SWM) signals has become an important measure as it lends itself to accurate, effective, and cost-effective drowsiness detection. In this dissertation, novel technologies for drowsiness detection using Inertial Measurement Units (IMUs) are investigated and described. IMUs are an umbrella group of kinetic sensors (including accelerometers and gyroscopes) which transduce physical motions into data. Driving performances were recorded using IMUs as the primary sensors, and the resulting data were used by artificial intelligence algorithms, specifically Support Vector Machines (SVMs) to determine whether or not the individual was still fit to operate a motor vehicle. Results demonstrated high accuracy of the method in classifying drowsiness. It was also shown that the use of a smartphone-based approach to IMU monitoring of drowsiness will result in the initiation of feedback mechanisms upon a positive detection of drowsiness. These feedback mechanisms are intended to notify the driver of their drowsy state, and to dissuade further driving which could lead to crashes and/or fatalities. The novel methods not only demonstrated the ability to qualitatively determine a drivers drowsy state, but they were also low-cost, easy to implement, and unobtrusive to drivers. The efficacy, ease of use, and ease of access to these methods could potentially eliminate many barriers to the implementation of the technologies. Ultimately, it is hoped that these findings will help enhance traveler safety and prevent deaths and injuries to users