Electro-Encephalography and Electro-Oculography in Aeronautics: A Review Over the Last Decade (2010–2020)

Electro-encephalography (EEG) and electro-oculography (EOG) are methods of electrophysiological monitoring that have potentially fruitful applications in neuroscience, clinical exploration, the aeronautical industry, and other sectors. These methods are often the most straightforward way of evaluating brain oscillations and eye movements, as they use standard laboratory or mobile techniques. This review describes the potential of EEG and EOG systems and the application of these methods in aeronautics. For example, EEG and EOG signals can be used to design brain-computer interfaces (BCI) and to interpret brain activity, such as monitoring the mental state of a pilot in determining their workload. The main objectives of this review are to, (i) offer an in-depth review of literature on the basics of EEG and EOG and their application in aeronautics; (ii) to explore the methodology and trends of research in combined EEG-EOG studies over the last decade; and (iii) to provide methodological guidelines for beginners and experts when applying these methods in environments outside the laboratory, with a particular focus on human factors and aeronautics. The study used databases from scientific, clinical, and neural engineering fields. The review first introduces the characteristics and the application of both EEG and EOG in aeronautics, undertaking a large review of relevant literature, from early to more recent studies. We then built a novel taxonomy model that includes 150 combined EEG-EOG papers published in peer-reviewed scientific journals and conferences from January 2010 to March 2020. Several data elements were reviewed for each study (e.g., pre-processing, extracted features and performance metrics), which were then examined to uncover trends in aeronautics and summarize interesting methods from this important body of literature. Finally, the review considers the advantages and limitations of these methods as well as future challenges

Novel technologies for the detection and mitigation of drowsy driving

In the human control of motor vehicles, there are situations regularly encountered wherein the vehicle operator becomes drowsy and fatigued due to the influence of long work days, long driving hours, or low amounts of sleep. Although various methods are currently proposed to detect drowsiness in the operator, they are either obtrusive, expensive, or otherwise impractical. The method of drowsy driving detection through the collection of Steering Wheel Movement (SWM) signals has become an important measure as it lends itself to accurate, effective, and cost-effective drowsiness detection. In this dissertation, novel technologies for drowsiness detection using Inertial Measurement Units (IMUs) are investigated and described. IMUs are an umbrella group of kinetic sensors (including accelerometers and gyroscopes) which transduce physical motions into data. Driving performances were recorded using IMUs as the primary sensors, and the resulting data were used by artificial intelligence algorithms, specifically Support Vector Machines (SVMs) to determine whether or not the individual was still fit to operate a motor vehicle. Results demonstrated high accuracy of the method in classifying drowsiness. It was also shown that the use of a smartphone-based approach to IMU monitoring of drowsiness will result in the initiation of feedback mechanisms upon a positive detection of drowsiness. These feedback mechanisms are intended to notify the driver of their drowsy state, and to dissuade further driving which could lead to crashes and/or fatalities. The novel methods not only demonstrated the ability to qualitatively determine a drivers drowsy state, but they were also low-cost, easy to implement, and unobtrusive to drivers. The efficacy, ease of use, and ease of access to these methods could potentially eliminate many barriers to the implementation of the technologies. Ultimately, it is hoped that these findings will help enhance traveler safety and prevent deaths and injuries to users

自然視条件下脳波計測の精度向上を可能にする眼球運動情報を用いた解析方法に関する研究

As the technique of electroencephalogram (EEG) developed for such many years, its application spreads and permeates into different areas, such like, clinical diagnosis, brain-computer interface, mental state estimation, and so on. Recently, using EEG as a tool for estimate people’s mental state and its extensional applications have jump into the limelight. These practical applications are urgently needed because the lack of subjectively estimating methods for the so called metal states, such as the concentration during study, the cognitive workload in driving, the calmness under mental training and so on. On the other hand, the application of EEG signals under daily life conditions especially when eye movements are totally without any constrains under a ‘fully free-view’ condition are obedient to the traditional ocular artifact suppression methods and how it meets the neuroscience standard has not been clearly expounded. This cause the ambiguities of explaining the obtain data and lead to susceptive results from data analysis. In our research, based on the basic idea of employing and extending EEG as the main tool for the estimation to mental state for daily life use, we confirmed the direction sensitivity of ocular artifacts induced by various types of eye movements and showed the most sensitive areas to the influence from it by multi zone-of-view experiment with standard neuroscience-targeted EEG devices. Enlightened from the results, we extended heuristic result on the use of more practical portable EEG devices. Besides, for a more realistic solution of the EEG based mental state estimation which is supposed to be applied for daily life environment, we studied the signal processing techniques of artifact suppression on low density electrode EEG and showed the importance of taking direction/eye position information into account when ocular artifact removal/suppression. In summary, this thesis has helped pave the practical way of using EEG signals toward the general use in daily life which has irregular eye movement patterns. We also pointed out the view-direction sensitivity of ocular artifact which helps the future studies to overcome the difficulties imposed on EEG applications by the free-view EEG tasks.九州工業大学博士学位論文 学位記番号:生工博甲第262号 学位授与年月日:平成28年3月26日1 Introduction|2 EEG measurements and ocular artifacts|3 Regression based solutions to ocular artifact suppression or removal in EEG|4 Measuring EEG with eye-tracking system|5 Direction and viewing area-sensitive influence of EOG artifacts revealed in the EEG topographic pattern analysis|6 Performance improvement of artifact removal with ocular information|7 Summary九州工業大学平成27年

Controlling a Mouse Pointer with a Single-Channel EEG Sensor

Goals: The purpose of this study was to analyze the feasibility of using the information obtained from a one-channel electro-encephalography (EEG) signal to control a mouse pointer. We used a low-cost headset, with one dry sensor placed at the FP1 position, to steer a mouse pointer and make selections through a combination of the user’s attention level with the detection of voluntary blinks. There are two types of cursor movements: spinning and linear displacement. A sequence of blinks allows for switching between these movement types, while the attention level modulates the cursor’s speed. The influence of the attention level on performance was studied. Additionally, Fitts’ model and the evolution of the emotional states of participants, among other trajectory indicators, were analyzed. (2) Methods: Twenty participants distributed into two groups (Attention and No-Attention) performed three runs, on different days, in which 40 targets had to be reached and selected. Target positions and distances from the cursor’s initial position were chosen, providing eight different indices of difficulty (IDs). A self-assessment manikin (SAM) test and a final survey provided information about the system’s usability and the emotions of participants during the experiment. (3) Results: The performance was similar to some brain–computer interface (BCI) solutions found in the literature, with an averaged information transfer rate (ITR) of 7 bits/min. Concerning the cursor navigation, some trajectory indicators showed our proposed approach to be as good as common pointing devices, such as joysticks, trackballs, and so on. Only one of the 20 participants reported difficulty in managing the cursor and, according to the tests, most of them assessed the experience positively. Movement times and hit rates were significantly better for participants belonging to the attention group. (4) Conclusions: The proposed approach is a feasible low-cost solution to manage a mouse pointe

Driver drowsiness monitoring using eye movement features derived from electrooculography

The increase in vehicle accidents due to the driver drowsiness over the last years highlights the need for developing reliable drowsiness assistant systems by a reference drowsiness measure. Therefore, the thesis at hand is aimed at classifying the driver vigilance state based on eye movements using electrooculography (EOG). In order to give an insight into the states of driving, which lead to critical safety situations, first, driver drowsiness, distraction and different terminologies in this context are described. Afterwards, countermeasures as techniques for keeping a driver awake and consequently preventing car crashes are reviewed. Since countermeasures do not have a long-lasting effect on the driver vigilance, intelligent driver drowsiness detection systems are needed. In the recent past, such systems have been developed on the market, some of which are introduced in this study. As also stated in previous studies, driver state is quantifiable by objective and subjective measures. The objective measures monitor the driver either directly or indirectly. For indirect monitoring of the driver, one uses the driving performance measures such as the lane keeping behavior or steering wheel movements. On the contrary, direct monitoring mainly comprises the driver’s physiological measures such as the brain activities, heart rate and eye movements. In order to assess these objective measures, subjective measures such as self-rating scores are required. This study introduces these measures and discusses the concerns about their interpretation and reliability. The developed drowsiness assistant systems on the market are all based on driving performance measures. These measures presuppose that the vehicle is steered solely by the driver himself. As long as other assistance systems with the concept to keep the vehicle in the middle of the lane are activated, driving performance measures would make wrong decisions about warnings. The reason is what the sensors measure is a combination of the driver’s behavior and the activated assistance system. In fact, the drowsiness warning system cannot determine the contribution of the driver in the driving task. This underscores the need for the direct monitoring of the driver. Previous works have introduced the drop of the alpha spindle rate (ASR) as a drowsiness indicator. This rate is a feature extracted out of the brain activity signals during the direct monitoring the driver. Additionally, ASR was shown to be sensitive to driver distraction, especially a visual one with an counteracting effect. We develop an algorithm based on eye movements to reduce the negative effect of the driver visual distraction on the ASR. This helps to partially improve the association of ASR with the driver drowsiness. Since the focus of this study is on driver eye movements, we introduce the human visual system and describe the idea of what and where to define the visual attention. Further, the structure of the human eye and relevant types of eye movements during driving are defined. We also categorize eye movements into two groups of slow and fast eye movements. We show that blinks, in principle, can belong to both of these groups depending on the driver’s vigilance state. EOG as a tool to measure the driver eye movements allows us to distinguish between drowsiness or distraction-related and driving situation dependent eye movements. Thus, in a pilot study, an experiment under fully controlled conditions is carried out on a proving ground to investigate the relationship between driver eye movements and different real driving scenarios. In this experiment, unwanted head vibrations within EOG signals and the sawtooth pattern (optokinetic nystagmus, OKN) of eyes are realized as situation dependent eye movements. The former occurs due to ground excitation and the latter happens during small radius (50m) curve negotiation. The statistical investigation expresses a significant variation of EOG due to unwanted head vibrations. Moreover, an analytical model is developed to explain the possible relationship of KON and tangent point of the curve. The developed model is validated against the real data on a high curvature track. In order to cover all relevant eye movement patterns during awake and drowsy driving, different experiments are conducted in this work including daytime and nighttime experiments under real road and simulated driving conditions. Based on the measured signals in the experiments, we study different eye movement detection approaches. We, first, investigate the conventional blink detection method based on the median filtering and show its drawback in detecting slow blinks and saccades. Afterwards, an adaptive detection approach is proposed based on the derivative of the EOG signal to simultaneously detect not only the eye blinks, but also other driving-relevant eye movements such as saccades and microsleep events. Moreover, in spite of the fact that drowsiness influences eye movement patterns, the proposed algorithm distinguishes between the often confused driving-related saccades and decreased amplitude blinks of a drowsy driver. The evaluation of results shows that the presented detection algorithm outperforms the common method based on median filtering so that fast eye movements are detected correctly during both awake and drowsy phases. Further, we address the detection of slower eye blinks, which are referred to as typical patterns of the drowsiness, by applying the continuous wavelet transform to EOG signals. In our proposed algorithm, by adjusting parameters of the wavelet transform, fast and slow blinks are detected simultaneously. However, this approach suffers from a larger false detection rate in comparison to the derivative-based method. As a result, for blink detection in this work, a combination of these two methods is applied. To improve the quality of the collected EOG signals, the discrete wavelet transform is benefited to remove noise and drift. For the noise removal, an adaptive thresholding strategy within the discrete wavelet transform is proposed which avoids sacrificing noise removal for saving blink amplitude or vice versa. In previous research, driver eye blink features (blink frequency, duration, etc.) have shown to be correlated to some extent with drowsiness. Hence, within a level of uncertainty they can contribute to driver drowsiness warning systems. In order to improve such systems, we investigate characteristics of detected blinks with respect to their different origins. We observed that in a real road experiment, blinks occur both spontaneously or due to gaze shifts. Gaze shifts between fixed positions, which occurred due to secondary visuomotor task, induced and modulated the occurrence of blinks. Moreover, the direction of the gaze shifts affected the occurrence of such blinks. Based on the eye movements during another experiment in a driving simulator without a secondary task, we found that the amount of gaze shifts (between various positions) is positively correlated with the probability of the blink occurrence. Therefore, we recommend handling gaze shift-induced blinks (e.g. during visual distraction) differently from those occurring spontaneously in drowsiness warning systems that rely solely on the variation of blink frequency as a driver state indicator. After studying dependencies between blink occurrence and gaze shifts, we extract 19 features out of each detected blink event of 43 subjects collected under both simulated and real driving conditions during 67 hours of both daytime and nighttime driving. This corresponds to the largest number of extracted eye blink features and the largest number of subjects among previous studies. We propose two approaches for aggregating features to improve their association with the slowly evolving drowsiness. In the first approach, we solely investigate parts of the collected data which are best correlated with the subjective self-rating score, i.e. Karolinska Sleepiness Scale. In the second approach, however, the entire data set with the maximum amount of information regarding driver drowsiness is scrutinized. For both approaches, the dependency between single features and drowsiness is studied statistically using correlation coefficients. The results show that the drowsiness dependency to features evolves to a larger extent non-linearly rather than linearly. Moreover, we show that for some features, different trends with respect to drowsiness are possible among different subjects. Consequently, we challenge warning systems which rely only on a single feature for their decision strategy and underscore that they are prone to high false alarm rates. In order to study whether a single feature is suitable for predicting safety-critical events, we study the overall variation of the features for all subjects shortly before the occurrence of the first unintentional lane departure and first unintentional microsleep in comparison to the beginning of the drive. Based on statistical tests, before the lane departure, most of the features change significantly. Therefore, we justify the role of blink features for the early driver drowsiness detection. However, this is not valid for the variation of features before the microsleep. We also focus on all 19 blink-based features together as one set. We assess the driver state by artificial neural network, support vector machine and k-nearest neighbors classifiers for both binary and multi-class cases. There, binary classifiers are trained both subject-independent and subject-dependent to address the generalization aspects of the results for unseen data. For the binary driver state prediction (awake vs. drowsy) using blink features, we have attained an average detection rate of 83% for each classifier separately. For 3-class classification (awake vs. medium vs. drowsy), however, the result was only 67%, possibly due to inaccurate self-rated vigilance states. Moreover, the issue of imbalanced data is addressed using classifier-dependent and classifier-independent approaches. We show that for reliable driver state classification, it is crucial to have events of both awake and drowsy phases in the data set in a balanced manner. The reason is that the proposed solutions in previous researches to deal with imbalanced data sets do not generalize the classifiers, but lead to their overfitting. The drawback of driving simulators in comparison to real driving is also discussed and to this end we perform a data reduction approach as a first remedy. As the second approach, we apply our trained classifiers to unseen drowsy data collected under real driving condition to investigate whether the drowsiness in driving simulators is representative of the drowsiness under real road conditions. With an average detection rate of about 68% for all classifiers, we conclude their similarity. Finally, we discuss feature dimension reduction approaches to determine the applicability of extracted features for in-vehicle warning systems. On this account, filter and wrapper approaches are introduced and compared with each other. Our comparison results show that wrapper approaches outperform the filter-based methods

Effects of circadian rhythm phase alteration on physiological and psychological variables: Implications to pilot performance (including a partially annotated bibliography)

The effects of environmental synchronizers upon circadian rhythmic stability in man and the deleterious alterations in performance and which result from changes in this stability are points of interest in a review of selected literature published between 1972 and 1980. A total of 2,084 references relevant to pilot performance and circadian phase alteration are cited and arranged in the following categories: (1) human performance, with focus on the effects of sleep loss or disturbance and fatigue; (2) phase shift in which ground based light/dark alteration and transmeridian flight studies are discussed; (3) shiftwork; (4)internal desynchronization which includes the effect of evironmental factors on rhythmic stability, and of rhythm disturbances on sleep and psychopathology; (5) chronotherapy, the application of methods to ameliorate desynchronization symptomatology; and (6) biorythm theory, in which the birthdate based biorythm method for predicting aircraft accident susceptability is critically analyzed. Annotations are provided for most citations

Sleep and Breathing at High Altitude

This thesis describes the work carried out during four treks, each over 10-11 days, from 1400m to 5000m in the Nepal Himalaya and further work performed during several two-night sojourns at the Barcroft Laboratory at 3800m on White Mountain in California, USA. Nineteen volunteers were studied during the treks in Nepal and seven volunteers were studied at White Mountain. All subjects were normal, healthy individuals who had not travelled to altitudes higher than 1000m in the previous twelve months. The aims of this research were to examine the effects on sleep, and the ventilatory patterns during sleep, of incremental increases in altitude by employing portable polysomnography to measure and record physiological signals. A further aim of this research was to examine the relationship between the ventilatory responses to hypoxia and hypercapnia, measured at sea level, and the development of periodic breathing during sleep at high altitude. In the final part of this thesis the possibility of preventing and treating Acute Mountain Sickness with non-invasive positive pressure ventilation while sleeping at high altitude was tested. Chapter 1 describes the background information on sleep, and breathing during sleep, at high altitudes. Most of these studies were performed in hypobaric chambers to simulate various high altitudes. One study measured sleep at high altitude after trekking, but there are no studies which systematically measure sleep and breathing throughout the whole trek. Breathing during sleep at high altitude and the physiological elements of the control of breathing (under normal/sea level conditions and under the hypobaric, hypoxic conditions present at high altitude) are described in this Chapter. The occurrence of Acute Mountain Sickness (AMS) in subjects who travel form near sea level to altitudes above 3000m is common but its pathophysiology not well understood. The background research into AMS and its treatment and prevention are also covered in Chapter 1. Chapter 2 describes the equipment and methods used in this research, including the polysomnographic equipment used to record sleep and breathing at sea level and the high altitude locations, the portable blood gas analyser used in Nepal and the equipment and methodology used to measure each individual’s ventilatory response to hypoxia and hypercapnia at sea level before ascent to the high altitude locations. Chapter 3 reports the findings on the changes to sleep at high altitude, with particular focus on changes in the amounts of total sleep, the duration of each sleep stage and its percentage of total sleep, and the number and causes of arousals from sleep that occurred during sleep at increasing altitudes. The lightest stage of sleep, Stage 1 non-rapid eye movement (NREM) sleep, was increased, as expected with increases in altitude, while the deeper stages of sleep (Stages 3 and 4 NREM sleep, also called slow wave sleep), were decreased. The increase in Stage 1 NREM in this research is in agreement with all previous findings. However, slow wave sleep, although decreased, was present in most of our subjects at all altitudes in Nepal; this finding is in contrast to most previous work, which has found a very marked reduction, even absence, of slow wave sleep at high altitude. Surprisingly, unlike experimental animal studies of chronic hypoxia, REM sleep was well maintained at all altitudes. Stage 2 NREM and REM sleep, total sleep time, sleep efficiency and spontaneous arousals were maintained at near sea level values. The total arousal index was increased with increasing altitude and this was due to the increasing severity of periodic breathing as altitude increased. An interesting finding of this research was that fewer than half the periodic breathing apneas and hypopneas resulted in arousal from sleep. There was a minor degree of upper airway obstruction in some subjects at sea level but this was almost resolved by 3500m. Chapter 4 reports the findings on the effects on breathing during sleep of the progressive increase of altitude, in particular the occurrence of periodic breathing. This Chapter also reports the results of changes to arterial blood gases as subjects ascended to higher altitudes. As expected, arterial blood gases were markedly altered at even the lowest altitude in Nepal (1400m) and this change became more pronounced at each new, higher altitude. Most subjects developed periodic breathing at high altitude but there was a wide variability between subjects as well as variability in the degree of periodic breathing that individual subjects developed at different altitudes. Some subjects developed periodic breathing at even the lowest altitude and this increased with increasing altitude; other subjects developed periodic breathing at one or two altitudes, while four subjects did not develop periodic breathing at any altitude. Ventilatory responses to hypoxia and hypercapnia, measured at sea level before departure to high altitude, was not significantly related to the development of periodic breathing when the group was analysed as a whole. However, when the subjects were grouped according to the steepness of their ventilatory response slopes, there was a pattern of higher amounts of periodic breathing in subjects with steeper ventilatory responses. Chapter 5 reports the findings of an experimental study carried out in the University of California, San Diego, Barcroft Laboratory on White Mountain in California. Seven subjects drove from sea level to 3800m in one day and stayed at this altitude for two nights. On one of the nights the subjects slept using a non-invasive positive pressure device via a face mask and this was found to significantly improve the sleeping oxyhemoglobin saturation. The use of the device was also found to eliminate the symptoms of Acute Mountain Sickness, as measured by the Lake Louise scoring system. This finding appears to confirm the hypothesis that lower oxygen saturation, particularly during sleep, is strongly correlated to the development of Acute Mountain Sickness and may represent a new treatment and prevention strategy for this very common high altitude disorder

Advanced Signal Processing in Wearable Sensors for Health Monitoring

Smart, wearables devices on a miniature scale are becoming increasingly widely available, typically in the form of smart watches and other connected devices. Consequently, devices to assist in measurements such as electroencephalography (EEG), electrocardiogram (ECG), electromyography (EMG), blood pressure (BP), photoplethysmography (PPG), heart rhythm, respiration rate, apnoea, and motion detection are becoming more available, and play a significant role in healthcare monitoring. The industry is placing great emphasis on making these devices and technologies available on smart devices such as phones and watches. Such measurements are clinically and scientifically useful for real-time monitoring, long-term care, and diagnosis and therapeutic techniques. However, a pertaining issue is that recorded data are usually noisy, contain many artefacts, and are affected by external factors such as movements and physical conditions. In order to obtain accurate and meaningful indicators, the signal has to be processed and conditioned such that the measurements are accurate and free from noise and disturbances. In this context, many researchers have utilized recent technological advances in wearable sensors and signal processing to develop smart and accurate wearable devices for clinical applications. The processing and analysis of physiological signals is a key issue for these smart wearable devices. Consequently, ongoing work in this field of study includes research on filtration, quality checking, signal transformation and decomposition, feature extraction and, most recently, machine learning-based methods