A Brain-Controlled Vehicle System Based on Steady State Visual Evoked Potentials

In this paper, we propose a human-vehicle cooperative driving system. The objectives of this research are twofold: (1) providing a feasible brain-controlled vehicle (BCV) mode; (2) providing a human-vehicle cooperative control mode. For the first aim, through a brain-computer interface (BCI), we can analyse the EEG signal and get the driving intentions of the driver. For the second aim, the human-vehicle cooperative control is manifested in the BCV combined with the obstacle detection assistance. Considering the potential dangers of driving a real motor vehicle in the outdoor, an obstacle detection module is essential in the human-vehicle cooperative driving system. Obstacle detection and emergency braking can ensure the safety of the driver and the vehicle during driving. EEG system based on steady-state visual evoked potential (SSVEP) is used in the BCI. Simulation and real vehicle driving experiment platform are designed to verify the feasibility of the proposed human-vehicle cooperative driving system. Five subjects participated in the simulation experiment and real the vehicle driving experiment. The outdoor experimental results show that the average accuracy of intention recognition is 90.68 ± 2.96% on the real vehicle platform. In this paper, we verified the feasibility of the SSVEP-based BCV mode and realised the human-vehicle cooperative driving system.Fil: Zhang, Zhao. Civil Aviation University Of China; ChinaFil: Han, Shuning. Universitat de Vic - Universitat Central de Catalunya ; EspañaFil: Yi, Huaihai. China University Of Geosciences; ChinaFil: Duan, Feng. Nankai University; ChinaFil: Kang, Fei. Maebashi Institute Of Technology; JapónFil: Sun, Zhe. Riken; JapónFil: Solé Casals, Jordi. Universitat de Vic - Universitat Central de Catalunya; España. Nankai University; China. University of Cambridge; Reino UnidoFil: Caiafa, César Federico. Provincia de Buenos Aires. Gobernación. Comisión de Investigaciones Científicas. Instituto Argentino de Radioastronomía. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - La Plata. Instituto Argentino de Radioastronomía; Argentina. Nankai University; Chin

Applications of brain imaging methods in driving behaviour research

Applications of neuroimaging methods have substantially contributed to the scientific understanding of human factors during driving by providing a deeper insight into the neuro-cognitive aspects of driver brain. This has been achieved by conducting simulated (and occasionally, field) driving experiments while collecting driver brain signals of certain types. Here, this sector of studies is comprehensively reviewed at both macro and micro scales. Different themes of neuroimaging driving behaviour research are identified and the findings within each theme are synthesised. The surveyed literature has reported on applications of four major brain imaging methods. These include Functional Magnetic Resonance Imaging (fMRI), Electroencephalography (EEG), Functional Near-Infrared Spectroscopy (fNIRS) and Magnetoencephalography (MEG), with the first two being the most common methods in this domain. While collecting driver fMRI signal has been particularly instrumental in studying neural correlates of intoxicated driving (e.g. alcohol or cannabis) or distracted driving, the EEG method has been predominantly utilised in relation to the efforts aiming at development of automatic fatigue/drowsiness detection systems, a topic to which the literature on neuro-ergonomics of driving particularly has shown a spike of interest within the last few years. The survey also reveals that topics such as driver brain activity in semi-automated settings or the brain activity of drivers with brain injuries or chronic neurological conditions have by contrast been investigated to a very limited extent. Further, potential topics in relation to driving behaviour are identified that could benefit from the adoption of neuroimaging methods in future studies

Convolutional Spiking Neural Networks for Detecting Anticipatory Brain Potentials Using Electroencephalogram

Spiking neural networks (SNNs) are receiving increased attention as a means to develop "biologically plausible" machine learning models. These networks mimic synaptic connections in the human brain and produce spike trains, which can be approximated by binary values, precluding high computational cost with floating-point arithmetic circuits. Recently, the addition of convolutional layers to combine the feature extraction power of convolutional networks with the computational efficiency of SNNs has been introduced. In this paper, the feasibility of using a convolutional spiking neural network (CSNN) as a classifier to detect anticipatory slow cortical potentials related to braking intention in human participants using an electroencephalogram (EEG) was studied. The EEG data was collected during an experiment wherein participants operated a remote controlled vehicle on a testbed designed to simulate an urban environment. Participants were alerted to an incoming braking event via an audio countdown to elicit anticipatory potentials that were then measured using an EEG. The CSNN's performance was compared to a standard convolutional neural network (CNN) and three graph neural networks (GNNs) via 10-fold cross-validation. The results showed that the CSNN outperformed the other neural networks.Comment: 14 pages, 6 figures, Scientific Reports submissio

Recognizing driver braking intention with vehicle data using unsupervised learning methods

Recently, the development of braking assistance system has largely benefit the safety of both driver and pedestrians. A robust prediction and detection of driver braking intention will enable driving assistance system response to traffic situation correctly and improve the driving experience of intelligent vehicles. Unsupervised machine learning algorithms has been widely used in clustering and pattern mining in previous researches. In this paper, a various unsupervised clustering methods will be used to build a driver braking intention predictor which can accurately recognize the braking maneuver based on vehicle data captured with CAN bus. The braking maneuver along with other driving maneuvers such as normal driving will be clustered and the results from different methods like K-means and Gaussian mixture model will be compared. Additionally, the evaluation of features from raw data, which are important to driving maneuvers clustering will be proposed. The experiment data are collected from one hybrid electric vehicle in real world. Final results show that the proposed method can detect driver’s braking intention in a very beginning moment with a high accuracy and the most important sets of feature for driving maneuver clustering will be discussed.Recently, the development of braking assistance system has largely benefit the safety of both driver and pedestrians. A robust prediction and detection of driver braking intention will enable driving assistance system response to traffic situation correctly and improve the driving experience of intelligent vehicles. Unsupervised machine learning algorithms has been widely used in clustering and pattern mining in previous researches. In this paper, a various unsupervised clustering methods will be used to build a driver braking intention predictor which can accurately recognize the braking maneuver based on vehicle data captured with CAN bus. The braking maneuver along with other driving maneuvers such as normal driving will be clustered and the results from different methods like K-means and Gaussian mixture model will be compared. Additionally, the evaluation of features from raw data, which are important to driving maneuvers clustering will be proposed. The experiment data are collected from one hybrid electric vehicle in real world. Final results show that the proposed method can detect driver’s braking intention in a very beginning moment with a high accuracy and the most important sets of feature for driving maneuver clustering will be discussed

Action prediction based on anticipatory brain potentials during simulated driving

Objective. The ability of an automobile to infer the driver's upcoming actions directly from neural signals could enrich the interaction of the car with its driver. Intelligent vehicles fitted with an on-board brain–computer interface able to decode the driver's intentions can use this information to improve the driving experience. In this study we investigate the neural signatures of anticipation of specific actions, namely braking and accelerating. Approach. We investigated anticipatory slow cortical potentials in electroencephalogram recorded from 18 healthy participants in a driving simulator using a variant of the contingent negative variation (CNV) paradigm with Go and No-go conditions: count-down numbers followed by 'Start'/'Stop' cue. We report decoding performance before the action onset using a quadratic discriminant analysis classifier based on temporal features. Main results. (i) Despite the visual and driving related cognitive distractions, we show the presence of anticipatory event related potentials locked to the stimuli onset similar to the widely reported CNV signal (with an average peak value of −8 μV at electrode Cz). (ii) We demonstrate the discrimination between cases requiring to perform an action upon imperative subsequent stimulus (Go condition, e.g. a 'Red' traffic light) versus events that do not require such action (No-go condition; e.g. a 'Yellow' light); with an average single trial classification performance of 0.83 ± 0.13 for braking and 0.79 ± 0.12 for accelerating (area under the curve). (iii) We show that the centro-medial anticipatory potentials are observed as early as 320 ± 200 ms before the action with a detection rate of 0.77 ± 0.12 in offline analysis. Significance. We show for the first time the feasibility of predicting the driver's intention through decoding anticipatory related potentials during simulated car driving with high recognition rates

Decoding Neural Correlates of Cognitive States to Enhance Driving Experience

Modern cars can support their drivers by assessing and autonomously performing different driving maneuvers based on information gathered by in-car sensors. We propose that brain–machine interfaces (BMIs) can provide complementary information that can ease the interaction with intelligent cars in order to enhance the driving experience. In our approach, the human remains in control, while a BMI is used to monitor the driver's cognitive state and use that information to modulate the assistance provided by the intelligent car. In this paper, we gather our proof-of-concept studies demonstrating the feasibility of decoding electroencephalography correlates of upcoming actions and those reflecting whether the decisions of driving assistant systems are in-line with the drivers' intentions. Experimental results while driving both simulated and real cars consistently showed neural signatures of anticipation, movement preparation, and error processing. Remarkably, despite the increased noise inherent to real scenarios, these signals can be decoded on a single-trial basis, reflecting some of the cognitive processes that take place while driving. However, moderate decoding performance compared to the controlled experimental BMI paradigms indicate there exists room for improvement of the machine learning methods typically used in the state-of-the-art BMIs. We foresee that neural fusion correlates with information extracted from other physiological measures, e.g., eye movements or electromyography as well as contextual information gathered by in-car sensors will allow intelligent cars to provide timely and tailored assistance only if it is required; thus, keeping the user in the loop and allowing him to fully enjoy the driving experience

Revealing driver psychophysiological response to emergency braking in distracted driving based on field experiments

Purpose: The purpose of this paper is to characterize distracted driving by quantifying the response time and response intensity to an emergency stop using the driver’s physiological states. Design/methodology/approach: Field tests with 17 participants were conducted in the connected and automated vehicle test field. All participants were required to prioritize their primary driving tasks while a secondary nondriving task was asked to be executed. Demographic data, vehicle trajectory data and various physiological data were recorded through a biosignalsplux signal data acquisition toolkit, such as electrocardiograph for heart rate, electromyography for muscle strength, electrodermal activity for skin conductance and force-sensing resistor for braking pressure. Findings: This study quantified the psychophysiological responses of the driver who returns to the primary driving task from the secondary nondriving task when an emergency occurs. The results provided a prototype analysis of the time required for making a decision in the context of advanced driver assistance systems or for rebuilding the situational awareness in future automated vehicles when a driver’s take-over maneuver is needed. Originality/value: The hypothesis is that the secondary task will result in a higher mental workload and a prolonged reaction time. Therefore, the driver states in distracted driving are significantly different than in regular driving, the physiological signal improves measuring the brake response time and distraction levels and brake intensity can be expressed as functions of driver demographics. To the best of the authors’ knowledge, this is the first study using psychophysiological measures to quantify a driver’s response to an emergency stop during distracted driving

Revealing driver psychophysiological response to emergency braking in distracted driving based on field experiments

Purpose – The purpose of this paper is to characterize distracted driving by quantifying the response time and response intensity to an emergency stop using the driver’s physiological states. Design/methodology/approach – Field tests with 17 participants were conducted in the connected and automated vehicle test field. All participants were required to prioritize their primary driving tasks while a secondary nondriving task was asked to be executed. Demographic data, vehicle trajectory data and various physiological data were recorded through a biosignalsplux signal data acquisition toolkit, such as electrocardiograph for heart rate, electromyography for muscle strength, electrodermal activity for skin conductance and force-sensing resistor for braking pressure. Findings – This study quantified the psychophysiological responses of the driver who returns to the primary driving task from the secondary nondriving task when an emergency occurs. The results provided a prototype analysis of the time required for making a decision in the context of advanced driver assistance systems or for rebuilding the situational awareness in future automated vehicles when a driver’s take-over maneuver is needed. Originality/value – The hypothesis is that the secondary task will result in a higher mental workload and a prolonged reaction time. Therefore, the driver states in distracted driving are significantly different than in regular driving, the physiological signal improves measuring the brake response time and distraction levels and brake intensity can be expressed as functions of driver demographics. To the best of the authors’ knowledge, this is the first study using psychophysiological measures to quantify a driver’s response to an emergency stop during distracted driving

The Berlin Brain-Computer Interface: Progress Beyond Communication and Control

The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.EC/FP7/611570/EU/Symbiotic Mind Computer Interaction for Information Seeking/MindSeeEC/FP7/625991/EU/Hyperscanning 2.0 Analyses of Multimodal Neuroimaging Data: Concept, Methods and Applications/HYPERSCANNING 2.0DFG, 103586207, GRK 1589: Verarbeitung sensorischer Informationen in neuronalen Systeme

Systems engineering approaches to safety in transport systems

openDuring driving, driver behavior monitoring may provide useful information to prevent road traffic accidents caused by driver distraction. It has been shown that 90% of road traffic accidents are due to human error and in 75% of these cases human error is the only cause. Car manufacturers have been interested in driver monitoring research for several years, aiming to enhance the general knowledge of driver behavior and to evaluate the functional state as it may drastically influence driving safety by distraction, fatigue, mental workload and attention. Fatigue and sleepiness at the wheel are well known risk factors for traffic accidents. The Human Factor (HF) plays a fundamental role in modern transport systems. Drivers and transport operators control a vehicle towards its destination in according to their own sense, physical condition, experience and ability, and safety strongly relies on the HF which has to take the right decisions. On the other hand, we are experiencing a gradual shift towards increasingly autonomous vehicles where HF still constitutes an important component, but may in fact become the "weakest link of the chain", requiring strong and effective training feedback. The studies that investigate the possibility to use biometrical or biophysical signals as data sources to evaluate the interaction between human brain activity and an electronic machine relate to the Human Machine Interface (HMI) framework. The HMI can acquire human signals to analyse the specific embedded structures and recognize the behavior of the subject during his/her interaction with the machine or with virtual interfaces as PCs or other communication systems. Based on my previous experience related to planning and monitoring of hazardous material transport, this work aims to create control models focused on driver behavior and changes of his/her physiological parameters. Three case studies have been considered using the interaction between an EEG system and external device, such as driving simulators or electronical components. A case study relates to the detection of the driver's behavior during a test driver. Another case study relates to the detection of driver's arm movements according to the data from the EEG during a driver test. The third case is the setting up of a Brain Computer Interface (BCI) model able to detect head movements in human participants by EEG signal and to control an electronic component according to the electrical brain activity due to head turning movements. Some videos showing the experimental results are available at https://www.youtube.com/channel/UCj55jjBwMTptBd2wcQMT2tg.openXXXIV CICLO - INFORMATICA E INGEGNERIA DEI SISTEMI/ COMPUTER SCIENCE AND SYSTEMS ENGINEERING - Ingegneria dei sistemiZero, Enric