Combining fNIRS and EEG to improve motor cortex activity classification during an imagined movement-based task

This work serves as an initial investigation into improvements to classification accuracy of an imagined movement-based Brain Computer Interface (BCI) by combining the feature spaces of two unique measurement modalities: functional near infrared spectroscopy (fNIRS) and electroencephalography (EEG). Our dual-modality system recorded concurrent and co-locational hemodynamic and electrical responses in the motor cortex during an imagined movement task, participated in by two subjects. Offline analysis and classification of fNIRS and EEG data was performed using leave-one-out cross-validation (LOOCV) and linear discriminant analysis (LDA). Classification of 2-dimensional fNIRS and EEG feature spaces was performed separately and then their feature spaces were combined for further classification. Results of our investigation indicate that by combining feature spaces, modest gains in classification accuracy of an imagined movement-based BCI can be achieved by employing a supplemental measurement modality. It is felt that this technique may be particularly useful in the design of BCI devices for the augmentation of rehabilitation therapy

Combining fNIRS and EEG to improve motor cortex activity classification during an imagined movement-based task

Included in Presentatio

A Survey of Brain Computer Interface Using Non-Invasive Methods

Research on Brain-Computer Interface (BCI) began in the 1970s and has increased in volume and diversified significantly since then. Today BCI is widely used for applications like assistive devices for physically challenged users, mental state monitoring, input devices for hands-free applications, marketing, education, security, games and entertainment. This article explores the advantages and disadvantages of invasive and non-invasive BCI technologies and focuses on use cases of several non-invasive technologies, namely electroencephalogram (EEG), functional Magnetic Resonance Imaging (fMRI), Near Infrared Spectroscopy (NIRs) and hybrid systems

Speech Processes for Brain-Computer Interfaces

Speech interfaces have become widely used and are integrated in many applications and devices. However, speech interfaces require the user to produce intelligible speech, which might be hindered by loud environments, concern to bother bystanders or the general in- ability to produce speech due to disabilities. Decoding a usera s imagined speech instead of actual speech would solve this problem. Such a Brain-Computer Interface (BCI) based on imagined speech would enable fast and natural communication without the need to actually speak out loud. These interfaces could provide a voice to otherwise mute people. This dissertation investigates BCIs based on speech processes using functional Near In- frared Spectroscopy (fNIRS) and Electrocorticography (ECoG), two brain activity imaging modalities on opposing ends of an invasiveness scale. Brain activity data have low signal- to-noise ratio and complex spatio-temporal and spectral coherence. To analyze these data, techniques from the areas of machine learning, neuroscience and Automatic Speech Recog- nition are combined in this dissertation to facilitate robust classification of detailed speech processes while simultaneously illustrating the underlying neural processes. fNIRS is an imaging modality based on cerebral blood flow. It only requires affordable hardware and can be set up within minutes in a day-to-day environment. Therefore, it is ideally suited for convenient user interfaces. However, the hemodynamic processes measured by fNIRS are slow in nature and the technology therefore offers poor temporal resolution. We investigate speech in fNIRS and demonstrate classification of speech processes for BCIs based on fNIRS. ECoG provides ideal signal properties by invasively measuring electrical potentials artifact- free directly on the brain surface. High spatial resolution and temporal resolution down to millisecond sampling provide localized information with accurate enough timing to capture the fast process underlying speech production. This dissertation presents the Brain-to- Text system, which harnesses automatic speech recognition technology to decode a textual representation of continuous speech from ECoG. This could allow to compose messages or to issue commands through a BCI. While the decoding of a textual representation is unparalleled for device control and typing, direct communication is even more natural if the full expressive power of speech - including emphasis and prosody - could be provided. For this purpose, a second system is presented, which directly synthesizes neural signals into audible speech, which could enable conversation with friends and family through a BCI. Up to now, both systems, the Brain-to-Text and synthesis system are operating on audibly produced speech. To bridge the gap to the final frontier of neural prostheses based on imagined speech processes, we investigate the differences between audibly produced and imagined speech and present first results towards BCI from imagined speech processes. This dissertation demonstrates the usage of speech processes as a paradigm for BCI for the first time. Speech processes offer a fast and natural interaction paradigm which will help patients and healthy users alike to communicate with computers and with friends and family efficiently through BCIs

Towards simultaneous electroencephalography and functional near-infrared spectroscopy for improving diagnostic accuracy in prolonged disorders of consciousness: a healthy cohort study

Qualitative clinical assessments of the recovery of awareness after severe brain injury require an assessor to differentiate purposeful behaviour from spontaneous behaviour. As many such behaviours are minimal and inconsistent, behavioural assessments are susceptible to diagnostic errors. Advanced neuroimaging tools such as functional magnetic resonance imaging and electroencephalography (EEG) can bypass behavioural responsiveness and reveal evidence of covert awareness and cognition within the brains of some patients, thus providing a means for more accurate diagnoses, more accurate prognoses, and, in some instances, facilitated communication. As each individual neuroimaging method has its own advantages and disadvantages (e.g., signal resolution, accessibility, etc.), this thesis studies on healthy individuals a burgeoning technique of non-invasive electrical and optical neuroimaging—simultaneous EEG and functional near-infrared spectroscopy (fNIRS)—that can be applied at the bedside. Measuring reliable covert behaviours is correlated with participant engagement, instrumental sensitivity and the accurate localisation of responses, aspects which are further addressed over three studies. Experiment 1 quantifies the typical EEG changes in response to covert commands in the absence and presence of an object. This is investigated to determine whether a goal-directed task can yield greater EEG control accuracy over simple monotonous imagined single-joint actions. Experiment 2 characterises frequency domain NIRS changes in response to overt and covert hand movements. A method for reconstructing haemodynamics using the less frequently investigated phase parameter is outlined and the impact of noise contaminated NIRS measurements are discussed. Furthermore, classification performances between frequency-domain and continuous-wave-like signals are compared. Experiment 3 lastly applies these techniques to determine the potential of simultaneous EEG-fNIRS classification. Here a sparse channel montage that would ultimately favour clinical utility is used to demonstrate whether such a hybrid method containing rich spatial and temporal information can improve the classification of covert responses in comparison to unimodal classification of signals. The findings and discussions presented within this thesis identify a direction for future research in order to more accurately translate the brain state of patients with a prolonged disorder of consciousness

Analysis of Human Gait Using Hybrid EEG-fNIRS-Based BCI System: A Review

Human gait is a complex activity that requires high coordination between the central nervous system, the limb, and the musculoskeletal system. More research is needed to understand the latter coordination\u27s complexity in designing better and more effective rehabilitation strategies for gait disorders. Electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) are among the most used technologies for monitoring brain activities due to portability, non-invasiveness, and relatively low cost compared to others. Fusing EEG and fNIRS is a well-known and established methodology proven to enhance brain–computer interface (BCI) performance in terms of classification accuracy, number of control commands, and response time. Although there has been significant research exploring hybrid BCI (hBCI) involving both EEG and fNIRS for different types of tasks and human activities, human gait remains still underinvestigated. In this article, we aim to shed light on the recent development in the analysis of human gait using a hybrid EEG-fNIRS-based BCI system. The current review has followed guidelines of preferred reporting items for systematic reviews and meta-Analyses (PRISMA) during the data collection and selection phase. In this review, we put a particular focus on the commonly used signal processing and machine learning algorithms, as well as survey the potential applications of gait analysis. We distill some of the critical findings of this survey as follows. First, hardware specifications and experimental paradigms should be carefully considered because of their direct impact on the quality of gait assessment. Second, since both modalities, EEG and fNIRS, are sensitive to motion artifacts, instrumental, and physiological noises, there is a quest for more robust and sophisticated signal processing algorithms. Third, hybrid temporal and spatial features, obtained by virtue of fusing EEG and fNIRS and associated with cortical activation, can help better identify the correlation between brain activation and gait. In conclusion, hBCI (EEG + fNIRS) system is not yet much explored for the lower limb due to its complexity compared to the higher limb. Existing BCI systems for gait monitoring tend to only focus on one modality. We foresee a vast potential in adopting hBCI in gait analysis. Imminent technical breakthroughs are expected using hybrid EEG-fNIRS-based BCI for gait to control assistive devices and Monitor neuro-plasticity in neuro-rehabilitation. However, although those hybrid systems perform well in a controlled experimental environment when it comes to adopting them as a certified medical device in real-life clinical applications, there is still a long way to go

Hybrid EEG-fNIRS asynchronous brain-computer interface for multiple motor tasks

Non-invasive Brain-Computer Interfaces (BCI) have demonstrated great promise for neuroprosthetics and assistive devices. Here we aim to investigate methods to combine Electroencephalography (EEG) and functional Near-Infrared Spectroscopy (fNIRS) in an asynchronous Sensory Motor rhythm (SMR)-based BCI. We attempted to classify 4 different executed movements, namely, Right-Arm—Left-Arm—Right-Hand—Left-Hand tasks. Previous studies demonstrated the benefit of EEG-fNIRS combination. However, since normally fNIRS hemodynamic response shows a long delay, we investigated new features, involving slope indicators, in order to immediately detect changes in the signals. Moreover, Common Spatial Patterns (CSPs) have been applied to both EEG and fNIRS signals. 15 healthy subjects took part in the experiments and since 25 trials per class were available, CSPs have been regularized with information from the entire population of participants and optimized using genetic algorithms. The different features have been compared in terms of performance and the dynamic accuracy over trials shows that the introduced methods diminish the fNIRS delay in the detection of changes