Reading Your Own Mind: Dynamic Visualization of Real-Time Neural Signals

Brain Computer Interfaces: BCI) systems which allow humans to control external devices directly from brain activity, are becoming increasingly popular due to dramatic advances in the ability to both capture and interpret brain signals. Further advancing BCI systems is a compelling goal both because of the neurophysiology insights gained from deriving a control signal from brain activity and because of the potential for direct brain control of external devices in applications such as brain injury recovery, human prosthetics, and robotics. The dynamic and adaptive nature of the brain makes it difficult to create classifiers or control systems that will remain effective over time. However it is precisely these qualities that offer the potential to use feedback to build on simple features and create complex control features that are robust over time. This dissertation presents work that addresses these opportunities for the specific case of Electrocorticography: ECoG) recordings from clinical epilepsy patients. First, queued patient tasks were used to explore the predictive nature of both local and global features of the ECoG signal. Second, an algorithm was developed and tested for estimating the most informative features from naive observations of ECoG signal. Third, a software system was built and tested that facilitates real-time visualizations of ECoG signal patients and allows ECoG epilepsy patients to engage in an interactive BCI control feature screening process

Detecting single-trial EEG evoked potential using a wavelet domain linear mixed model: application to error potentials classification

Objective. The main goal of this work is to develop a model for multi-sensor signals such as MEG or EEG signals, that accounts for the inter-trial variability, suitable for corresponding binary classification problems. An important constraint is that the model be simple enough to handle small size and unbalanced datasets, as often encountered in BCI type experiments. Approach. The method involves linear mixed effects statistical model, wavelet transform and spatial filtering, and aims at the characterization of localized discriminant features in multi-sensor signals. After discrete wavelet transform and spatial filtering, a projection onto the relevant wavelet and spatial channels subspaces is used for dimension reduction. The projected signals are then decomposed as the sum of a signal of interest (i.e. discriminant) and background noise, using a very simple Gaussian linear mixed model. Main results. Thanks to the simplicity of the model, the corresponding parameter estimation problem is simplified. Robust estimates of class-covariance matrices are obtained from small sample sizes and an effective Bayes plug-in classifier is derived. The approach is applied to the detection of error potentials in multichannel EEG data, in a very unbalanced situation (detection of rare events). Classification results prove the relevance of the proposed approach in such a context. Significance. The combination of linear mixed model, wavelet transform and spatial filtering for EEG classification is, to the best of our knowledge, an original approach, which is proven to be effective. This paper improves on earlier results on similar problems, and the three main ingredients all play an important role

Towards simultaneous electroencephalography and functional near-infrared spectroscopy for improving diagnostic accuracy in prolonged disorders of consciousness: a healthy cohort study

Qualitative clinical assessments of the recovery of awareness after severe brain injury require an assessor to differentiate purposeful behaviour from spontaneous behaviour. As many such behaviours are minimal and inconsistent, behavioural assessments are susceptible to diagnostic errors. Advanced neuroimaging tools such as functional magnetic resonance imaging and electroencephalography (EEG) can bypass behavioural responsiveness and reveal evidence of covert awareness and cognition within the brains of some patients, thus providing a means for more accurate diagnoses, more accurate prognoses, and, in some instances, facilitated communication. As each individual neuroimaging method has its own advantages and disadvantages (e.g., signal resolution, accessibility, etc.), this thesis studies on healthy individuals a burgeoning technique of non-invasive electrical and optical neuroimaging—simultaneous EEG and functional near-infrared spectroscopy (fNIRS)—that can be applied at the bedside. Measuring reliable covert behaviours is correlated with participant engagement, instrumental sensitivity and the accurate localisation of responses, aspects which are further addressed over three studies. Experiment 1 quantifies the typical EEG changes in response to covert commands in the absence and presence of an object. This is investigated to determine whether a goal-directed task can yield greater EEG control accuracy over simple monotonous imagined single-joint actions. Experiment 2 characterises frequency domain NIRS changes in response to overt and covert hand movements. A method for reconstructing haemodynamics using the less frequently investigated phase parameter is outlined and the impact of noise contaminated NIRS measurements are discussed. Furthermore, classification performances between frequency-domain and continuous-wave-like signals are compared. Experiment 3 lastly applies these techniques to determine the potential of simultaneous EEG-fNIRS classification. Here a sparse channel montage that would ultimately favour clinical utility is used to demonstrate whether such a hybrid method containing rich spatial and temporal information can improve the classification of covert responses in comparison to unimodal classification of signals. The findings and discussions presented within this thesis identify a direction for future research in order to more accurately translate the brain state of patients with a prolonged disorder of consciousness

Graph analysis of functional brain networks: practical issues in translational neuroscience

The brain can be regarded as a network: a connected system where nodes, or units, represent different specialized regions and links, or connections, represent communication pathways. From a functional perspective communication is coded by temporal dependence between the activities of different brain areas. In the last decade, the abstract representation of the brain as a graph has allowed to visualize functional brain networks and describe their non-trivial topological properties in a compact and objective way. Nowadays, the use of graph analysis in translational neuroscience has become essential to quantify brain dysfunctions in terms of aberrant reconfiguration of functional brain networks. Despite its evident impact, graph analysis of functional brain networks is not a simple toolbox that can be blindly applied to brain signals. On the one hand, it requires a know-how of all the methodological steps of the processing pipeline that manipulates the input brain signals and extract the functional network properties. On the other hand, a knowledge of the neural phenomenon under study is required to perform physiological-relevant analysis. The aim of this review is to provide practical indications to make sense of brain network analysis and contrast counterproductive attitudes

Modified Spatio-Temporal Matched Filtering for Brain Responses Classification

In this article, we apply the method of spatio-temporal filtering (STF) to electroencephalographic (EEG) data processing for brain responses classification. The method operates similarly to linear discriminant analysis (LDA) but contrary to most applied classifiers, it uses the whole recorded EEG signal as a source of information instead of the precisely selected brain responses, only. This way it avoids the limitations of LDA and improves the classification accuracy. We emphasize the significance of the STF learning phase. To preclude the negative influence of super–Gaussian artifacts on accomplishment of this phase, we apply the discrete cosine transform (DCT) based method for their rejection. Later, we estimate the noise covariance matrix using all data available, and we improve the STF template construction. The further modifications are related with the constructed filters operation and consist in the changes of the STF interpretation rules. Consequently, a new tool for evoked potentials (EPs) classification has been developed. Applied to the analysis of signals stored in a publicly available database, prepared for the assessment of modern algorithms aimed in EPs detection (in the frames of the 2019 IFMBE Scientific Challenge), it allowed to achieve the second best result, very close to the best one, and significantly better than the ones achieved by other contestants of the challeng

Breathing is coupled with voluntary initiation of mental imagery

Previous research has suggested that bodily signals from internal organs are associated with diverse cortical and subcortical processes involved in sensory-motor functions, beyond homeostatic reflexes. For instance, a recent study demonstrated that the preparation and execution of voluntary actions, as well as its underlying neural activity, are coupled with the breathing cycle. In the current study, we investigated whether such breathing-action coupling is limited to voluntary motor action or whether it is also present for mental actions not involving any overt bodily movement. To answer this question, we recorded electroencephalography (EEG), electromyography (EMG), and respiratory signals while participants were conducting a voluntary action paradigm including self-initiated motor execution (ME), motor imagery (MI), and visual imagery (VI) tasks. We observed that the voluntary initiation of ME, MI, and VI are similarly coupled with the respiration phase. In addition, EEG analysis revealed the existence of readiness potential (RP) waveforms in all three tasks (i.e., ME, MI, VI), as well as a coupling between the RP amplitude and the respiratory phase. Our findings show that the voluntary initiation of both imagined and overt action is coupled with respiration, and further suggest that the breathing system is involved in preparatory processes of voluntary action by contributing to the temporal decision of when to initiate the action plan, regardless of whether this culminates in overt movements

Menetelmiä MEG:hen ja liikkeen kuvitteluun perustuviin aivokäyttöliittymiin

Brain–computer interfaces (BCI) are systems that translate the user's brain activity into commands for external devices in real time. Magnetoencephalography (MEG) measures electromagnetic brain activity noninvasively and can be used in BCIs. The aim of this thesis was to develop an MEG-based BCI for decoding hand motor imagery. The BCI could eventually serve as a therapeutic method for patients recovering from e.g.cerebral stroke. Here, we validated machine-learning methods for decoding motor imagery (MI)-related brain activity with healthy subjects' MEG measurements. In addition, we studied the effect of different BCI feedback modalities on the subjects' brain function related to MI.In Study I, we compared feature extraction methods for classifying left- vs right-hand MI, and MI vs rest. We found that spatial filtering and further extraction of bandpower features yielded better classification accuracy than time–frequency features extracted from MEG channels above the parietal area. Furthermore, prior spatial filtering improved the discrimination capability of time–frequency features.The training data for a BCI are typically collected in the beginning of each measurement session. However, as this can be time-consuming and exhausting for patients, data from other subjects' measurements could be used for training as well. In Study II, methods for across-subject classification of MI were compared. The results showed that a classifier based on multi-task learning with a l2,1-norm regularized logistic regression was the best method for across-subject decoding for both MEG and electroencephalography (EEG). In Study II, we also compared the decoding results of simultaneously measured EEG and MEG data, and investigated whether MEG responses to passive hand movements could be used to train a classifier to detect MI. MEG yielded slightly better results than EEG. Training the classifiers with the subject's own or other subjects' passive movements did not result in high accuracy. Passive movements should thus not be used for calibrating an MI-BCI.In Study III, we investigated how the amplitude of sensorimotor rhythms (SMR) changes while the subjects practise hand MI with a BCI. We compared the effect of visual and proprioceptive feedback on brain functional changes during a single measurement session. In subjects receiving proprioceptive feedback, the power of SMR increased linearly over the session in motor cortical regions, while similar effect was not observed in subjects receiving purely visual feedback. According to these results, proprioceptive feedback should be preferred over visual feedback especially in BCIs aiming at recovery of hand functions.The methods presented in this thesis are suitable for an MEG-based BCI. The decoding results can be used as a benchmark when developing classifiers specifically for MI-related MEG data.Aivokäyttöliittymien avulla voidaan ohjata ulkoisia laitteita käyttäen aivoista mitattuja signaaleja. Magnetoenkefalografia (MEG) mittaa aivojen toimintaa kajoamattomasti ja sitä voidaan käyttää myös aivokäyttöliittymissä. Väitöskirjan tavoitteena oli kehittää käden liikkeen kuvittelua luokitteleva MEG-aivokäyttöliittymä, jota voidaan myöhemmin käyttää aivoinfarktipotilaiden kuntoutukseen. Työssä validoitiin terveiden koehenkilöiden MEG-mittausten perusteella koneoppimismenetelmiä aivokäyttöliittymiin sekä tutkittiin, miten eri palautemodaliteetit vaikuttavat aivotoimintaan koehenkilöiden opetellessa käyttämään aivokäyttöliittymää.Ensimmäisessä osatyössä vertailtiin piirteenirrotusmenetelmiä, joita käytetään erottamaan toisistaan vasemman ja oikean käden kuvitteluun sekä liikkeen kuvitteluun ja lepotilaan liittyvät MEG-signaalit. Spatiaalisesti suodatettujen signaalien teho luokittelupiirteenä tuotti parempia luokittelutarkkuuksia kuin parietaalisista MEG-kanavista mitatut aika-taajuuspiirteet. Edeltävä spatiaalinen suodatus paransi myös aika-taajuuspiirteiden erottelukykyä luokittelutehtävissä.Aivokäyttöliittymän opetusdata kerätään yleensä kunkin mittauskerran alussa. Koska tämä voi olla hidasta ja uuvuttavaa potilaille, opetusdatana voidaan käyttää myös muilta henkilöiltä mitattuja signaaleja. Toisessa osatyössä vertailtiin koehenkilöiden väliseen luokitteluun soveltuvia menetelmiä. Monitehtäväoppimista ja l2,1-regularisoitua logistista regressiota käyttävä luokittelija soveltui tähän parhaiten.Toisessa osatyössä vertailtiin myös MEG:n ja elektroenkefalografian (EEG) tuottamia luokittelutuloksia, sekä tutkittiin voidaanko passiivisten käden liikkeiden aiheuttamia MEG-vasteita käyttää liikkeen kuvittelua tunnistavien luokittelijoiden opetukseen. MEG tuotti hieman parempia tuloksia kuin EEG. Luokittelijoiden opetus koehenkilöiden omilla tai muiden koehenkilöiden passiiviliikkeillä ei tuottanut hyviä luokittelutuloksia.Passiiviliikkeitä ei siis tulisi käyttää liikkeen kuvittelua tunnistavan aivo-käyttöliittymän kalibrointiin.Kolmannessa osatyössä tutkittiin, miten sensorimotoristen rytmien (SMR) amplitudi muuttuu koehenkilöiden harjoitellessa käden liikkeiden kuvittelua aivokäyttöliittymän avulla. Työssä vertailtiin visuaalisen ja proprioseptiivisen palautteen aiheuttamia SMR:n muutoksia yhden harjoituskerran aikana. Proprioseptiivista palautetta saaneilla koehenkilöillä SMR:n teho kasvoi harjoittelun aikana lineaarisesti liikkeitä koordinoivilla aivoalueilla. Visuaalista palautetta saaneilla tätä ilmiötä ei havaittu. Propriosep-tiivista palautetta tulisi siten käyttää visuaalisen sijaan erityisesti käden liikkeiden kuntoutukseen tähtäävissä aivokäyttöliittymissä.Esitettyjä menetelmiä voidaan käyttää MEG:hen perustuvissa aivokäyttöliittymissä. Luokittelutuloksia voidaan käyttää vertailukohtana kehitettäessä liikkeen kuvitteluun liittyvän MEG-datan luokittelijoita

Brain signal recognition using deep learning

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel UniversityBrain Computer Interface (BCI) has the potential to offer a new generation of applications independent of muscular activity and controlled by the human brain. Brain imaging technologies are used to transfer the cognitive tasks into control commands for a BCI system. The electroencephalography (EEG) technology serves as the best available non-invasive solution for extracting signals from the brain. On the other hand, speech is the primary means of communication, but for patients suffering from locked-in syndrome, there is no easy way to communicate. Therefore, an ideal communication system for locked-in patients is a thought-to-speech BCI system. This research aims to investigate methods for the recognition of imagined speech from EEG signals using deep learning techniques. In order to design an optimal imagined speech recognition BCI, variety of issues have been solved. These include 1) proposing new feature extraction and classification framework for recognition of imagined speech from EEG signals, 2) grammatical class recognition of imagined words from EEG signals, 3) discriminating different cognitive tasks associated with speech in the brain such as overt speech, covert speech, and visual imagery. In this work machine learning, deep learning methods were used to analyze EEG signals. For recognition of imagined speech from EEG signals, a new EEG database was collected while the participants mentally spoke (imagined speech) the presented words. Along with imagined speech, EEG data was recorded for visual imagery (imagining a scene or an image) and overt speech (verbal speech). Spectro-temporal and spatio-temporal domain features were investigated for the classification of imagined words from EEG signals. Further, a deep learning framework using the convolutional network and attention mechanism was implemented for learning features in the spatial, temporal, and spectral domains. The method achieved a recognition rate of 76.6% for three binary word pairs. These experiments show that deep learning algorithms are ideal for imagined speech recognition from EEG signals due to their ability to interpret features from non-linear and non-stationary signals. Grammatical classes of imagined words from EEG signals were also recognized using a multi-channel convolution network framework. This method was extended to a multi-level recognition system for multi-class classification of imagined words which achieved an accuracy of 52.9% for 10 words, which is much better in comparison to previous work. In order to investigate the difference between imagined speech with verbal speech and visual imagery from EEG signals, we used multivariate pattern analysis (MVPA). MVPA provided the time segments when the neural oscillation for the different cognitive tasks was linearly separable. Further, frequencies that result in most discrimination between the different cognitive tasks were also explored. A framework was proposed to discriminate two cognitive tasks based on the spatio-temporal patterns in EEG signals. The proposed method used the K-means clustering algorithm to find the best electrode combination and convolutional-attention network for feature extraction and classification. The proposed method achieved a high recognition rate of 82.9% and 77.7%. The results in this research suggest that a communication based BCI system can be designed using deep learning methods. Further, this work add knowledge to the existing work in the field of communication based BCI system

Electroencephalograph (EEG) signal processing techniques for motor imagery Brain Computer interface systems

Brain-Computer Interface (BCI) system provides a channel for the brain to control external devices using electrical activities of the brain without using the peripheral nervous system. These BCI systems are being used in various medical applications, for example controlling a wheelchair and neuroprosthesis devices for the disabled, thereby assisting them in activities of daily living. People suffering from Amyotrophic Lateral Sclerosis (ALS), Multiple Sclerosis and completely locked in are unable to perform any body movements because of the damage of the peripheral nervous system, but their cognitive function is still intact. BCIs operate external devices by acquiring brain signals and converting them to control commands to operate external devices. Motor-imagery (MI) based BCI systems, in particular, are based on the sensory-motor rhythms which are generated by the imagination of body limbs. These signals can be decoded as control commands in BCI application. Electroencephalogram (EEG) is commonly used for BCI applications because it is non-invasive. The main challenges of decoding the EEG signal are because it is non-stationary and has a low spatial resolution. The common spatial pattern algorithm is considered to be the most effective technique for discrimination of spatial filter but is easily affected by the presence of outliers. Therefore, a robust algorithm is required for extraction of discriminative features from the motor imagery EEG signals. This thesis mainly aims in developing robust spatial filtering criteria which are effective for classification of MI movements. We have proposed two approaches for the robust classification of MI movements. The first approach is for the classification of multiclass MI movements based on the thinICA (Independent Component Analysis) and mCSP (multiclass Common Spatial Pattern Filter) method. The observed results indicate that these approaches can be a step towards the development of robust feature extraction for MI-based BCI system. The main contribution of the thesis is the second criterion, which is based on Alpha- Beta logarithmic-determinant divergence for the classification of two class MI movements. A detailed study has been done by obtaining a link between the AB log det divergence and CSP criterion. We propose a scaling parameter to enable a similar way for selecting the respective filters like the CSP algorithm. Additionally, the optimization of the gradient of AB log-det divergence for this application was also performed. The Sub-ABLD (Subspace Alpha-Beta Log-Det divergence) algorithm is proposed for the discrimination of two class MI movements. The robustness of this algorithm is tested with both the simulated and real data from BCI competition dataset. Finally, the resulting performances of the proposed algorithms have been favorably compared with other existing algorithms

The effects of mental training on brain computer interface performance with distractions

The overall success of a brain computer interface (BCI) is largely dependent on the features used to make decisions. Noise in the electroencephalography (EEG) increases the difficulty of acquiring meaningful features. Previous literature suggests teaching subjects meditation and relaxation techniques may improve features relevant to BCI operation. The purpose of this study was to investigate performance on several cognitive protocols for both individuals who use meditation techniques and those who do not use these techniques. Both groups were given a motor imagery based BCI protocol, a P300 speller BCI, a verbal learning task, and an N-back test. No significant difference in performance was found between meditation and control groups. Our research does suggest however, significant differences for the P300 and motor imagery protocols may be found if a larger group (\u3e20 subjects per class) is recruited