Advancing Pattern Recognition Techniques for Brain-Computer Interfaces: Optimizing Discriminability, Compactness, and Robustness

In dieser Dissertation formulieren wir drei zentrale Zielkriterien zur systematischen Weiterentwicklung der Mustererkennung moderner Brain-Computer Interfaces (BCIs). Darauf aufbauend wird ein Rahmenwerk zur Mustererkennung von BCIs entwickelt, das die drei Zielkriterien durch einen neuen Optimierungsalgorithmus vereint. Darüber hinaus zeigen wir die erfolgreiche Umsetzung unseres Ansatzes für zwei innovative BCI Paradigmen, für die es bisher keine etablierte Mustererkennungsmethodik gibt

Real-time decoding of question-and-answer speech dialogue using human cortical activity.

Natural communication often occurs in dialogue, differentially engaging auditory and sensorimotor brain regions during listening and speaking. However, previous attempts to decode speech directly from the human brain typically consider listening or speaking tasks in isolation. Here, human participants listened to questions and responded aloud with answers while we used high-density electrocorticography (ECoG) recordings to detect when they heard or said an utterance and to then decode the utterance's identity. Because certain answers were only plausible responses to certain questions, we could dynamically update the prior probabilities of each answer using the decoded question likelihoods as context. We decode produced and perceived utterances with accuracy rates as high as 61% and 76%, respectively (chance is 7% and 20%). Contextual integration of decoded question likelihoods significantly improves answer decoding. These results demonstrate real-time decoding of speech in an interactive, conversational setting, which has important implications for patients who are unable to communicate

Probabilistic graphical models for brain computer interfaces

Brain computer interfaces (BCI) are systems that aim to establish a new communication path for subjects who su er from motor disabilities, allowing interaction with the environment through computer systems. BCIs make use of a diverse group of physiological phenomena recorded using electrodes placed on the scalp (Electroencephalography, EEG) or electrodes placed directly over the brain cortex (Electrocorticography, ECoG). One commonly used phenomenon is the activity observed in specific areas of the brain in response to external events, called Event Related Potentials (ERP). Among those, a type of response called P300 is the most used phenomenon. The P300 has found application in spellers that make use of the brain's response to the presentation of a sequence of visual stimuli. Another commonly used phenomenon is the synchronization or desynchronization of brain rhythms during the execution or imagination of a motor task, which can be used to differentiate between two or more subject intentions. In the most basic scenario, a BCI system calculates the differences in the power of the EEG rhythms during execution of different tasks. Based on those differences, the BCI decides which task has been executed (e.g., motor imagination of left or right hand). Current approaches are mainly based on machine learning techniques that learn the distribution of the power values of the brain signals for each of the possible classes. In this thesis, making use of EEG and ECoG recording methods, we propose the use of probabilistic graphical models for brain computer interfaces. In the case of ERPs, in particular P300-based spellers, we propose the incorporation of language models at the level of words to increase significantly the performance of the spelling system. The proposed framework allows also the incorporation of different methods that take into account language models based on n-grams, all of this in an integrated structure whose parameters can be efficiently learned. In the context of execution or imagination of motor tasks, we propose techniques that take into account the temporal structure of the signals. Stochastic processes that model temporal dynamics of the brain signals in different frequency bands such as non-parametric Bayesian hidden Markov models are proposed in order to solve the problem of selection of the number of brain states during the execution of motor tasks as well as the selection of the number of components used to model the distribution of the brain signals. Following up on the same line of thought, hidden conditional random fields are proposed for classification of synchronous motor tasks. The combination of hidden states with the discriminative power of conditional random fields is shown to increase the classification performance of imaginary motor movements. In the context of asynchronous BCIs, we propose a method based on latent dynamic conditional random fields that is capable of modeling the internal temporal dynamics related to the generation of the brain signals, and external brain dynamics related to the execution of different mental tasks. Finally, in the context of asynchronous BCIs a model based on discriminative graphical models is presented for continuous classification of finger movements from ECoG data. We show that the incorporation of temporal dynamics of the brain signals in the classification stages increases significantly the classification accuracy of different mental states which can lead to a more effective interaction between the subject and the environment

Developing robust movement decoders for local field potentials

textBrain Computer Interfaces (BCI) are devices that translate acquired neural signals to command and control signals. Applications of BCI include neural rehabilitation and neural prosthesis (thought controlled wheelchair, thought controlled speller etc.) to aid patients with disabilities and to augment human computer interaction. A successful practical BCI requires a faithful acquisition modality to record high quality neural signals; a signal processing system to construct appropriate features from these signals; and an algorithm to translate these features to appropriate outputs. Intracortical recordings like local field potentials provide reliable high SNR signals over long periods and suit BCI applications well. However, the non-stationarity of neural signals poses a challenge in robust decoding of subject behavior. Most BCI research focuses either on developing daily re-calibrated decoders that require exhaustive training sessions; or on providing cross-validation results. Such results ignore the variation of signal characteristics over different sessions and provide an optimistic estimate of BCI performance. Specifically, traditional BCI algorithms fail to perform at the same level on chronological data recordings. Neural signals are susceptible to variations in signal characteristics due to changes in subject behavior and learning, and variability in electrode characteristics due to tissue interactions. While training day-specific BCI overcomes signal variability, BCI re-training causes user frustration and exhaustion. This dissertation presents contributions to solve these challenges in BCI research. Specifically, we developed decoders trained on a single recording session and applied them on subsequently recorded sessions. This strategy evaluates BCI in a practical scenario with a potential to alleviate BCI user frustration without compromising performance. The initial part of the dissertation investigates extracting features that remain robust to changes in neural signal over several days of recordings. It presents a qualitative feature extraction technique based on ranking the instantaneous power of multichannel data. These qualitative features remain robust to outliers and changes in the baseline of neural recordings, while extracting discriminative information. These features form the foundation in developing robust decoders. Next, this dissertation presents a novel algorithm based on the hypothesis that multiple neural spatial patterns describe the variation in behavior. The presented algorithm outperforms the traditional methods in decoding over chronological recordings. Adapting such a decoder over multiple recording sessions (over 6 weeks) provided > 90% accuracy in decoding eight movement directions. In comparison, performance of traditional algorithms like Common Spatial Patterns deteriorates to 16% over the same time. Over time, adaptation reinforces some spatial patterns while diminishing others. Characterizing these spatial patterns reduces model complexity without user input, while retaining the same accuracy levels. Lastly, this dissertation provides an algorithm that overcomes the variation in recording quality. Chronic electrode implantation causes changes in signal-to-noise ratio (SNR) of neural signals. Thus, some signals and their corresponding features available during training become unavailable during testing and vice-versa. The proposed algorithm uses prior knowledge on spatial pattern evolution to estimate unknown neural features. This algorithm overcomes SNR variations and provides up to 93% decoding of eight movement directions over 6 weeks. Since model training requires only one session, this strategy reduces user frustration. In a practical closed-loop BCI, the user learns to produce stable spatial patterns, which improves performance of the proposed algorithms.Electrical and Computer Engineerin

Effective EEG analysis for advanced AI-driven motor imagery BCI systems

Developing effective signal processing for brain-computer interfaces (BCIs) and brain-machine interfaces (BMIs) involves factoring in three aspects of functionality: classification performance, execution time, and the number of data channels used. The contributions in this thesis are centered on these three issues. Contributions are focused on the classification of motor imagery (MI) data, which is generated during imagined movements. Typically, EEG time-series data is segmented for data augmentation or to mimic buffering that happens in an online BCI. A multi-segment decision fusion approach is presented, which takes consecutive temporal segments of EEG data, and uses decision fusion to boost classification performance. It was computationally lightweight and improved the performance of four conventional classifiers. Also, an analysis of the contributions of electrodes from different scalp regions is presented, and a subset of channels is recommended. Sparse learning (SL) classifiers have exhibited strong classification performance in the literature. However, they are computationally expensive. To reduce the test-set execution times, a novel EEG classification pipeline consisting of a genetic-algorithm (GA) for channel selection and a dictionary-based SL module for classification, called GABSLEEG, is presented. Subject-specific channel selection was carried out, in which the channels are selected based on training data from the subject. Using the GA-recommended subset of EEG channels reduced the execution time by 60% whilst preserving classification performance. Although subject-specific channel selection is widely used in the literature, effective subject-independent channel selection, in which channels are detected using data from other subjects, is an ideal aim because it leads to lower training latency and reduces the number of electrodes needed. A novel convolutional neural network (CNN)-based subject-independent channels selection method is presented, called the integrated channel selection (ICS) layer. It performed on-a-par with or better than subject-specific channel selection. It was computationally efficient, operating 12-17 times faster than the GA channel selection module. The ICS layer method was versatile, performing well with two different CNN architectures and datasets.Developing effective signal processing for brain-computer interfaces (BCIs) and brain-machine interfaces (BMIs) involves factoring in three aspects of functionality: classification performance, execution time, and the number of data channels used. The contributions in this thesis are centered on these three issues. Contributions are focused on the classification of motor imagery (MI) data, which is generated during imagined movements. Typically, EEG time-series data is segmented for data augmentation or to mimic buffering that happens in an online BCI. A multi-segment decision fusion approach is presented, which takes consecutive temporal segments of EEG data, and uses decision fusion to boost classification performance. It was computationally lightweight and improved the performance of four conventional classifiers. Also, an analysis of the contributions of electrodes from different scalp regions is presented, and a subset of channels is recommended. Sparse learning (SL) classifiers have exhibited strong classification performance in the literature. However, they are computationally expensive. To reduce the test-set execution times, a novel EEG classification pipeline consisting of a genetic-algorithm (GA) for channel selection and a dictionary-based SL module for classification, called GABSLEEG, is presented. Subject-specific channel selection was carried out, in which the channels are selected based on training data from the subject. Using the GA-recommended subset of EEG channels reduced the execution time by 60% whilst preserving classification performance. Although subject-specific channel selection is widely used in the literature, effective subject-independent channel selection, in which channels are detected using data from other subjects, is an ideal aim because it leads to lower training latency and reduces the number of electrodes needed. A novel convolutional neural network (CNN)-based subject-independent channels selection method is presented, called the integrated channel selection (ICS) layer. It performed on-a-par with or better than subject-specific channel selection. It was computationally efficient, operating 12-17 times faster than the GA channel selection module. The ICS layer method was versatile, performing well with two different CNN architectures and datasets

Machine Learning Methods for Image Analysis in Medical Applications, from Alzheimer\u27s Disease, Brain Tumors, to Assisted Living

Healthcare has progressed greatly nowadays owing to technological advances, where machine learning plays an important role in processing and analyzing a large amount of medical data. This thesis investigates four healthcare-related issues (Alzheimer\u27s disease detection, glioma classification, human fall detection, and obstacle avoidance in prosthetic vision), where the underlying methodologies are associated with machine learning and computer vision. For Alzheimer’s disease (AD) diagnosis, apart from symptoms of patients, Magnetic Resonance Images (MRIs) also play an important role. Inspired by the success of deep learning, a new multi-stream multi-scale Convolutional Neural Network (CNN) architecture is proposed for AD detection from MRIs, where AD features are characterized in both the tissue level and the scale level for improved feature learning. Good classification performance is obtained for AD/NC (normal control) classification with test accuracy 94.74%. In glioma subtype classification, biopsies are usually needed for determining different molecular-based glioma subtypes. We investigate non-invasive glioma subtype prediction from MRIs by using deep learning. A 2D multi-stream CNN architecture is used to learn the features of gliomas from multi-modal MRIs, where the training dataset is enlarged with synthetic brain MRIs generated by pairwise Generative Adversarial Networks (GANs). Test accuracy 88.82% has been achieved for IDH mutation (a molecular-based subtype) prediction. A new deep semi-supervised learning method is also proposed to tackle the problem of missing molecular-related labels in training datasets for improving the performance of glioma classification. In other two applications, we also address video-based human fall detection by using co-saliency-enhanced Recurrent Convolutional Networks (RCNs), as well as obstacle avoidance in prosthetic vision by characterizing obstacle-related video features using a Spiking Neural Network (SNN). These investigations can benefit future research, where artificial intelligence/deep learning may open a new way for real medical applications

Predictive decoding of neural data

In the last five decades the number of techniques available for non-invasive functional imaging has increased dramatically. Researchers today can choose from a variety of imaging modalities that include EEG, MEG, PET, SPECT, MRI, and fMRI. This doctoral dissertation offers a methodology for the reliable analysis of neural data at different levels of investigation. By using statistical learning algorithms the proposed approach allows single-trial analysis of various neural data by decoding them into variables of interest. Unbiased testing of the decoder on new samples of the data provides a generalization assessment of decoding performance reliability. Through consecutive analysis of the constructed decoder\u27s sensitivity it is possible to identify neural signal components relevant to the task of interest. The proposed methodology accounts for covariance and causality structures present in the signal. This feature makes it more powerful than conventional univariate methods which currently dominate the neuroscience field. Chapter 2 describes the generic approach toward the analysis of neural data using statistical learning algorithms. Chapter 3 presents an analysis of results from four neural data modalities: extracellular recordings, EEG, MEG, and fMRI. These examples demonstrate the ability of the approach to reveal neural data components which cannot be uncovered with conventional methods. A further extension of the methodology, Chapter 4 is used to analyze data from multiple neural data modalities: EEG and fMRI. The reliable mapping of data from one modality into the other provides a better understanding of the underlying neural processes. By allowing the spatial-temporal exploration of neural signals under loose modeling assumptions, it removes potential bias in the analysis of neural data due to otherwise possible forward model misspecification. The proposed methodology has been formalized into a free and open source Python framework for statistical learning based data analysis. This framework, PyMVPA, is described in Chapter 5