Developing robust movement decoders for local field potentials

textBrain Computer Interfaces (BCI) are devices that translate acquired neural signals to command and control signals. Applications of BCI include neural rehabilitation and neural prosthesis (thought controlled wheelchair, thought controlled speller etc.) to aid patients with disabilities and to augment human computer interaction. A successful practical BCI requires a faithful acquisition modality to record high quality neural signals; a signal processing system to construct appropriate features from these signals; and an algorithm to translate these features to appropriate outputs. Intracortical recordings like local field potentials provide reliable high SNR signals over long periods and suit BCI applications well. However, the non-stationarity of neural signals poses a challenge in robust decoding of subject behavior. Most BCI research focuses either on developing daily re-calibrated decoders that require exhaustive training sessions; or on providing cross-validation results. Such results ignore the variation of signal characteristics over different sessions and provide an optimistic estimate of BCI performance. Specifically, traditional BCI algorithms fail to perform at the same level on chronological data recordings. Neural signals are susceptible to variations in signal characteristics due to changes in subject behavior and learning, and variability in electrode characteristics due to tissue interactions. While training day-specific BCI overcomes signal variability, BCI re-training causes user frustration and exhaustion. This dissertation presents contributions to solve these challenges in BCI research. Specifically, we developed decoders trained on a single recording session and applied them on subsequently recorded sessions. This strategy evaluates BCI in a practical scenario with a potential to alleviate BCI user frustration without compromising performance. The initial part of the dissertation investigates extracting features that remain robust to changes in neural signal over several days of recordings. It presents a qualitative feature extraction technique based on ranking the instantaneous power of multichannel data. These qualitative features remain robust to outliers and changes in the baseline of neural recordings, while extracting discriminative information. These features form the foundation in developing robust decoders. Next, this dissertation presents a novel algorithm based on the hypothesis that multiple neural spatial patterns describe the variation in behavior. The presented algorithm outperforms the traditional methods in decoding over chronological recordings. Adapting such a decoder over multiple recording sessions (over 6 weeks) provided > 90% accuracy in decoding eight movement directions. In comparison, performance of traditional algorithms like Common Spatial Patterns deteriorates to 16% over the same time. Over time, adaptation reinforces some spatial patterns while diminishing others. Characterizing these spatial patterns reduces model complexity without user input, while retaining the same accuracy levels. Lastly, this dissertation provides an algorithm that overcomes the variation in recording quality. Chronic electrode implantation causes changes in signal-to-noise ratio (SNR) of neural signals. Thus, some signals and their corresponding features available during training become unavailable during testing and vice-versa. The proposed algorithm uses prior knowledge on spatial pattern evolution to estimate unknown neural features. This algorithm overcomes SNR variations and provides up to 93% decoding of eight movement directions over 6 weeks. Since model training requires only one session, this strategy reduces user frustration. In a practical closed-loop BCI, the user learns to produce stable spatial patterns, which improves performance of the proposed algorithms.Electrical and Computer Engineerin

Brain-Machine Interfaces: A Tale of Two Learners

Brain-machine interface (BMI) technology has rapidly matured over the last two decades, mainly thanks to the introduction of artificial intelligence (AI) methods, in particular, machine-learning algorithms. Yet, the need for subjects to learn to modulate their brain activity is a key component of successful BMI control. Blending machine and subject learning, or mutual learning, is widely acknowledged in the BMI field. Nevertheless, we posit that current research trends are heavily biased toward the machine-learning side of BMI training. In this article, we take a critical view of the relevant literature, and our own previous work, to identify the key issues for more effective mutual-learning schemes in translational BMIs that are specifically tailored to promote subject learning. We identify the main caveats in the literature on subject learning in BMI, in particular, the lack of longitudinal studies involving end users and shortcomings in quantifying subject learning, and pinpoint critical improvements for future experimental designs

The Cybathlon BCI race: Successful longitudinal mutual learning with two tetraplegic users

This work aims at corroborating the importance and efficacy of mutual learning in motor imagery (MI) brain–computer interface (BCI) by leveraging the insights obtained through our participation in the BCI race of the Cybathlon event. We hypothesized that, contrary to the popular trend of focusing mostly on the machine learning aspects of MI BCI training, a comprehensive mutual learning methodology that reinstates the three learning pillars (at the machine, subject, and application level) as equally significant could lead to a BCI–user symbiotic system able to succeed in real-world scenarios such as the Cybathlon event. Two severely impaired participants with chronic spinal cord injury (SCI), were trained following our mutual learning approach to control their avatar in a virtual BCI race game. The competition outcomes substantiate the effectiveness of this type of training. Most importantly, the present study is one among very few to provide multifaceted evidence on the efficacy of subject learning during BCI training. Learning correlates could be derived at all levels of the interface—application, BCI output, and electroencephalography (EEG) neuroimaging—with two end-users, sufficiently longitudinal evaluation, and, importantly, under real-world and even adverse conditions

Multi-Scale Architectures for Human Pose Estimation

In this dissertation we present multiple state-of-the-art deep learning methods for computer vision tasks using multi-scale approaches for two main tasks: pose estimation and semantic segmentation. For pose estimation, we introduce a complete framework expanding the fields-of-view of the network through a multi-scale approach, resulting in a significant increasing the effectiveness of conventional backbone architectures, for several pose estimation tasks without requiring a larger network or postprocessing. Our multi-scale pose estimation framework contributes to research on methods for single-person pose estimation in both 2D and 3D scenarios, pose estimation in videos, and the estimation of multiple people’s pose in a single image for both top-down and bottom-up approaches. In addition to the enhanced capability of multi-person pose estimation generated by our multi-scale approach, our framework also demonstrates a superior capacity to expanded the more detailed and heavier task of full-body pose estimation, including up to 133 joints per person. For segmentation, we present a new efficient architecture for semantic segmentation, based on a “Waterfall” Atrous Spatial Pooling architecture, that achieves a considerable accuracy increase while decreasing the number of network parameters and memory footprint. The proposed Waterfall architecture leverages the efficiency of progressive filtering in the cascade architecture while maintaining multi-scale fields-of-view comparable to spatial pyramid configurations. Additionally, our method does not rely on a postprocessing stage with conditional random fields, which further reduces complexity and required training time

Combining Shape and Learning for Medical Image Analysis

Automatic methods with the ability to make accurate, fast and robust assessments of medical images are highly requested in medical research and clinical care. Excellent automatic algorithms are characterized by speed, allowing for scalability, and an accuracy comparable to an expert radiologist. They should produce morphologically and physiologically plausible results while generalizing well to unseen and rare anatomies. Still, there are few, if any, applications where today\u27s automatic methods succeed to meet these requirements.\ua0The focus of this thesis is two tasks essential for enabling automatic medical image assessment, medical image segmentation and medical image registration. Medical image registration, i.e. aligning two separate medical images, is used as an important sub-routine in many image analysis tools as well as in image fusion, disease progress tracking and population statistics. Medical image segmentation, i.e. delineating anatomically or physiologically meaningful boundaries, is used for both diagnostic and visualization purposes in a wide range of applications, e.g. in computer-aided diagnosis and surgery.The thesis comprises five papers addressing medical image registration and/or segmentation for a diverse set of applications and modalities, i.e. pericardium segmentation in cardiac CTA, brain region parcellation in MRI, multi-organ segmentation in CT, heart ventricle segmentation in cardiac ultrasound and tau PET registration. The five papers propose competitive registration and segmentation methods enabled by machine learning techniques, e.g. random decision forests and convolutional neural networks, as well as by shape modelling, e.g. multi-atlas segmentation and conditional random fields

Breaking Down the Barriers To Operator Workload Estimation: Advancing Algorithmic Handling of Temporal Non-Stationarity and Cross-Participant Differences for EEG Analysis Using Deep Learning

This research focuses on two barriers to using EEG data for workload assessment: day-to-day variability, and cross- participant applicability. Several signal processing techniques and deep learning approaches are evaluated in multi-task environments. These methods account for temporal, spatial, and frequential data dependencies. Variance of frequency- domain power distributions for cross-day workload classification is statistically significant. Skewness and kurtosis are not significant in an environment absent workload transitions, but are salient with transitions present. LSTMs improve day- to-day feature stationarity, decreasing error by 59% compared to previous best results. A multi-path convolutional recurrent model using bi-directional, residual recurrent layers significantly increases predictive accuracy and decreases cross-participant variance. Deep learning regression approaches are applied to a multi-task environment with workload transitions. Accounting for temporal dependence significantly reduces error and increases correlation compared to baselines. Visualization techniques for LSTM feature saliency are developed to understand EEG analysis model biases

Data-Driven Approaches For Decoding Volitional Movement Intent From Bioelectrical Signals

There are nearly two million limb amputees living in the United States of America. Loss of limbs results in profound changes in one's life. However, the underlying neural circuitry and much of the ability to sense and control movements of their missing limb is retained even after limb loss. This means that amputee has the ability to control artificial limbs in a manner similar to how the limb was controlled before the loss. The goal of this research is to develop technologies for creating prosthetics arms that behave like the natural limb. Movement intent decoders allow amputees to control prostheses by interpreting motor-related bioelectrical signals, restoring their ability to perform day-to-day tasks. Such systems have to overcome a number of challenges before they can become practical. These challenges include the recursive nature of the human decision-making process, the limited amount of data typically available for training and the time-varying proper! ties of the nervous system. In this dissertation, we apply data-driven techniques to develop precise movement intent decoders and prosthetic controllers. Specifically, this work makes three major contributions to the field: 1- We developed movement intent decoders based on different neural network architectures including multilayer perceptron networks, convolutional neural networks, and long short-term memory neural networks. These systems were trained with a dataset aggregation (DAgger) approach, an imitation learning algorithm. DAgger augments the training set based on the decoder outputs in the training stage, mitigating possible mistakes that the decoders could make. The decoders were validated in offline analyses using data from two amputee arm subjects. The results demonstrated an improvement of up to 60% in the normalized mean-square decoding error over state-of-the-art decoders. 2- Movement intent decoders can be of different types, including proportional controllers, classification-based decoders or goal-based estimators. Each of these types of decoders come with their own set of advantages and weaknesses. We developed a shared-controller framework able to combine multiple decoders to control a prosthetic limb taking advantage of the individual strengths of the component decoders. The shared-controller framework was validated using two shared controller-systems. The first one combined a Kalman Filter (KF)-based decoder and a classifier-based decoder. The second system consisted of a KF-based decoder and a controller with knowledge of the final goal with a substantial amount of uncertainty. The controllers were validated using three amputees and three intact-arm subjects. The shared-controller systems outperformed the component decoders in most of the used metrics. An example of this is the subjects were able to stay in the intended position 70% longer using the KF-based decoder combined with a classifier-based decoder when compared with the KF-based decoder alone and 283% longer when compared with the classifier-based decoder alone. 3- Although the human body is a time-varying system, the decoders parameters are kept unchanged after training in many prosthesis systems. This causes a performance deterioration for the decoders over time. We developed an online-learning algorithm that is able to adapt itself during the post-training phase. The performance of such decoders were validated using data from two amputee subjects with transradial amputation. After 5 months of the initial training, the decoder with adaptation exhibited a 27% lower normalized mean-squared decoding error when compared with the same decoder without adaptation. In summary, the contributions of this research resulted in better training algorithms creating more accurate volitional movement intent decoders than previously possible, shared prosthesis controllers that combine multiple decoders in ways that perform better than the component decoders, and an online learning algorithm that enables the decoders to perform significantly better in the long term than current decoder realizations. Together, these contributions have brought us closer to the goal of creating limb prostheses that work and feel like the real limb