Algorithms for Neural Prosthetic Applications

abstract: In the last 15 years, there has been a significant increase in the number of motor neural prostheses used for restoring limb function lost due to neurological disorders or accidents. The aim of this technology is to enable patients to control a motor prosthesis using their residual neural pathways (central or peripheral). Recent studies in non-human primates and humans have shown the possibility of controlling a prosthesis for accomplishing varied tasks such as self-feeding, typing, reaching, grasping, and performing fine dexterous movements. A neural decoding system comprises mainly of three components: (i) sensors to record neural signals, (ii) an algorithm to map neural recordings to upper limb kinematics and (iii) a prosthetic arm actuated by control signals generated by the algorithm. Machine learning algorithms that map input neural activity to the output kinematics (like finger trajectory) form the core of the neural decoding system. The choice of the algorithm is thus, mainly imposed by the neural signal of interest and the output parameter being decoded. The various parts of a neural decoding system are neural data, feature extraction, feature selection, and machine learning algorithm. There have been significant advances in the field of neural prosthetic applications. But there are challenges for translating a neural prosthesis from a laboratory setting to a clinical environment. To achieve a fully functional prosthetic device with maximum user compliance and acceptance, these factors need to be addressed and taken into consideration. Three challenges in developing robust neural decoding systems were addressed by exploring neural variability in the peripheral nervous system for dexterous finger movements, feature selection methods based on clinically relevant metrics and a novel method for decoding dexterous finger movements based on ensemble methods.Dissertation/ThesisDoctoral Dissertation Bioengineering 201

Developing robust movement decoders for local field potentials

textBrain Computer Interfaces (BCI) are devices that translate acquired neural signals to command and control signals. Applications of BCI include neural rehabilitation and neural prosthesis (thought controlled wheelchair, thought controlled speller etc.) to aid patients with disabilities and to augment human computer interaction. A successful practical BCI requires a faithful acquisition modality to record high quality neural signals; a signal processing system to construct appropriate features from these signals; and an algorithm to translate these features to appropriate outputs. Intracortical recordings like local field potentials provide reliable high SNR signals over long periods and suit BCI applications well. However, the non-stationarity of neural signals poses a challenge in robust decoding of subject behavior. Most BCI research focuses either on developing daily re-calibrated decoders that require exhaustive training sessions; or on providing cross-validation results. Such results ignore the variation of signal characteristics over different sessions and provide an optimistic estimate of BCI performance. Specifically, traditional BCI algorithms fail to perform at the same level on chronological data recordings. Neural signals are susceptible to variations in signal characteristics due to changes in subject behavior and learning, and variability in electrode characteristics due to tissue interactions. While training day-specific BCI overcomes signal variability, BCI re-training causes user frustration and exhaustion. This dissertation presents contributions to solve these challenges in BCI research. Specifically, we developed decoders trained on a single recording session and applied them on subsequently recorded sessions. This strategy evaluates BCI in a practical scenario with a potential to alleviate BCI user frustration without compromising performance. The initial part of the dissertation investigates extracting features that remain robust to changes in neural signal over several days of recordings. It presents a qualitative feature extraction technique based on ranking the instantaneous power of multichannel data. These qualitative features remain robust to outliers and changes in the baseline of neural recordings, while extracting discriminative information. These features form the foundation in developing robust decoders. Next, this dissertation presents a novel algorithm based on the hypothesis that multiple neural spatial patterns describe the variation in behavior. The presented algorithm outperforms the traditional methods in decoding over chronological recordings. Adapting such a decoder over multiple recording sessions (over 6 weeks) provided > 90% accuracy in decoding eight movement directions. In comparison, performance of traditional algorithms like Common Spatial Patterns deteriorates to 16% over the same time. Over time, adaptation reinforces some spatial patterns while diminishing others. Characterizing these spatial patterns reduces model complexity without user input, while retaining the same accuracy levels. Lastly, this dissertation provides an algorithm that overcomes the variation in recording quality. Chronic electrode implantation causes changes in signal-to-noise ratio (SNR) of neural signals. Thus, some signals and their corresponding features available during training become unavailable during testing and vice-versa. The proposed algorithm uses prior knowledge on spatial pattern evolution to estimate unknown neural features. This algorithm overcomes SNR variations and provides up to 93% decoding of eight movement directions over 6 weeks. Since model training requires only one session, this strategy reduces user frustration. In a practical closed-loop BCI, the user learns to produce stable spatial patterns, which improves performance of the proposed algorithms.Electrical and Computer Engineerin

Object Counting with Deep Learning

This thesis explores various empirical aspects of deep learning or convolutional network based models for efficient object counting. First, we train moderately large convolutional networks on comparatively smaller datasets containing few hundred samples from scratch with conventional image processing based data augmentation. Then, we extend this approach for unconstrained, outdoor images using more advanced architectural concepts. Additionally, we propose an efficient, randomized data augmentation strategy based on sub-regional pixel distribution for low-resolution images. Next, the effectiveness of depth-to-space shuffling of feature elements for efficient segmentation is investigated for simpler problems like binary segmentation -- often required in the counting framework. This depth-to-space operation violates the basic assumption of encoder-decoder type of segmentation architectures. Consequently, it helps to train the encoder model as a sparsely connected graph. Nonetheless, we have found comparable accuracy to that of the standard encoder-decoder architectures with our depth-to-space models. After that, the subtleties regarding the lack of localization information in the conventional scalar count loss for one-look models are illustrated. At this point, without using additional annotations, a possible solution is proposed based on the regulation of a network-generated heatmap in the form of a weak, subsidiary loss. The models trained with this auxiliary loss alongside the conventional loss perform much better compared to their baseline counterparts, both qualitatively and quantitatively. Lastly, the intricacies of tiled prediction for high-resolution images are studied in detail, and a simple and effective trick of eliminating the normalization factor in an existing computational block is demonstrated. All of the approaches employed here are thoroughly benchmarked across multiple heterogeneous datasets for object counting against previous, state-of-the-art approaches

Modeling and Analysis of Neural Spike Trains

open

Learning to process with spikes and to localise pulses

In the last few decades, deep learning with artificial neural networks (ANNs) has emerged as one of the most widely used techniques in tasks such as classification and regression, achieving competitive results and in some cases even surpassing human-level performance. Nonetheless, as ANN architectures are optimised towards empirical results and departed from their biological precursors, how exactly human brains process information using these short electrical pulses called spikes remains a mystery. Hence, in this thesis, we explore the problem of learning to process with spikes and to localise pulses. We first consider spiking neural networks (SNNs), a type of ANN that more closely mimic biological neural networks in that neurons communicate with one another using spikes. This unique architecture allows us to look into the role of heterogeneity in learning. Since it is conjectured that the information is encoded by the timing of spikes, we are particularly interested in the heterogeneity of time constants of neurons. We then trained SNNs for classification tasks on a range of visual and auditory neuromorphic datasets, which contain streams of events (spike times) instead of the conventional frame-based data, and show that the overall performance is improved by allowing the neurons to have different time constants, especially on tasks with richer temporal structure. We also find that the learned time constants are distributed similarly to those experimentally observed in some mammalian cells. Besides, we demonstrate that learning with heterogeneity improves robustness against hyperparameter mistuning. These results suggest that heterogeneity may be more than the byproduct of noisy processes and perhaps serves a key role in learning in changing environments, yet heterogeneity has been overlooked in basic artificial models. While neuromorphic datasets, which are often captured by neuromorphic devices that closely model the corresponding biological systems, have enabled us to explore the more biologically plausible SNNs, there still exists a gap in understanding how spike times encode information in actual biological neural networks like human brains, as such data is difficult to acquire due to the trade-off between the timing precision and the number of cells simultaneously recorded electrically. Instead, what we usually obtain is the low-rate discrete samples of trains of filtered spikes. Hence, in the second part of the thesis, we focus on a different type of problem involving pulses, that is to retrieve the precise pulse locations from these low-rate samples. We make use of the finite rate of innovation (FRI) sampling theory, which states that perfect reconstruction is possible for classes of continuous non-bandlimited signals that have a small number of free parameters. However, existing FRI methods break down under very noisy conditions due to the so-called subspace swap event. Thus, we present two novel model-based learning architectures: Deep Unfolded Projected Wirtinger Gradient Descent (Deep Unfolded PWGD) and FRI Encoder-Decoder Network (FRIED-Net). The former is based on the existing iterative denoising algorithm for subspace-based methods, while the latter models directly the relationship between the samples and the locations of the pulses using an autoencoder-like network. Using a stream of K Diracs as an example, we show that both algorithms are able to overcome the breakdown inherent in the existing subspace-based methods. Moreover, we extend our FRIED-Net framework beyond conventional FRI methods by considering when the shape is unknown. We show that the pulse shape can be learned using backpropagation. This coincides with the application of spike detection from real-world calcium imaging data, where we achieve competitive results. Finally, we explore beyond canonical FRI signals and demonstrate that FRIED-Net is able to reconstruct streams of pulses with different shapes.Open Acces

Tehisnärvivõrgud bioloogiliste andmete analüüsimiseks

Väitekirja elektrooniline versioon ei sisalda publikatsiooneTehisnärvivõrgud viimastel aastatel populaarsust kogunud masinõppe algoritm, mis on võimeline näidete põhjal õppima. Erinevad tehisnärvivõrkude alamtüübid on kasutusel mitmetes arvutiteaduse harudes: konvolutsioonilisi võrke rakendatakse objekti- ja näotuvastuses; rekurrentsed võrgud on efektiivsed kõnetuvastuses ja keeletehnoloogias. Need ei ole aga ainsad võimalikud tehisnärvivõrkude rakendamise valdkonnad - selles doktoritöös näitasime me tehisnärvivõrkude kasulikkust kahe bioloogilise probleemi lahendamisel. Esiteks küsisime, kas ainult DNA jupis sisalduva info põhjal on võimalik ennustada, kas see järjestus pärineb viiruse (ja mitte mõnda muud tüüpi organismi) genoomist. Läbi kahe publikatsiooni tõestasime me, et masinõppe algoritmid on selleks tõesti võimelised. Parima täpsuse saavutas konvolutsiooniline närvivõrk. Loodud lahendus võimaldab viroloogidel tuvastada seni tundmatuid viiruseliike, millel võib olla oluline mõju inimese tervisele. Teine käsitletud bioloogiline andmestik pärineb neuroteadusest. Imetajate hipokampuses esineb nn. koharakke, mis aktiveeruvad vaid juhul, kui loom asub teatud ruumipunktis. Näitasime, et rekurrentsete närvivõrkude abil saab vaid mõnekümne koharaku aktiivsuse põhjal ennustada roti asukohta ligi 10 cm täpsusega. Rekurrentsed võrgud osutusid efektiivsemaks kui neuroteaduses enim levinud Bayesi meetodid. Need võrgud suudavad kasutada rakkude eelnevat aktiivsust kontekstina, mis aitab täpsustada asukoha ennustust. Ka teistes neuroandmestikes võib eelnev ajuaktiivsus peegeldada konteksti, mis sisaldab olulist infot hetkel toimuva kohta. Seega võivad rekurrentsed tehisnärvivõrgud osutuda ajusignaalide mõistmisel ülimalt kasulikuks. Samuti on bioinformaatikas veel hulk andmestikke, kus konvolutsioonilised võrgud võivad osutuda efektiivsemaks kui senised meetodid. Loodame, et käesolev töö julgustab teadlasi tehisnärvivõrke proovima ka oma andmestikel.Artificial neural networks (ANNs) are a machine learning algorithm that has gained popularity in recent years. Different subtypes of ANNs are used in various fields of computer science. For example, convolutional networks are useful in object and face recognition systems; whereas recurrent neural networks are effective in speech recognition and natural language processing. However, these examples are not the only possible applications of neural nets - in this thesis we demonstrated the benefits of ANNs in analyzing two biological datasets. First, we investigated if based only on the information contained within a DNA snippet it is possible to predict if the snippet originates from a viral genome or not. Through two publications we demonstrated that machine learning algorithms can make this prediction. Convolutional neural networks (CNNs) proved to be the most accurate. The recommendation system created allows virologists to identify yet unknown viral species, which may have important effects on human health. The second biological dataset analyzed originates from neuroscience. In mammalian hippocampus there are so called place cells which activate only if the animal is in a specific location in space. We showed that recurrent neural networks (RNNs) allow to predict the animal’s location with ~10cm precision based on the activity of only a few dozen place cells. RNNs proved to be more effective than the most commonly used Bayesian methods. These networks use the past neuronal activity as a context that helps fine-tune the location predictions. Also in many other neural datasets the prior brain activity might reflect important information about the current behaviour. Hence, RNNs might turn out to be very useful in making sense of brain signals. Similarly, CNNs are likely to prove more efficient than the currently used methods on many other bioinformatics datasets. We hope this thesis encourages more scientists to try neural networks on their own datasets.https://www.ester.ee/record=b536839

PROGRAMMABLE NEURAL PROCESSING FRAMEWORK FOR IMPLANTABLE WIRELESS BRAIN-COMPUTER INTERFACES

Brain-computer interfaces (BCIs) are able to translate cerebral cortex neural activity into control signals for computer cursors or prosthetic limbs. Such neural prosthetics offer tremendous potential for improving the quality of life for disabled individuals. Despite the success of laboratory-based neural prosthetic systems, there is a long way to go before it makes a clinically viable device. The major obstacles include lack of portability due to large physical footprint and performance-power inefficiency of current BCI platforms. Thus, there are growing interests in integrating more BCI's components into a tiny implantable unit, which can minimize the surgical risk and maximize the usability. To date, real-time neural prosthetic systems in laboratory require a wired connection penetrating the skull to a bulky external power/processing unit. For the wireless implantable BCI devices, only the data acquisition and spike detection stages are fully integrated. The rest digital post-processing can only be performed on one chosen channel via custom ASICs, whose lack of flexibility and long development cycle are likely to slow down the ongoing clinical research.This thesis proposes and tests the feasibility of performing on-chip, real-time spike sorting/neural decoding on a programmable wireless sensor network (WSN) node, which is chosen as a compact, low-power platform representative of a future implantable chip. The final accuracy is comparable to state-of-the-art open-loop neural decoder. A detailed power/performance trade-off analysis is presented. Our experimental results show that: 1)direct on-chip neural decoding without spike sorting can achieve 30Hz updating rate, with power density lower than 62mW/cm2; 2)the execution time and power density meet the requirements to perform real-time spike sorting and wireless transmission on a single neural channel. For the option of having spike sorting in order to keep all neural information, we propose a new neural processing workflow that incorporates a light-weight neuron selection method to the training process to reduce the number of channels required for processing. Experimental results show that the proposed method not only narrows the gap between the system requirement and current hardware technology, but also increase the accuracy of the neural decoder by 3%-22%, due to elimination of noisy channels

Error-related potentials for adaptive decoding and volitional control

Locked-in syndrome (LIS) is a condition characterized by total or near-total paralysis with preserved cognitive and somatosensory function. For the locked-in, brain-machine interfaces (BMI) provide a level of restored communication and interaction with the world, though this technology has not reached its fullest potential. Several streams of research explore improving BMI performance but very little attention has been given to the paradigms implemented and the resulting constraints imposed on the users. Learning new mental tasks, constant use of external stimuli, and high attentional and cognitive processing loads are common demands imposed by BMI. These paradigm constraints negatively affect BMI performance by locked-in patients. In an effort to develop simpler and more reliable BMI for those suffering from LIS, this dissertation explores using error-related potentials, the neural correlates of error awareness, as an access pathway for adaptive decoding and direct volitional control. In the first part of this thesis we characterize error-related local field potentials (eLFP) and implement a real-time decoder error detection (DED) system using eLFP while non-human primates controlled a saccade BMI. Our results show specific traits in the eLFP that bridge current knowledge of non-BMI evoked error-related potentials with error-potentials evoked during BMI control. Moreover, we successfully perform real-time DED via, to our knowledge, the first real-time LFP-based DED system integrated into an invasive BMI, demonstrating that error-based adaptive decoding can become a standard feature in BMI design. In the second part of this thesis, we focus on employing electroencephalography error-related potentials (ErrP) for direct volitional control. These signals were employed as an indicator of the user’s intentions under a closed-loop binary-choice robot reaching task. Although this approach is technically challenging, our results demonstrate that ErrP can be used for direct control via binary selection and, given the appropriate levels of task engagement and agency, single-trial closed-loop ErrP decoding is possible. Taken together, this work contributes to a deeper understanding of error-related potentials evoked during BMI control and opens new avenues of research for employing ErrP as a direct control signal for BMI. For the locked-in community, these advancements could foster the development of real-time intuitive brain-machine control