Real-Time Decision Fusion for Multimodal Neural Prosthetic Devices

The field of neural prosthetics aims to develop prosthetic limbs with a brain-computer interface (BCI) through which neural activity is decoded into movements. A natural extension of current research is the incorporation of neural activity from multiple modalities to more accurately estimate the user's intent. The challenge remains how to appropriately combine this information in real-time for a neural prosthetic device., i.e., fusing predictions from several single-modality decoders to produce a more accurate device state estimate. We examine two algorithms for continuous variable decision fusion: the Kalman filter and artificial neural networks (ANNs). Using simulated cortical neural spike signals, we implemented several successful individual neural decoding algorithms, and tested the capabilities of each fusion method in the context of decoding 2-dimensional endpoint trajectories of a neural prosthetic arm. Extensively testing these methods on random trajectories, we find that on average both the Kalman filter and ANNs successfully fuse the individual decoder estimates to produce more accurate predictions.Our results reveal that a fusion-based approach has the potential to improve prediction accuracy over individual decoders of varying quality, and we hope that this work will encourage multimodal neural prosthetics experiments in the future

Toward the Restoration of Hand Use to a Paralyzed Monkey: Brain-Controlled Functional Electrical Stimulation of Forearm Muscles

Loss of hand use is considered by many spinal cord injury survivors to be the most devastating consequence of their injury. Functional electrical stimulation (FES) of forearm and hand muscles has been used to provide basic, voluntary hand grasp to hundreds of human patients. Current approaches typically grade pre-programmed patterns of muscle activation using simple control signals, such as those derived from residual movement or muscle activity. However, the use of such fixed stimulation patterns limits hand function to the few tasks programmed into the controller. In contrast, we are developing a system that uses neural signals recorded from a multi-electrode array implanted in the motor cortex; this system has the potential to provide independent control of multiple muscles over a broad range of functional tasks. Two monkeys were able to use this cortically controlled FES system to control the contraction of four forearm muscles despite temporary limb paralysis. The amount of wrist force the monkeys were able to produce in a one-dimensional force tracking task was significantly increased. Furthermore, the monkeys were able to control the magnitude and time course of the force with sufficient accuracy to track visually displayed force targets at speeds reduced by only one-third to one-half of normal. Although these results were achieved by controlling only four muscles, there is no fundamental reason why the same methods could not be scaled up to control a larger number of muscles. We believe these results provide an important proof of concept that brain-controlled FES prostheses could ultimately be of great benefit to paralyzed patients with injuries in the mid-cervical spinal cord

A Wireless Brain-Machine Interface for Real-Time Speech Synthesis

Background: Brain-machine interfaces (BMIs) involving electrodes implanted into the human cerebral cortex have recently been developed in an attempt to restore function to profoundly paralyzed individuals. Current BMIs for restoring communication can provide important capabilities via a typing process, but unfortunately they are only capable of slow communication rates. In the current study we use a novel approach to speech restoration in which we decode continuous auditory parameters for a real-time speech synthesizer from neuronal activity in motor cortex during attempted speech. Methodology/Principal Findings: Neural signals recorded by a Neurotrophic Electrode implanted in a speech-related region of the left precentral gyrus of a human volunteer suffering from locked-in syndrome, characterized by near-total paralysis with spared cognition, were transmitted wirelessly across the scalp and used to drive a speech synthesizer. A Kalman filter-based decoder translated the neural signals generated during attempted speech into continuous parameters for controlling a synthesizer that provided immediate (within 50 ms) auditory feedback of the decoded sound. Accuracy of the volunteer's vowel productions with the synthesizer improved quickly with practice, with a 25% improvement in average hit rate (from 45% to 70%) and 46% decrease in average endpoint error from the first to the last block of a three-vowel task. Conclusions/Significance: Our results support the feasibility of neural prostheses that may have the potential to provide near-conversational synthetic speech output for individuals with severely impaired speech motor control. They also provide an initial glimpse into the functional properties of neurons in speech motor cortical areas.National Institute on Deafness and Other Communication Disorders (U.S.) (Grant R44-DC007050)National Institute on Deafness and Other Communication Disorders (U.S.) (Grant R01-DC007683)National Institute on Deafness and Other Communication Disorders (U.S.) (Grant R01-DC002852)Center of Excellence for Learning in Education, Science, and Technology (SBE-0354378

A simulation study on the effects of neuronal ensemble properties on decoding algorithms for intracortical brain-machine interfaces

Background: Intracortical brain-machine interfaces (BMIs) harness movement information by sensing neuronal activities using chronic microelectrode implants to restore lost functions to patients with paralysis. However, neuronal signals often vary over time, even within a day, forcing one to rebuild a BMI every time they operate it. The term "rebuild" means overall procedures for operating a BMI, such as decoder selection, decoder training, and decoder testing. It gives rise to a practical issue of what decoder should be built for a given neuronal ensemble. This study aims to address it by exploring how decoders' performance varies with the neuronal properties. To extensively explore a range of neuronal properties, we conduct a simulation study. Methods: Focusing on movement direction, we examine several basic neuronal properties, including the signal-to-noise ratio of neurons, the proportion of well-tuned neurons, the uniformity of their preferred directions (PDs), and the non-stationarity of PDs. We investigate the performance of three popular BMI decoders: Kalman filter, optimal linear estimator, and population vector algorithm. Results: Our simulation results showed that decoding performance of all the decoders was affected more by the proportion of well-tuned neurons that their uniformity. Conclusions: Our study suggests a simulated scenario of how to choose a decoder for intracortical BMIs in various neuronal conditions