Decoding hand movement velocity from electroencephalogram signals during a drawing task

<p>Abstract</p> <p>Background</p> <p>Decoding neural activities associated with limb movements is the key of motor prosthesis control. So far, most of these studies have been based on invasive approaches. Nevertheless, a few researchers have decoded kinematic parameters of single hand in non-invasive ways such as magnetoencephalogram (MEG) and electroencephalogram (EEG). Regarding these EEG studies, center-out reaching tasks have been employed. Yet whether hand velocity can be decoded using EEG recorded during a self-routed drawing task is unclear.</p> <p>Methods</p> <p>Here we collected whole-scalp EEG data of five subjects during a sequential 4-directional drawing task, and employed spatial filtering algorithms to extract the amplitude and power features of EEG in multiple frequency bands. From these features, we reconstructed hand movement velocity by Kalman filtering and a smoothing algorithm.</p> <p>Results</p> <p>The average Pearson correlation coefficients between the measured and the decoded velocities are 0.37 for the horizontal dimension and 0.24 for the vertical dimension. The channels on motor, posterior parietal and occipital areas are most involved for the decoding of hand velocity. By comparing the decoding performance of the features from different frequency bands, we found that not only slow potentials in 0.1-4 Hz band but also oscillatory rhythms in 24-28 Hz band may carry the information of hand velocity.</p> <p>Conclusions</p> <p>These results provide another support to neural control of motor prosthesis based on EEG signals and proper decoding methods.</p

Noninvasive neural decoding of overt and covert hand movement

It is generally assumed that the signal-to-noise ratio and information content of neural data acquired noninvasively via magnetoencephalography (MEG) or scalp electroencephalography (EEG) are insufficient to extract detailed information about natural, multi-joint movements of the upper limb. If valid, this assumption could severely limit the practical usage of noninvasive signals in brain-computer interface (BCI) systems aimed at continuous complex control of arm-like prostheses for movement impaired persons. Fortunately this dissertation research casts doubt on the veracity of this assumption by extracting continuous hand kinematics from MEG signals collected during a 2D center-out drawing task (Bradberry et al. 2009, NeuroImage, 47:1691-700) and from EEG signals collected during a 3D center-out reaching task (Bradberry et al. 2010, Journal of Neuroscience, 30:3432-7). In both studies, multiple regression was performed to find a matrix that mapped past and current neural data from multiple sensors to current hand kinematic data (velocity). A novel method was subsequently devised that incorporated the weights of the mapping matrix and the standardized low resolution electromagnetic tomography (sLORETA) software to reveal that the brain sources that encoded hand kinematics in the MEG and EEG studies were corroborated by more traditional studies that required averaging across trials and/or subjects. Encouraged by the favorable results of these off-line decoding studies, a BCI system was developed for on-line decoding of covert movement intentions that provided users with real-time visual feedback of the decoder output. Users were asked to use only their thoughts to move a cursor to acquire one of four targets on a computer screen. With only one training session, subjects were able to accomplish this task. The promising results of this dissertation research significantly advance the state-of-the-art in noninvasive BCI systems

Brain Computer Interface for Gesture Control of a Social Robot: an Offline Study

Brain computer interface (BCI) provides promising applications in neuroprosthesis and neurorehabilitation by controlling computers and robotic devices based on the patient's intentions. Here, we have developed a novel BCI platform that controls a personalized social robot using noninvasively acquired brain signals. Scalp electroencephalogram (EEG) signals are collected from a user in real-time during tasks of imaginary movements. The imagined body kinematics are decoded using a regression model to calculate the user-intended velocity. Then, the decoded kinematic information is mapped to control the gestures of a social robot. The platform here may be utilized as a human-robot-interaction framework by combining with neurofeedback mechanisms to enhance the cognitive capability of persons with dementia.Comment: Presented in: 25th Iranian Conference on Electrical Engineering (ICEE

On the Usage of Linear Regression Models to Reconstruct Limb Kinematics from Low Frequency EEG Signals

Several works have reported on the reconstruction of 2D/3D limb kinematics from low-frequency EEG signals using linear regression models based on positive correlation values between the recorded and the reconstructed trajectories. This paper describes the mathematical properties of the linear model and the correlation evaluation metric that may lead to a misinterpretation of the results of this type of decoders. Firstly, the use of a linear regression model to adjust the two temporal signals (EEG and velocity profiles) implies that the relevant component of the signal used for decoding (EEG) has to be in the same frequency range as the signal to be decoded (velocity profiles). Secondly, the use of a correlation to evaluate the fitting of two trajectories could lead to overly-optimistic results as this metric is invariant to scale. Also, the correlation has a non-linear nature that leads to higher values for sinus/cosinus-like signals at low frequencies. Analysis of these properties on the reconstruction results was carried out through an experiment performed in line with previous studies, where healthy participants executed predefined reaching movements of the hand in 3D space. While the correlations of limb velocity profiles reconstructed from low-frequency EEG were comparable to studies in this domain, a systematic statistical analysis revealed that these results were not above the chance level. The empirical chance level was estimated using random assignments of recorded velocity profiles and EEG signals, as well as combinations of randomly generated synthetic EEG with recorded velocity profiles and recorded EEG with randomly generated synthetic velocity profiles. The analysis shows that the positive correlation results in this experiment cannot be used as an indicator of successful trajectory reconstruction based on a neural correlate. Several directions are herein discussed to address the misinterpretation of results as well as the implications on previous invasive and non-invasive works

EEG source analysis during circular rhythmic human arm movements

Tese de mestrado integrado, Engenharia Biomédica e Biofísica (Engenharia Clínica e Instrumentação Médica) Universidade de Lisboa, Faculdade de Ciências, 2017Decoding arm movement trajectory from brain signals would allow motor impaired people to control an arm prosthetic. Studies show that we can estimate a vector that points in the direction of arm movements based on single motor neuron activity - the population vector. This type of recording requires the surgical insertion of electrodes in the cerebral cortex. Although such invasive recordings would offer high spatial resolution, noninvasive recording have the advantage of high temporal resolution and no need for surgery. Researchers have managed to decode movement properties from noninvasive brain signals with similar accuracy as from invasive recordings. But can we find a noninvasive analogous of the population vector, a vector that points in the direction of the arm movement? This was the motivation for this thesis. To approach this question we acquired EEG, EOG and kinematic data from 12 healthy subjects while they performed a rhythmic circular right arm movement. We analyzed the data in the time and frequency domains. In the time domain we explored mainly the data averaged over cycles. We found a pattern that looked as if the potentials in the scalp rotated with the arm. To better visualize this rotation, we fit one dipole per time-stamp in the averaged cycle data of each subject to describe the scalp’s potentials. The dipoles rotated along the cycle for all subjects, most of them in the same direction and plane of rotation, with exception for two subjects whose rotation was opposite and three subjects with a slightly different rotation plan. In the frequency domain, we used the Source Power Comodulation algorithm (SPoC), an algorithm that searches for components whose power correlates with a target variable, in our case, the arm kinematics. By applying this algorithm to 20-24 Hz band-pass filtered data, we found two components per subject, each calculated with different kinematic target variables. The results show components that when applied to the non band-pass filtered data, created signals whose power spectrum highly correlated with the given targets (the average of the absolute correlations being 85.5%). The physiological reason for both these phenomena is not entirely understood. To find the analogous of the population vector there is still a long way to go, and we hope this thesis was a first step towards it.O cérebro controla direta ou indiretamente todas as ações do corpo humano, entre elas o nosso movimento. O movimento é uma capacidade fundamental ao ser humano e, por essa mesma razão, indivíduos que sofram de incapacidades motoras têm uma redução considerável da sua qualidade de vida. Uma interface cérebro-computador (mais conhecida pelo seu nome em inglês brain-computer interface (BCI)) é um sistema que permite o controlo de dispositivos externos usando sinais cerebrais. Esta tecnologia é particularmente interessante para pessoas com incapacidade motora uma vez que não necessita de input físico e poderia ser usada para controlar uma neuroprótese ou um braço robótico. Existem várias estratégias que possibilitam o controlo destes sistemas, mas para o controlo de uma prótese do braço seria preferível usar uma estratégia natural, que não implicasse uma aprendizagem exaustiva por parte do utilizador. Para esse fim, é necessário descodificar vários parâmetros motores de acordo com a intenção do utilizador, como por exemplo, a direção do braço. A possibilidade de um dia conseguir descodificar sinais cerebrais para o controlo de dispositivos externos já começa a ganhar forma, mas ainda não é possível a um nível suficientemente eficaz. Usando métodos invasivos de aquisição de sinais cerebrais que requerem cirurgia para implantar elétrodos no córtex cerebral, Georgopoulos et al. conseguiram distinguir entre movimentos direcionais (em 8 direções num plano horizontal) em macacos. Nessas experiências criou o conceito de vetor de população (population vector) que é um vetor calculado a partir da atividade de neurónios motores que tem a particularidade de apontar na direção do movimento executado. Já no campo dos métodos de aquisição não-invasivos podemos destacar o eletroencefalograma (EEG) e o magnetoencefalograma (MEG) que adquirem sinais elétricos e magnéticos (respetivamente) com sensores colocados fora do crânio. Vários investigadores usaram estes métodos de aquisição para descodificar sinais cerebrais durante tarefas de movimento direcionais usando regressões lineares em sinais de baixa frequência, e modulações em frequência para sinais na gama dos 50-90 Hz (banda de frequência ϒ) e em frequência mais baixas para os 10-30 Hz (bandas de frequência α e β). Algo que ainda não foi estudado é a possibilidade de encontrar um análogo ao vetor população usando métodos não-invasivos. Este não teria os mesmos princípios do vetor de Georgopoulos, uma vez que nos é impossível inferir a atividade de neurónios singulares em métodos não-invasivos, mas teria o mesmo objetivo: apontar na direção do movimento executado. Para explorar este conceito realizámos aquisição de dados EEG, eletrooculograma (EOG) e dados cinéticos do braço direito de 12 sujeitos saudáveis, enquanto estes executavam um movimento rítmico, circular, no sentido dos ponteiros do relógio num plano vertical à sua frente. Durante a aquisição, os sujeitos focaram o seu olhar numa cruz mostrada através de um monitor colocado a sua frente, de forma a minimizar os movimentos oculares. Adicionalmente, uma divisória foi colocada perto do lado direito da face de cada sujeito impedindo os mesmos de observarem o seu braço enquanto realizavam o movimento requisitado, não obtendo assim qualquer feedback visual do seu membro superior. Os dados cinéticos foram adquiridos com um sensor Kinect para a Xbox 360 que ao longo da experiência localizou as junções do braço direito dos sujeitos. Os dados cinéticos foram filtrados com um passa-banda 0.3-0.8 Hz e, ao longo dos ciclos do braço, os pontos extremos do braço (i.e., os máximos e mínimos nas coordenadas vertical e horizontal) foram anotados nos dados para possibilitar a associação dos sinais cerebrais com a trajetória do braço em cada ciclo. Para cada sujeito os canais EEG ruidosos foram interpolados, os dados foram referenciados à média comum de todos os canais, e os sinais foram filtrados numa banda de frequência 0.25-100 Hz e com um filtro tapa banda nos 50 e nos 100 Hz, este último para rejeitar o ruído de fundo. Os sinais de EEG e EOG foram separados em épocas conforme a posição do braço, sendo que cada época passou então a consistir num ciclo do braço completo que começa no ponto mais alto da coordenada vertical. Cada época foi depois temporalmente distorcida para que todas tivessem a mesma duração. As épocas com artefactos foram rejeitadas da análise usando métodos automáticos de rejeição. Independent Component Analysis (ICA) foi utilizada para identificar e posteriormente rejeitar componentes independentes referentes a movimentos musculares e oculares. Por fim, os dados foram explorados em ambos os domínios de tempo e frequência. No domínio do tempo, estudámos mais especificamente a média das épocas de EEG e EOG durante os ciclos do braço. Uma vez que sinais não-invasivos são muito sujeitos a ruído, a média elimina artefactos singulares e acentua os sinais que aparecem constantemente nos dados. Os sinais do ciclo médio mostraram um padrão interessante para todos os sujeitos; um comportamento rotacional ao longo da rotação do braço direito. Para acompanhar a rotação dos potenciais, procurámos por um dipolo que descrevesse a distribuição topográfica a cada ponto do tempo. A rotação dos potenciais do EEG ao longo do ciclo médio foram verificados com a rotação da direção do dipolo ao longo do ciclo. A grande maioria dos sujeitos obteve um dipolo a rodar no mesmo sentido no mesmo plano (segundo a regra da mão direita, com um vetor de rotação a apontar para a zona frontal esquerda do cérebro). Cinco sujeitos foram a exceção, 2 desses cujo dipolo rodava no sentido contrário, e os restantes 3 sujeitos cujo dipolo rodava no mesmo sentido, mas num plano ligeiramente diferente. Em todos os sujeitos o dipolo ajustado rodava, de forma relativamente uniforme. No domínio da frequência, estudámos em particular a banda de frequência dos 20 aos 24 Hz. Escolheuse esta banda de frequência pois demonstrou os resultados mais interessantes e já tinha sido utilizada em estudos prévios. Usámos um algoritmo chamado SPoC (Source Power Comodulation) que encontra componentes de atividade cerebral cuja amplitude em frequência correlacione com uma variável alvo. Como variável alvo usámos os dados cinéticos do braço direito, e como input os dados cerebrais filtrados por um filtro passa-banda (20-24 Hz). Os resultados traduziram-se numa série de componentes cuja amplitude correlacionava ou anti-correlacionava com o movimento do braço, muitas delas com projecções topográficas consistentes com as áreas cerebrais motoras. Encontraram-se algumas semelhanças entre os padrões de ativação das componentes do SPoC dos vários sujeitos, ainda que os resultados variassem entre cada um. Ao projetar as componentes aos dados não-filtrados pelo passa-banda, verificamos que as modelações em frequência de facto correlacionam com as variáveis-alvo como esperado, com uma média da norma das correlações de todos os sujeitos a 85,5%. No domínio temporal, ainda que recorrendo à média de todos os ciclos (épocas), este é o primeiro estudo que demonstra de forma não-invasiva, a existência de um dipolo com comportamento rotacional ao longo da rotação do braço. Para o seu uso em tecnologias de BCI, é necessário encontrar o mesmo fenómeno em épocas únicas, tornando possível uma classificação em single-trial e em tempo real. No que toca aos resultados no domínio da frequência, a procura por componentes cuja fonte poderia estar envolvida na criação do movimento circular foi também bem-sucedida. Este estudo abriu portas para uma série de investigações futuras. Para trabalhos posteriores destaco a necessidade de uma análise estatística, de usar mais do que um dipolo para descrever a distribuição de potenciais no domínio temporal, de explorar os dados em cada movimento e não apenas a sua média, e de explorar paradigmas semelhantes durante o movimento do braço esquerdo. Os resultados desta tese serviram, portanto, como primeiro passo na direção de encontrar o análogo não-invasivo do vetor de população

A Wireless Brain-Machine Interface for Real-Time Speech Synthesis

Background: Brain-machine interfaces (BMIs) involving electrodes implanted into the human cerebral cortex have recently been developed in an attempt to restore function to profoundly paralyzed individuals. Current BMIs for restoring communication can provide important capabilities via a typing process, but unfortunately they are only capable of slow communication rates. In the current study we use a novel approach to speech restoration in which we decode continuous auditory parameters for a real-time speech synthesizer from neuronal activity in motor cortex during attempted speech. Methodology/Principal Findings: Neural signals recorded by a Neurotrophic Electrode implanted in a speech-related region of the left precentral gyrus of a human volunteer suffering from locked-in syndrome, characterized by near-total paralysis with spared cognition, were transmitted wirelessly across the scalp and used to drive a speech synthesizer. A Kalman filter-based decoder translated the neural signals generated during attempted speech into continuous parameters for controlling a synthesizer that provided immediate (within 50 ms) auditory feedback of the decoded sound. Accuracy of the volunteer's vowel productions with the synthesizer improved quickly with practice, with a 25% improvement in average hit rate (from 45% to 70%) and 46% decrease in average endpoint error from the first to the last block of a three-vowel task. Conclusions/Significance: Our results support the feasibility of neural prostheses that may have the potential to provide near-conversational synthetic speech output for individuals with severely impaired speech motor control. They also provide an initial glimpse into the functional properties of neurons in speech motor cortical areas.National Institute on Deafness and Other Communication Disorders (U.S.) (Grant R44-DC007050)National Institute on Deafness and Other Communication Disorders (U.S.) (Grant R01-DC007683)National Institute on Deafness and Other Communication Disorders (U.S.) (Grant R01-DC002852)Center of Excellence for Learning in Education, Science, and Technology (SBE-0354378

Intrinsic and Extrinsic Biomechanical Factors in a Co-adaptive ECoG-based Brain Computer Interface

Paralysis, due to spinal cord injury, amyotrophic lateral sclerosis (ALS), or stroke, is the result of severed communication between the brain and the motor periphery. Brain computer interfaces (BCIs) are neuroprosthetic devices that create novel communication pathways by measuring and transforming neural activity into operational commands. State of the art BCI systems measure brain activity using penetrating electrode arrays able to record from hundreds of individual cortical neurons simultaneously. Unfortunately, these systems are highly susceptible to signal degradation which limits their efficacy to 1-2 years. However, electrocorticography (ECoG) signals recorded from the surface of the brain deliver a more competitive balance between surgical risk, long-term stability, signal bandwidth, and signal-to-noise ratio when compared to both the aforementioned intracortical systems and the more common non-invasive electroencephalography (EEG) technologies. Historically, neural signals for controlling a computer cursor or robotic arm have been mapped to extrinsic, kinematic (i.e. position or velocity) variables. Although this strategy is adequate for use in simple environments, it may not be ideal for control of real-world prosthetic devices that are subject to external and unexpected forces. When reaching for an object, the trajectory of the hand through space can be defined in either extrinsic (e.g. Cartesian) or intrinsic (e.g. joint angles, muscle forces) frames of reference. During this movement, the brain has to perform a series of sensorimotor transformations that involve solving a complex, 2nd order differential equation (i.e. musculoskeletal biomechanics) in order to determine the appropriate muscle activations. Functional neuromuscular stimulation (FNS) is a desirable BCI application because it attempts to restore motor function to paralyzed limbs through electrical excitation of muscles. Rather than applying the conventional extrinsic kinematic control signals to such a system, it may be more appropriate to map neural activity to muscle activation directly and allow the brain to develop its own transfer function. This dissertation examines the application of intrinsic decoding schemes to control an upper limb using ECoG in non-human primates. ECoG electrode arrays were chronically implanted in rhesus monkeys over sensorimotor cortex. A novel multi-joint reaching task was developed to train the subjects to control a virtual arm simulating muscle and inertial forces. Utilizing a co-adaptive algorithm (where both the brain adapts via biofeedback and the decoding algorithm adapts to improve performance), new decoding models were initially built over the course of the first 3-5 minutes of each daily experimental session and then continually adapted throughout the day. Three subjects performed the task using neural control signals mapped to 1) joint angular velocity, 2) joint torque, and 3) muscle forces of the virtual arm. Performance exceeded 97%, 93%, and 89% accuracy for the three control paradigms respectively. Neural control features in the upper gamma frequency bands (70-115 and 130-175 Hz) were found to be directionally tuned in an ordered fashion, with preferred directions varying topographically in the mediolateral direction without distinction between motor and sensory areas. Long-term stability was demonstrated by all three monkeys, which maintained performance at 42, 55, and 57 months post-implantation. These results provide insights into the capabilities of sensorimotor cortex for control of non-linear multi-joint reaching dynamics and present a first step toward design of intrinsic, force-based BCI systems suitable for long-term FNS applications

A Wireless Brain-Machine Interface for Real-Time Speech Synthesis

This is the published version, also available here: http://dx.doi.org/10.1371/journal.pone.0008218.Background Brain-machine interfaces (BMIs) involving electrodes implanted into the human cerebral cortex have recently been developed in an attempt to restore function to profoundly paralyzed individuals. Current BMIs for restoring communication can provide important capabilities via a typing process, but unfortunately they are only capable of slow communication rates. In the current study we use a novel approach to speech restoration in which we decode continuous auditory parameters for a real-time speech synthesizer from neuronal activity in motor cortex during attempted speech. Methodology/Principal Findings Neural signals recorded by a Neurotrophic Electrode implanted in a speech-related region of the left precentral gyrus of a human volunteer suffering from locked-in syndrome, characterized by near-total paralysis with spared cognition, were transmitted wirelessly across the scalp and used to drive a speech synthesizer. A Kalman filter-based decoder translated the neural signals generated during attempted speech into continuous parameters for controlling a synthesizer that provided immediate (within 50 ms) auditory feedback of the decoded sound. Accuracy of the volunteer's vowel productions with the synthesizer improved quickly with practice, with a 25% improvement in average hit rate (from 45% to 70%) and 46% decrease in average endpoint error from the first to the last block of a three-vowel task. Conclusions/Significance Our results support the feasibility of neural prostheses that may have the potential to provide near-conversational synthetic speech output for individuals with severely impaired speech motor control. They also provide an initial glimpse into the functional properties of neurons in speech motor cortical areas

ECoG correlates of visuomotor transformation, neural plasticity, and application to a force-based brain computer interface

Electrocorticography: ECoG) has gained increased notoriety over the past decade as a possible recording modality for Brain-Computer Interface: BCI) applications that offers a balance of minimal invasiveness to the patient in addition to robust spectral information over time. More recently, the scale of ECoG devices has begun to shrink to the order of micrometer diameter contacts and millimeter spacings with the intent of extracting more independent signals for BCI control within less cortical real-estate. However, most control signals to date, whether within the field of ECoG or any of the more seasoned recording techniques, have translated their control signals to kinematic control parameters: i.e. position or velocity of an object) which may not be practical for certain BCI applications such as functional neuromuscular stimulation: FNS). Thus, the purpose of this dissertation was to present a novel application of ECoG signals to a force-based control algorithm and address its feasibility for such a BCI system. Micro-ECoG arrays constructed from thin-film polyimide were implanted epidurally over areas spanning premotor, primary motor, and parietal cortical areas of two monkeys: three hemispheres, three arrays). Monkeys first learned to perform a classic center-out task using a brain signal-to-velocity mapping for control of a computer cursor. The BCI algorithm utilized day-to-day adaptation of the decoding model to match the task intention of the monkeys with no need for pre-screeening of movement-related ECoG signals. Using this strategy, subjects showed notable 2-D task profiency and increased task-related modulation of ECoG features within five training sessions. After fixing the last model trained for velocity control of the cursor, the monkeys then utilized this decoding model to control the acceleration of the cursor in the same center-out task. Cursor movement profiles under this mapping paralleled those demonstrated using velocity control, and neural control signal profiles revealed the monkeys actively accelerated and decelerated the cursor within a limited time window: 1-1.5 seconds). The fixed BCI decoding model was recast once again to control the force on a virtual cursor in a novel mass-grab task. This task required targets not only to reach to peripheral targets but also account for an additional virtual mass as they grabbed each target and moved it to a second target location in the presence of the external force of gravity. Examination of the ensemble control signals showed neural adaptation to variations in the perceived mass of the target as well as the presence or absence of gravity. Finally, short rest periods were interleaved within blocks of each task type to elucidate differences between active BCI intention and rest. Using a post-hoc state-decoder model, periods of active BCI task control could be distinguished from periods of rest with a very high degree of accuracy: ~99%). Taken together, the results from these experiments present a first step toward the design of a dynamics-based BCI system suitable for FNS applications as well as a framework for implementation of an asyncrhonous ECoG BCI