Unsupervised decoding of long-term, naturalistic human neural recordings with automated video and audio annotations

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Most ongoing efforts have focused on training decoders on specific, stereotyped tasks in laboratory settings. Implementing brain-computer interfaces (BCIs) in natural settings requires adaptive strategies and scalable algorithms that require minimal supervision. Here we propose an unsupervised approach to decoding neural states from human brain recordings acquired in a naturalistic context. We demonstrate our approach on continuous long-term electrocorticographic (ECoG) data recorded over many days from the brain surface of subjects in a hospital room, with simultaneous audio and video recordings. We first discovered clusters in high-dimensional ECoG recordings and then annotated coherent clusters using speech and movement labels extracted automatically from audio and video recordings. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Our results show that our unsupervised approach can discover distinct behaviors from ECoG data, including moving, speaking and resting. We verify the accuracy of our approach by comparing to manual annotations. Projecting the discovered cluster centers back onto the brain, this technique opens the door to automated functional brain mapping in natural settings

Gesture Decoding Using ECoG Signals from Human Sensorimotor Cortex: A Pilot Study

Electrocorticography (ECoG) has been demonstrated as a promising neural signal source for developing brain-machine interfaces (BMIs). However, many concerns about the disadvantages brought by large craniotomy for implanting the ECoG grid limit the clinical translation of ECoG-based BMIs. In this study, we collected clinical ECoG signals from the sensorimotor cortex of three epileptic participants when they performed hand gestures. The ECoG power spectrum in hybrid frequency bands was extracted to build a synchronous real-time BMI system. High decoding accuracy of the three gestures was achieved in both offline analysis (85.7%, 84.5%, and 69.7%) and online tests (80% and 82%, tested on two participants only). We found that the decoding performance was maintained even with a subset of channels selected by a greedy algorithm. More importantly, these selected channels were mostly distributed along the central sulcus and clustered in the area of 3 interelectrode squares. Our findings of the reduced and clustered distribution of ECoG channels further supported the feasibility of clinically implementing the ECoG-based BMI system for the control of hand gestures

Cortical Mirror-System Activation During Real-Life Game Playing: An Intracranial Electroencephalography (EEG) Study

Analogous to the mirror neuron system repeatedly described in monkeys as a possible substrate for imitation learning and/or action understanding, a neuronal execution/observation matching system (OEMS) is assumed in humans, but little is known to what extent this system is activated in non-experimental, real-life conditions. In the present case study, we investigated brain activity of this system during natural, non-experimental motor behavior as it occurred during playing of the board game "Malefiz". We compared spectral modulations of the high-gamma band related to ipsilateral reaching movement execution and observation of the same kind of movement using electrocorticography (ECoG) in one participant. Spatially coincident activity during both conditions execution and observation was recorded at electrode contacts over the premotor/primary motor cortex. The topography and amplitude of the high-gamma modulations related to both, movement observation and execution were clearly spatially correlated over several fronto-parietal brain areas. Thus, our findings indicate that a network of cortical areas contributes to the human OEMS, beyond primary/premotor cortex including Brocas area and the temporo-parieto-occipital junction area, in real-life conditions.Comment: 4 pages, 2 figure, CCN 2018 conference pape

Noninvasive Neuroprosthetic Control of Grasping by Amputees

Smooth coordination and fine temporal control of muscles by the brain allows us to effortlessly pre-shape our hand to grasp different objects. Correlates of motor control for grasping have been found across wide-spread cortical areas, with diverse signal features. These signals have been harnessed by implanting intracortical electrodes and used to control the motion of robotic hands by tetraplegics, using algorithms called brain-machine interfaces (BMIs). Signatures of motor control signal encoding mechanisms of the brain in macro-scale signals such as electroencephalography (EEG) are unknown, and could potentially be used to develop noninvasive brain-machine interfaces. Here we show that a) low frequency (0.1 – 1 Hz) time domain EEG contains information about grasp pre-shaping in able-bodies individuals, and b) This information can be used to control pre-shaping motion of a robotic hand by amputees. In the first study, we recorded simultaneous EEG and hand kinematics as 5 able-bodies individuals grasped various objects. Linear decoders using low delta band EEG amplitudes accurately predicted hand pre-shaping kinematics during grasping. Correlation coefficient between predicted and actual kinematics was r = 0.59 ± 0.04, 0.47 ± 0.06 and 0.32 ± 0.05 for the first 3 synergies. In the second study, two transradial amputees (A1 and A2) controlled a prosthetic hand to grasp two objects using a closed-loop BMI with low delta band EEG. This study was conducted longitudinally in 12 sessions spread over 38 days. A1 achieved a 63% success rate, with 11 sessions significantly above chance. A2 achieved a 32% success rate, with 2 sessions significantly above chance. Previous methods of EEG-based BMIs used frequency domain features, and were thought to have a low signal-to-noise ratio making them unsuitable for control of dexterous tasks like grasping. Our results demonstrate that time-domain EEG contains information about grasp pre-shaping, which can be harnessed for neuroprosthetic control.Electrical and Computer Engineering, Department o

Doctor of Philosophy

dissertationRecording the neural activity of human subjects is indispensable for fundamental neuroscience research and clinical applications. Human studies range from examining the neural activity of large regions of the cortex using electroencephalography (EEG) or electrocorticography (ECoG) to single neurons or small populations of neurons using microelectrode arrays. In this dissertation, microscale recordings in the human cortex were analyzed during administration of propofol anesthesia and articulate movements such as speech, finger flexion, and arm reach. Recordings were performed on epilepsy patients who required long-term electrocorticographic monitoring and were implanted with penetrating or surface microelectrode arrays. We used penetrating microelectrode arrays to investigate the effects of propofol anesthesia on action potentials (APs) and local field potentials (LFPs). Increased propofol concentration correlated with decreased high-frequency power in LFP spectra and decreased AP firing rates, as well as the generation of large amplitude spike-like LFP activity; however, the temporal relationship between APs and LFPs remained relatively consistent at all levels of propofol anesthesia. The propofol-induced suppression of neocortical network activity allowed LFPs to be dominated by low-frequency spike-like activity, and correlated with sedation and unconsciousness. As the low-frequency spike-like activity increased, and the AP-LFP relationship became more predictable, firing rate encoding capacity was impaired. This suggests a mechanism for decreased information processing in the neocortex that accounts for propofol-induced unconsciousness. We also demonstrated that speech, finger, and arm movements can be decoded from LFPs recorded with dense grids of microelectrodes placed on the surface of human cerebral cortex for brain computer interface (BCI) applications using LFPs recorded over face-motor area, vocalized articulations of ten different words and silence were classified on a trial-by-trial basis with 82.4% accuracy. Using LFPs recorded over the hand area of motor cortex, three individual finger movements and rest were classified on a trial-by-trial basis with 62% accuracy. LFPs recorded over the arm area of motor cortex were used to continuously decode the arm trajectory with a maximum correlation coefficient of 0.82 in the x-direction and 0.76 in the y-direction. These findings demonstrate that LFPs recorded by micro-ECoG grids from the surface of the cerebral cortex contain sufficient information to provide rapid and intuitive control a BCI communication or motor prosthesis