Real-time decoding of question-and-answer speech dialogue using human cortical activity.

Natural communication often occurs in dialogue, differentially engaging auditory and sensorimotor brain regions during listening and speaking. However, previous attempts to decode speech directly from the human brain typically consider listening or speaking tasks in isolation. Here, human participants listened to questions and responded aloud with answers while we used high-density electrocorticography (ECoG) recordings to detect when they heard or said an utterance and to then decode the utterance's identity. Because certain answers were only plausible responses to certain questions, we could dynamically update the prior probabilities of each answer using the decoded question likelihoods as context. We decode produced and perceived utterances with accuracy rates as high as 61% and 76%, respectively (chance is 7% and 20%). Contextual integration of decoded question likelihoods significantly improves answer decoding. These results demonstrate real-time decoding of speech in an interactive, conversational setting, which has important implications for patients who are unable to communicate

Understanding and Decoding Imagined Speech using Electrocorticographic Recordings in Humans

Certain brain disorders, resulting from brainstem infarcts, traumatic brain injury, stroke and amyotrophic lateral sclerosis, limit verbal communication despite the patient being fully aware. People that cannot communicate due to neurological disorders would benefit from a system that can infer internal speech directly from brain signals. Investigating how the human cortex encodes imagined speech remains a difficult challenge, due to the lack of behavioral and observable measures. As a consequence, the fine temporal properties of speech cannot be synchronized precisely with brain signals during internal subjective experiences, like imagined speech. This thesis aims at understanding and decoding the neural correlates of imagined speech (also called internal speech or covert speech), for targeting speech neuroprostheses. In this exploratory work, various imagined speech features, such as acoustic sound features, phonetic representations, and individual words were investigated and decoded from electrocorticographic signals recorded in epileptic patients in three different studies. This recording technique provides high spatiotemporal resolution, via electrodes placed beneath the skull, but without penetrating the cortex In the first study, we reconstructed continuous spectrotemporal acoustic features from brain signals recorded during imagined speech using cross-condition linear regression. Using this technique, we showed that significant acoustic features of imagined speech could be reconstructed in seven patients. In the second study, we decoded continuous phoneme sequences from brain signals recorded during imagined speech using hidden Markov models. This technique allowed incorporating a language model that defined phoneme transitions probabilities. In this preliminary study, decoding accuracy was significant across eight phonemes in one patients. In the third study, we classified individual words from brain signals recorded during an imagined speech word repetition task, using support-vector machines. To account for temporal irregularities during speech production, we introduced a non-linear time alignment into the classification framework. Classification accuracy was significant across five patients. In order to compare speech representations across conditions and integrate imagined speech into the general speech network, we investigated imagined speech in parallel with overt speech production and/or speech perception. Results shared across the three studies showed partial overlapping between imagined speech and speech perception/production in speech areas, such as superior temporal lobe, anterior frontal gyrus and sensorimotor cortex. In an attempt to understanding higher-level cognitive processing of auditory processes, we also investigated the neural encoding of acoustic features during music imagery using linear regression. Despite this study was not directly related to speech representations, it provided a unique opportunity to quantitatively study features of inner subjective experiences, similar to speech imagery. These studies demonstrated the potential of using predictive models for basic decoding of speech features. Despite low performance, results show the feasibility for direct decoding of natural speech. In this respect, we highlighted numerous challenges that were encountered, and suggested new avenues to improve performances

Electrophysiological signatures of event segmentation during movie viewing and recall

Tese de mestrado integrado, Engenharia Biomédica e Biofísica (Sinais e Imagens Médicas) Universidade de Lisboa, Faculdade de Ciências, 2018Perception and memory have been widely studied in the context of discrete pictures or words. However, in real-life, we are faced with a continuous stream of perceptual input that arrive on a wide range of timescales. Previous studies have shown that our brain can segment this continuous stream of information into events that not only reveal a hierarchy from coarse to fine time-scales, but also integrate them differently throughout the cortex, with processing timescales increasing from tens of milliseconds in early sensory regions up to hundreds of seconds in higherorder regions. However, the neural mechanisms that support such event segmentation process during online encoding of a naturalistic and continuous experience remain unknown. To address this issue, we tested whether the formation of meaningful event models could be expressed by specific patterns of electrophysiological activity recorded from healthy humans elicited during the online encoding of a 50 minutes movie and if these patterns were predictive of participant’s later memory recall of the encoded events during a free verbal and unguided recall test. Our results provide the first electrophysiological evidence for a memory related oscillatory signature of the event segmentation process. We found that, when facing a naturalistic continuous stimuli, our brain perceives the information in the form of discrete events which are stored in memory during a process that seems to occur approximately 400 ms after the end of each event and that is indeed predictive of later reinstatement of those events. This neural process suggests a possible congruency/incongruency evaluation mechanism which might represent the errorbased updating mechanism, in which event models are updated at event boundaries in response to transient increases in prediction error, suggested by the Event Segmentation Theory. Our results also prove that naturalistic stimuli can be used in electroencephalography measurements, despite the natural limitations that arise with the use of such stimuli. In doing so we will be able to study, in a more ecological way, the mechanisms of memory formation during event segmentation.As memórias podem ser definidas como representações duradouras de eventos ocorridos no passado que se reflectem em pensamentos, experiências e comportamentos. Conscientemente ou não, elas são influenciadas pelo passado, necessárias para o nosso dia-a-dia e extremamente importantes no planeamento do nosso futuro. O armazenamento de memórias é feito por parte de inúmeras estruturas cerebrais pertencentes ao neocórtex - parte exterior do córtex - e pelo lobo temporal medial, onde podemos encontrar o hipocampo e seus tecidos envolventes. Estas duas regiões encontram-se em constante diálogo enquanto armazenamos e restabelecemos o fluxo de experiências diárias. Com base no estudo de lesões cerebrais nestas regiões, sabe-se hoje em dia que é possível dividir as memórias em diferentes categorias. As duas principais categorias são memória a curto prazo e a longo prazo. A primeira é responsável pelo armazenamento de informação temporariamente e se essa informação deve ou não ser transferida para a memória a longo prazo. Estas últimas são compostas por memórias conscientes e inconscientes, entre elas as memórias semânticas, episódicas e mais particularmente autobiográficas. Inicialmente as memórias são episódicas e armazenadas no hipocampo e com o tempo são transformadas em memórias semânticas no neocórtex. Recentemente, uma nova linha de investigação tem se focado não na distribuição espacial do processo de armazenamento de memórias mas sim nas suas propriedades temporais e capacidade de distinguir informação que nos é apresentada com diferentes escalas temporais. Por exemplo, quando falamos ou lemos um texto, somos obrigados a identificar as diferentes sílabas para conseguir formar uma palavra, a perceber o sentido dessa palavra no contexto de uma frase, e uma frase no contexto de uma conversa. Não só temos a capacidade de segmentar a informação que percepcionamos mas também de nos lembrarmos desta informação na forma de episódios representativos das nossas experiências prévias. A primeira teoria formulada sobre este processo de segmentação toma o nome de Event Segmentation Theory. Nela a informação processada pelo nosso cérebro é representada por uma série de modelos de eventos, implementados durante rápidas alterações neuronais que ocorrem devido a um mecanismo de avaliação de predição do que deverá acontecer no seguimento de certa experiência. Esta alteração no erro de predição leva à segmentação da informação em diferentes eventos durante momentos a que se dá o nome de fronteiras. Desde a formulação desta teoria inúmeros estudos foram desenvolvidos com o intuito de compreender o funcionamento deste processo de segmentação. Estes estudos permitiramnos obter provas de que a nossa actividade neuronal tem a capacidade de processar informação em diferentes escalas temporais, em diferentes regiões do cérebro. Zonas de processamento sensorial segmentam informação em eventos mais curtos, uma vez que os estímulos sensoriais são geralmente muito rápidos, e zonas de processamento elevado segmentam informação em eventos mais longos uma vez que têm de processar informação perceptual e cognitiva. No entanto, apenas recentemente começaram a surgir estudos que ligam este processo de segmentação à formação de memórias a longo prazo. Uma nova teoria, Theory of Event Segmentation and Memory, proposta o ano passado sugere que cada região cerebral processa informação na sua escala temporal preferida e que estes segmentos de informação são transmitidos de regiões que processam informação a escalas temporais longas para o hipocampo, minutos após ser formada uma fronteira entre dois eventos. Ao ser activado, o hipocampo processa a informação do evento acabado de percepcionar e armazena a informação para que esta possa ser mais tarde reativada nas mesmas regiões de escala temporal longa. Após a formalização desta teoria uma série de outros estudos têm sugerido como principal responsável para a integração da informação de um evento na memória, a resposta neuronal que parece ocorrer durante as fronteiras entre eventos. No entanto, os mecanismos neuronais que suportam esta segmentação e integração na memória durante o processamento de experiências naturais e contínuas continuam por explicar. Com o objetivo de explorar estes mecanismos, neste projeto testámos a possibilidade de a formação de modelos de eventos ser expressa por padrões eletrofisiológicos particulares a este processo. Para tal adquirimos dados de electroencefalograma (EEG) de 30 participantes saudáveis enquanto estes visualizavam 50 min de um filme. Para testar se o aparecimento de padrões neuronais específicos poderia ser preditivo de um correcto processo de memória pedimos aos participantes para, após a visualização do filme, relatar o que tinham acabado de ver de forma livre, mantendo a ordem em que a informação foi apresentada no filme e por quanto tempo conseguissem. Os dados adquiridos foram pré-processados de forma a eliminar a maior quantidade de ruído possível e um modelo com a possível segmentação do filme em diferentes cenas foi construído com base em anotações de seis participantes externos. Após verificar que o sinal adquirido podia ser dividido nos eventos compostos pelo modelo construído este foi utilizado na primeira parte da análise em que o objectivo era avaliar o que se passava no interior dos eventos. Para tal os padrões neuronais adquiridos tanto durante a visualização do filme como durante o relato do mesmo foram comparados entre si para cada participante e comparados entre participantes. Verificámos que os padrões neuronais eram semelhantes entre participantes tanto para os dados obtidos durante a visualização do filme, em que o estímulo é o mesmo para todos os participantes, como para os dados adquiridos durante o relato, em que os participantes descrevem o filme de forma diferente. Apenas obtivemos elevados valores de semelhança entre os padrões do filme e o relato quando recorremos a um algoritmo de segmentação baseado em Hidden Markov Models para que a segmentação dos dados do relato fosse feita de forma individual para cada participante. Estes resultados permitem-nos concluir que o processo de armazenamento e restabelecimento de memórias é feito de forma semelhante e com base em eventos. Correlações entre diferentes propriedades dos eventos (duração do evento, ordenação dos eventos durante o relato, detalhes relatados em cada evento e autocorrelação do padrão neuronal de cada evento) e a precisão com o que filme é relatado para cada participante foram calculadas de forma a perceber se alguma destas propriedades poderia prever se um evento seria mais tarde relembrado durante o relato ou não. Apenas a duração do evento mostrou ser significativa o que indica que os processos que se desenvolvem durante a visualização de um evento não parecem ser decisivos para a sua integração na memória. Após estudar o que se passava no interior de cada evento a nossa atenção voltou-se para as fronteiras entre eventos. Para tal começámos por realizar uma análise de similaridade espaçotemporal (STPS) em que comparámos os padrões neuronais 5 s após as fronteiras com os 5 s antes das fronteiras e observámos que de facto existe uma alteração nos padrões neuronais quando uma fronteira ocorre, e que esta alteração não pode simplesmente ser explicada por uma distância temporal entre os dois eventos. Para observarmos então com mais distinção a resposta neuronal durante a fronteira, calculámos os event related potentials (ERPs), 2 s após cada fronteira, para todas as fronteiras e todos os participantes. Nestes, encontrámos uma clara distinção entre fronteiras correspondentes a eventos que não foram e que foram mais tarde relatados. A resposta neuronal dos eventos mais tarde relembrados está marcada pelo aparecimento do componente N400, conhecido por aparecer quando ocorre uma incongruência na informação a ser percepcionada. Estes resultados sugerem que, quando uma fronteira ocorre dá-se uma avaliação de congruência com a informação do evento passado e, quando mais incongruente for esta informação, melhor será armazenada na memória e mais tarde relembrada. Este mecanismo está de acordo com o mecanismo de avaliação de previsão proposto pela Event Segmentation Theory. Em suma os nossos resultados demonstram a existência de um padrão neuronal característico do processo de segmentação com aparecimento aproximadamente 400 ms após a formação de uma fronteira entre eventos, crucial para a correta integração desse evento na memória. Os nossos resultados provam também a validade de utilização de um estímulo naturalístico em estudos de segmentação de memória que utilizam medições electrofisiológicas. Este estudo abre portas para investigações futuras em que será essencial determinar como ocorre a distribuição espacial deste padrão neuronal, aqui apenas sugerida devido à baixa resolução espacial do EEG, e validar a existência deste padrão em estudos cada vez mais naturalísticos, com recurso por exemplo a medições por electrocorticografia (ECoG)

Cortical encoding and decoding models of speech production

To speak is to dynamically orchestrate the movements of the articulators (jaw, tongue, lips, and larynx), which in turn generate speech sounds. It is an amazing mental and motor feat that is controlled by the brain and is fundamental for communication. Technology that could translate brain signals into speech would be transformative for people who are unable to communicate as a result of neurological impairments. This work first investigates how articulator movements that underlie natural speech production are represented in the brain. Building upon this, this work also presents a neural decoder that can synthesize audible speech from brain signals. Data to support these results were from direct cortical recordings of the human sensorimotor cortex while participants spoke natural sentences. Neural activity at individual electrodes encoded a diversity of articulatory kinematic trajectories (AKTs), each revealing coordinated articulator movements towards specific vocal tract shapes. The neural decoder was designed to leverage the kinematic trajectories encoded in the sensorimotor cortex which enhanced performance even with limited data. In closed vocabulary tests, listeners could readily identify and transcribe speech synthesized from cortical activity. These findings advance the clinical viability of using speech neuroprosthetic technology to restore spoken communication

Speech Processes for Brain-Computer Interfaces

Speech interfaces have become widely used and are integrated in many applications and devices. However, speech interfaces require the user to produce intelligible speech, which might be hindered by loud environments, concern to bother bystanders or the general in- ability to produce speech due to disabilities. Decoding a usera s imagined speech instead of actual speech would solve this problem. Such a Brain-Computer Interface (BCI) based on imagined speech would enable fast and natural communication without the need to actually speak out loud. These interfaces could provide a voice to otherwise mute people. This dissertation investigates BCIs based on speech processes using functional Near In- frared Spectroscopy (fNIRS) and Electrocorticography (ECoG), two brain activity imaging modalities on opposing ends of an invasiveness scale. Brain activity data have low signal- to-noise ratio and complex spatio-temporal and spectral coherence. To analyze these data, techniques from the areas of machine learning, neuroscience and Automatic Speech Recog- nition are combined in this dissertation to facilitate robust classification of detailed speech processes while simultaneously illustrating the underlying neural processes. fNIRS is an imaging modality based on cerebral blood flow. It only requires affordable hardware and can be set up within minutes in a day-to-day environment. Therefore, it is ideally suited for convenient user interfaces. However, the hemodynamic processes measured by fNIRS are slow in nature and the technology therefore offers poor temporal resolution. We investigate speech in fNIRS and demonstrate classification of speech processes for BCIs based on fNIRS. ECoG provides ideal signal properties by invasively measuring electrical potentials artifact- free directly on the brain surface. High spatial resolution and temporal resolution down to millisecond sampling provide localized information with accurate enough timing to capture the fast process underlying speech production. This dissertation presents the Brain-to- Text system, which harnesses automatic speech recognition technology to decode a textual representation of continuous speech from ECoG. This could allow to compose messages or to issue commands through a BCI. While the decoding of a textual representation is unparalleled for device control and typing, direct communication is even more natural if the full expressive power of speech - including emphasis and prosody - could be provided. For this purpose, a second system is presented, which directly synthesizes neural signals into audible speech, which could enable conversation with friends and family through a BCI. Up to now, both systems, the Brain-to-Text and synthesis system are operating on audibly produced speech. To bridge the gap to the final frontier of neural prostheses based on imagined speech processes, we investigate the differences between audibly produced and imagined speech and present first results towards BCI from imagined speech processes. This dissertation demonstrates the usage of speech processes as a paradigm for BCI for the first time. Speech processes offer a fast and natural interaction paradigm which will help patients and healthy users alike to communicate with computers and with friends and family efficiently through BCIs

A Novel Framework of Online, Task-Independent Cognitive State Transition Detection and Its Applications

Complex reach, grasp, and object manipulation tasks require sequential, temporal coordination of a movement plan by neurons in the brain. Detecting cognitive state transitions associated with motor tasks from sequential neural data is pivotal in rehabilitation engineering. The cognitive state detectors proposed thus far rely on task-dependent (TD) models, i.e., the detection strategy exploits a priori knowledge of the movement goals to determine the actual states, regardless of whether these cognitive states actually depend on the movement tasks or not. This approach, however, is not viable when the tasks are not known a priori (e.g., the subject performs many different tasks) or when there is a paucity of neural data for each task. Moreover, some cognitive states (e.g., holding) are invariant to the tasks performs. I first develop an offline, task-dependent cognitive state transition detector and a kinematics decoder to show the feasibility of distinguishing between cognitive states based on their inherent features extracted via a hidden Markov model (HMM) based detection framework. The proposed framework is designed to decode both cognitive states and kinematics from ensemble neural activity. The proposed decoding framework is able to a) automatically differentiate between baseline, plan, and movement, and b) determine novel holding epochs of neural activity and also estimate the epoch-dependent kinematics. Specifically, the framework is mainly composed of a hidden Markov model (HMM) state decoder and a switching linear system (S-LDS) kinematics decoder. I take a supervised approach and use a generative framework of neural activity and kinematics. I demonstrate the decoding framework using neural recordings from ventral premotor (PMv) and dorsal premotor (PMd) neurons of a non-human primate executing four complex reach-to-grasp tasks along with the corresponding kinematics recording. Using the HMM state decoder, I demonstrate that the transitions between neighboring epochs of neural activity, regardless of the existence of any external kinematics changes, can be detected with high accuracy (>85%) and short latencies (<150 ms). I further show that the joint angle kinematics can be estimated reliably with high accuracy (mean = 88%) using a S-LDS kinematics decoder. In addition, I demonstrate that the use of multiple latent state variables to model the within-epoch neural activity variability can improve the decoder performance. This unified decoding framework combining a HMM state decoder and a S-LDS may be useful in neural decoding of cognitive states and complex movements of prosthetic limbs in practical brain-computer interface implementations. I then develop a real-time (online) task-independent (TI) framework to detect cognitive state transitions from spike trains and kinematic measurements. I applied this framework to 226 single-unit recordings collected via multi-electrode arrays in the premotor dorsal and ventral (PMd and PMv) regions of the cortex of two non-human primates performing 3D multi-object reach-to-grasp tasks, and I used the detection latency and accuracy of state transitions to measure the performance. I found that, in both online and offline detection modes, (i) TI models have significantly better performance than TD models when using neuronal data alone, however (ii) during movements, the addition of the kinematics history to the TI models further improves detection performance. These findings suggest that TI models may be able to more accurately detect cognitive state transitions than TD under certain circumstances. The proposed framework could pave the way for a TI control of prosthesis from cortical neurons, a beneficial outcome when the choice of tasks is vast, but despite that the basic movement cognitive states need to be decoded. Based on the online cognitive state transition detector, I further construct an online task-independent kinematics decoder. I constructed our framework using single-unit recordings from 452 neurons and synchronized kinematics recordings from two non-human primates performing 3D multi-object reach-to-grasp tasks. I find that (i) the proposed TI framework performs significantly better than current frameworks that rely on TD models (p = 0.03); and (ii) modeling cognitive state information further improves decoding performance. These findings suggest that TI models with cognitive-state-dependent parameters may more accurately decode kinematics and could pave the way for more clinically viable neural prosthetics

Neural Connectivity with Hidden Gaussian Graphical State-Model

The noninvasive procedures for neural connectivity are under questioning. Theoretical models sustain that the electromagnetic field registered at external sensors is elicited by currents at neural space. Nevertheless, what we observe at the sensor space is a superposition of projected fields, from the whole gray-matter. This is the reason for a major pitfall of noninvasive Electrophysiology methods: distorted reconstruction of neural activity and its connectivity or leakage. It has been proven that current methods produce incorrect connectomes. Somewhat related to the incorrect connectivity modelling, they disregard either Systems Theory and Bayesian Information Theory. We introduce a new formalism that attains for it, Hidden Gaussian Graphical State-Model (HIGGS). A neural Gaussian Graphical Model (GGM) hidden by the observation equation of Magneto-encephalographic (MEEG) signals. HIGGS is equivalent to a frequency domain Linear State Space Model (LSSM) but with sparse connectivity prior. The mathematical contribution here is the theory for high-dimensional and frequency-domain HIGGS solvers. We demonstrate that HIGGS can attenuate the leakage effect in the most critical case: the distortion EEG signal due to head volume conduction heterogeneities. Its application in EEG is illustrated with retrieved connectivity patterns from human Steady State Visual Evoked Potentials (SSVEP). We provide for the first time confirmatory evidence for noninvasive procedures of neural connectivity: concurrent EEG and Electrocorticography (ECoG) recordings on monkey. Open source packages are freely available online, to reproduce the results presented in this paper and to analyze external MEEG databases