Leveraging Spatiotemporal Relationships of High-frequency Activation in Human Electrocorticographic Recordings for Speech Brain-Computer-Interface

Speech production is one of the most intricate yet natural human behaviors and is most keenly appreciated when it becomes difficult or impossible; as is the case for patients suffering from locked-in syndrome. Burgeoning understanding of the various cortical representations of language has brought into question the viability of a speech neuroprosthesis using implanted electrodes. The temporal resolution of intracranial electrophysiological recordings, frequently billed as a great asset of electrocorticography (ECoG), has actually been a hindrance as speech decoders have struggled to take advantage of this timing information. There have been few demonstrations of how well a speech neuroprosthesis will realistically generalize across contexts when constructed using causal feature extraction and language models that can be applied and adapted in real-time. The research detailed in this dissertation aims primarily to characterize the spatiotemporal relationships of high frequency activity across ECoG arrays during word production. Once identified, these relationships map to motor and semantic representations of speech through the use of algorithms and classifiers that rapidly quantify these relationships in single-trials. The primary hypothesis put forward by this dissertation is that the onset, duration and temporal profile of high frequency activity in ECoG recordings is a useful feature for speech decoding. These features have rarely been used in state-of-the-art speech decoders, which tend to produce output from instantaneous high frequency power across cortical sites, or rely upon precise behavioral time-locking to take advantage of high frequency activity at several time-points relative to behavioral onset times. This hypothesis was examined in three separate studies. First, software was created that rapidly characterizes spatiotemporal relationships of neural features. Second, semantic representations of speech were examined using these spatiotemporal features. Finally, utterances were discriminated in single-trials with low latency and high accuracy using spatiotemporal matched filters in a neural keyword-spotting paradigm. Outcomes from this dissertation inform implant placement for a human speech prosthesis and provide the scientific and methodological basis to motivate further research of an implant specifically for speech-based brain-computer-interfaces

음성 의미 지각시의 고등 언어 성분 처리 디코딩

학위논문(석사) -- 서울대학교대학원 : 자연과학대학 협동과정 뇌과학전공, 2022. 8. 정천기.High-level linguistic processing in the human brain remains incompletely understood and constitutes a challenging topic in speech neuroscience. While most studies focused on decoding low-level phonetic components using intracranial recordings of the human brain during speech perception, few studies have attempted to decode high-level syntactic or semantic features. If any, most of the research targeting semantic decoding is conducted with picture naming tasks, which only deal with visual language rather than spoken language. The presenting study is focused on better characterizing the neural representations of processing spoken language perception, namely speech perception. Especially not on the lower-level language components such as phonemes or phonetics, but the higher-level components such as syntax and semantics. Since it is widely accepted that the tripartite nature of language processing consists of phonology, syntax, and semantics, a strategical method for analyzing speech perception tasks that can reject the intervention of phonetic factors was mandatory. Therefore, we conducted a question-and-answer speech task containing four questions revolving around two semantic categories (alive, body parts) with phonetically controlled words. Intracranial neural signals were recorded during the question-and-answer speech task using electrocorticography (ECoG) electrodes for 14 epilepsy patients. Post hoc brain activity analysis was conducted for three subjects who answered correctly to every trial (144 trials in total) to ensure the analyzed data contained only brain signals collected during the correct semantic processing. The decoding results suggest that absolute and relative spectral neural feature trends occur across all participants in particular time windows. Furthermore, the spatial aspect of the neural features that yield the best decoding accuracy verifies the current biophysiological brain language model explaining the circular nature of word meaning comprehension in the left hemisphere language network.인간의 고등 성분 언어 처리와 관련한 두뇌 활동을 해독하는 연구는 신경언어학 분야에서도 아직 깊이 연구되지 않은 분야 중 하나이다. 침습적 전극을 통해 얻은 뇌피질 뇌파를 이용한 대부분의 언어 디코딩 연구는 음소나 음절 수준의 하위 언어 성분에서 진행되어 왔고, 통사나 의미와 같은 고등 언어 성분에 대한 디코딩 연구는 드물다. 드물게 진행된 고등 언어 성분 디코딩 연구 또한 대다수가 시각적 언어 처리를 연구한 결과들이며, 소리 언어 디코딩 연구는 태동 단계에 머무르고 있다. 본 연구는 소리 언어 지각시의 두뇌 활성을 분석하여 그 처리 과정의 뇌파 신호 특성을 규명하고자 한다. 특히 인간 음성 언어의 하위 구성 성분보다는 통사와 의미 위주의 고등 구성 성분을 처리하는 데에 관여하는 뇌파의 시간적, 주파수적, 공간적 특성에 집중하여 분석을 진행하였다. 언어 처리의 주된 세 가지 요소는 ‘음소 (phonetics)’, ‘통사 (syntactics)’, ‘의미 (semantics)’라는 점을 고려하여, 음소 소준의 뇌파 활동을 통제할 수 있는 실험 패러다임을 구상하였으며, 구체적으로는 두개의 다른 의미 범주 (생명, 신체)에 대해서 묻는 음소적으로 동등한 단어가 포함된 질문을 들려준 후 의미를 파악해 대답하는 과정의 뇌파를 기록하는 실험을 진행하였다. 뇌파 신호는 경막하 전극 삽입술 (Electrocorticography, ECoG)을 통해 14명의 뇌전증 환자로부터 침습적 방식으로 측정되었다. 뇌파 디코딩 분석에는 피험자의 두뇌가 옳은 방식으로 처리한 고등 언어 성분이 반영된 실험만을 포함하기 위해서 모든 실험에서 옳은 대답을 한 세 명의 환자만을 대상으로 하여 분석을 진행하였다. 디코딩 분석 결과 세 명의 환자에 걸쳐 핵심 단어 (‘것은’, ‘무엇입니까’) 음성 지각 이후 특정 시간대에서 특정 주파수대의 뇌파가 양 극단의 의미를 높은 수준의 정확도(%)로 분류하는 데에 사용될 수 있다는 것을 밝혔다. 또한 이러한 높은 정확도를 기록한 뇌파의 특성에는 모든 환자에 걸쳐 절대적 혹은 상대적 트렌드가 관찰되며, 관찰되는 뇌파의 공간적 특성은 현재 통용되는 신경언어학적 언어 처리 모델이 설명하는 음성 언어 처리 방식과 일맥상통함을 밝혔다.Abstract ⅰ 1. Introduction 1 2. Materials and Methods 4 3. Results 8 4. Discussion 12 References 15 List of Figures 20 Supplementary information 28 Abstract in Korean 36석

Dynamic activity patterns in the anterior temporal lobe represents object semantics.

The anterior temporal lobe (ATL) is considered a crucial area for the representation of transmodal concepts. Recent evidence suggests that specific regions within the ATL support the representation of individual object concepts, as shown by studies combining multivariate analysis methods and explicit measures of semantic knowledge. This research looks to further our understanding by probing conceptual representations at a spatially and temporally resolved neural scale. Representational similarity analysis was applied to human intracranial recordings from anatomically defined lateral to medial ATL sub-regions. Neural similarity patterns were tested against semantic similarity measures, where semantic similarity was defined by a hybrid corpus-based and feature-based approach. Analyses show that the perirhinal cortex, in the medial ATL, significantly related to semantic effects around 200 to 400 ms, and were greater than more lateral ATL regions. Further, semantic effects were present in low frequency (theta and alpha) oscillatory phase signals. These results provide converging support that more medial regions of the ATL support the representation of basic-level visual object concepts within the first 400 ms, and provide a bridge between prior fMRI and MEG work by offering detailed evidence for the presence of conceptual representations within the ATL

Characterization of high-gamma activity in electrocorticographic signals

INTRODUCTION: Electrocorticographic (ECoG) high-gamma activity (HGA) is a widely recognized and robust neural correlate of cognition and behavior. However, fundamental signal properties of HGA, such as the high-gamma frequency band or temporal dynamics of HGA, have never been systematically characterized. As a result, HGA estimators are often poorly adjusted, such that they miss valuable physiological information. METHODS: To address these issues, we conducted a thorough qualitative and quantitative characterization of HGA in ECoG signals. Our study is based on ECoG signals recorded from 18 epilepsy patients while performing motor control, listening, and visual perception tasks. In this study, we first categorize HGA into HGA types based on the cognitive/behavioral task. For each HGA type, we then systematically quantify three fundamental signal properties of HGA: the high-gamma frequency band, the HGA bandwidth, and the temporal dynamics of HGA. RESULTS: The high-gamma frequency band strongly varies across subjects and across cognitive/behavioral tasks. In addition, HGA time courses have lowpass character, with transients limited to 10 Hz. The task-related rise time and duration of these HGA time courses depend on the individual subject and cognitive/behavioral task. Task-related HGA amplitudes are comparable across the investigated tasks. DISCUSSION: This study is of high practical relevance because it provides a systematic basis for optimizing experiment design, ECoG acquisition and processing, and HGA estimation. Our results reveal previously unknown characteristics of HGA, the physiological principles of which need to be investigated in further studies

Perceptual and conceptual processing of visual objects across the adult lifespan

Abstract: Making sense of the external world is vital for multiple domains of cognition, and so it is crucial that object recognition is maintained across the lifespan. We investigated age differences in perceptual and conceptual processing of visual objects in a population-derived sample of 85 healthy adults (24–87 years old) by relating measures of object processing to cognition across the lifespan. Magnetoencephalography (MEG) was recorded during a picture naming task to provide a direct measure of neural activity, that is not confounded by age-related vascular changes. Multiple linear regression was used to estimate neural responsivity for each individual, namely the capacity to represent visual or semantic information relating to the pictures. We find that the capacity to represent semantic information is linked to higher naming accuracy, a measure of task-specific performance. In mature adults, the capacity to represent semantic information also correlated with higher levels of fluid intelligence, reflecting domain-general performance. In contrast, the latency of visual processing did not relate to measures of cognition. These results indicate that neural responsivity measures relate to naming accuracy and fluid intelligence. We propose that maintaining neural responsivity in older age confers benefits in task-related and domain-general cognitive processes, supporting the brain maintenance view of healthy cognitive ageing

Perceptual and conceptual processing of visual objects across the adult lifespan.

Making sense of the external world is vital for multiple domains of cognition, and so it is crucial that object recognition is maintained across the lifespan. We investigated age differences in perceptual and conceptual processing of visual objects in a population-derived sample of 85 healthy adults (24-87 years old) by relating measures of object processing to cognition across the lifespan. Magnetoencephalography (MEG) was recorded during a picture naming task to provide a direct measure of neural activity, that is not confounded by age-related vascular changes. Multiple linear regression was used to estimate neural responsivity for each individual, namely the capacity to represent visual or semantic information relating to the pictures. We find that the capacity to represent semantic information is linked to higher naming accuracy, a measure of task-specific performance. In mature adults, the capacity to represent semantic information also correlated with higher levels of fluid intelligence, reflecting domain-general performance. In contrast, the latency of visual processing did not relate to measures of cognition. These results indicate that neural responsivity measures relate to naming accuracy and fluid intelligence. We propose that maintaining neural responsivity in older age confers benefits in task-related and domain-general cognitive processes, supporting the brain maintenance view of healthy cognitive ageing.Research Foundation Flander

Cortical network responses map onto data-driven features that capture visual semantics of movie fragments

Research on how the human brain extracts meaning from sensory input relies in principle on methodological reductionism. In the present study, we adopt a more holistic approach by modeling the cortical responses to semantic information that was extracted from the visual stream of a feature film, employing artificial neural network models. Advances in both computer vision and natural language processing were utilized to extract the semantic representations from the film by combining perceptual and linguistic information. We tested whether these representations were useful in studying the human brain data. To this end, we collected electrocorticography responses to a short movie from 37 subjects and fitted their cortical patterns across multiple regions using the semantic components extracted from film frames. We found that individual semantic components reflected fundamental semantic distinctions in the visual input, such as presence or absence of people, human movement, landscape scenes, human faces, etc. Moreover, each semantic component mapped onto a distinct functional cortical network involving high-level cognitive regions in occipitotemporal, frontal and parietal cortices. The present work demonstrates the potential of the data-driven methods from information processing fields to explain patterns of cortical responses, and contributes to the overall discussion about the encoding of high-level perceptual information in the human brain

Characterization of High-Gamma Activity in Electrocorticographic Signals

IntroductionElectrocorticographic (ECoG) high-gamma activity (HGA) is a widely recognized and robust neural correlate of cognition and behavior. However, fundamental signal properties of HGA, such as the high-gamma frequency band or temporal dynamics of HGA, have never been systematically characterized. As a result, HGA estimators are often poorly adjusted, such that they miss valuable physiological information.MethodsTo address these issues, we conducted a thorough qualitative and quantitative characterization of HGA in ECoG signals. Our study is based on ECoG signals recorded from 18 epilepsy patients while performing motor control, listening, and visual perception tasks. In this study, we first categorize HGA into HGA types based on the cognitive/behavioral task. For each HGA type, we then systematically quantify three fundamental signal properties of HGA: the high-gamma frequency band, the HGA bandwidth, and the temporal dynamics of HGA.ResultsThe high-gamma frequency band strongly varies across subjects and across cognitive/behavioral tasks. In addition, HGA time courses have lowpass character, with transients limited to 10 Hz. The task-related rise time and duration of these HGA time courses depend on the individual subject and cognitive/behavioral task. Task-related HGA amplitudes are comparable across the investigated tasks.DiscussionThis study is of high practical relevance because it provides a systematic basis for optimizing experiment design, ECoG acquisition and processing, and HGA estimation. Our results reveal previously unknown characteristics of HGA, the physiological principles of which need to be investigated in further studies

When the Whole Is Less Than the Sum of Its Parts: Maximum Object Category Information and Behavioral Prediction in Multiscale Activation Patterns.

Neural codes are reflected in complex neural activation patterns. Conventional electroencephalography (EEG) decoding analyses summarize activations by averaging/down-sampling signals within the analysis window. This diminishes informative fine-grained patterns. While previous studies have proposed distinct statistical features capable of capturing variability-dependent neural codes, it has been suggested that the brain could use a combination of encoding protocols not reflected in any one mathematical feature alone. To check, we combined 30 features using state-of-the-art supervised and unsupervised feature selection procedures (n = 17). Across three datasets, we compared decoding of visual object category between these 17 sets of combined features, and between combined and individual features. Object category could be robustly decoded using the combined features from all of the 17 algorithms. However, the combination of features, which were equalized in dimension to the individual features, were outperformed across most of the time points by the multiscale feature of Wavelet coefficients. Moreover, the Wavelet coefficients also explained the behavioral performance more accurately than the combined features. These results suggest that a single but multiscale encoding protocol may capture the EEG neural codes better than any combination of protocols. Our findings put new constraints on the models of neural information encoding in EEG

Neural correlates of long-term memory: the interplay between encoding and retrieval

Neural correlates of human long-term memory encoding and retrieval have been studied in relative isolation. Memory performance, however, benefits from an overlap between processes engaged at encoding and retrieval. This thesis sought to determine how encoding-retrieval overlap affects neural correlates of memory. Four studies were conducted using electrical brain activity recorded from the scalps of healthy adults. The first experiment addressed whether congruency in mode of presentation across study and test (word or picture) influences encoding-related activity. The findings showed that this was indeed the case, but only for pictures. When a picture was probed with a picture, activity over anterior scalp sites predicted encoding success. When a picture was probed with a word, encoding-related activity was instead maximal over posterior sites. The remaining experiments determined whether the amount of perceptual information contained within a picture affects memory-related activity. Study-test congruency primarily affected encoding-related activity, irrespective of the amount of information that had to be encoded. Retrieval-related activity, by contrast, was instead sensitive to the amount of information contained within a retrieval probe. Analyses in the frequency domain suggested that memory retrieval relies on the reinstatement of neural activity across study and test. Oscillatory power in the theta frequency band (4-8 Hz) over frontal scalp sites at both encoding and retrieval was specific to the encoded amount of pictorial information. Together, the data demonstrate that neural correlates of long-term memory are sensitive to the similarity between processes engaged at study and test. However, study-test overlap may not be a universal organizing principle. Effects of study-test congruency depend on the type of information contained within a study event and test probe, and the type of neural activity that is considered