A Novel Synergistic Model Fusing Electroencephalography and Functional Magnetic Resonance Imaging for Modeling Brain Activities

Study of the human brain is an important and very active area of research. Unraveling the way the human brain works would allow us to better understand, predict and prevent brain related diseases that affect a significant part of the population. Studying the brain response to certain input stimuli can help us determine the involved brain areas and understand the mechanisms that characterize behavioral and psychological traits. In this research work two methods used for the monitoring of brain activities, Electroencephalography (EEG) and functional Magnetic Resonance (fMRI) have been studied for their fusion, in an attempt to bridge together the advantages of each one. In particular, this work has focused in the analysis of a specific type of EEG and fMRI recordings that are related to certain events and capture the brain response under specific experimental conditions. Using spatial features of the EEG we can describe the temporal evolution of the electrical field recorded in the scalp of the head. This work introduces the use of Hidden Markov Models (HMM) for modeling the EEG dynamics. This novel approach is applied for the discrimination of normal and progressive Mild Cognitive Impairment patients with significant results. EEG alone is not able to provide the spatial localization needed to uncover and understand the neural mechanisms and processes of the human brain. Functional Magnetic Resonance imaging (fMRI) provides the means of localizing functional activity, without though, providing the timing details of these activations. Although, at first glance it is apparent that the strengths of these two modalities, EEG and fMRI, complement each other, the fusion of information provided from each one is a challenging task. A novel methodology for fusing EEG spatiotemporal features and fMRI features, based on Canonical Partial Least Squares (CPLS) is presented in this work. A HMM modeling approach is used in order to derive a novel feature-based representation of the EEG signal that characterizes the topographic information of the EEG. We use the HMM model in order to project the EEG data in the Fisher score space and use the Fisher score to describe the dynamics of the EEG topography sequence. The correspondence between this new feature and the fMRI is studied using CPLS. This methodology is applied for extracting features for the classification of a visual task. The results indicate that the proposed methodology is able to capture task related activations that can be used for the classification of mental tasks. Extensions on the proposed models are examined along with future research directions and applications

Test-retest reliability of the magnetic mismatch negativity response to sound duration and omission deviants

Mismatch negativity (MMN) is a neurophysiological measure of auditory novelty detection that could serve as a translational biomarker of psychiatric disorders, such as schizophrenia. However, the replicability of its magnetoencephalographic (MEG) counterpart (MMNm) has been insufficiently addressed. In the current study, test-retest reliability of the MMNm response to both duration and omission deviants was evaluated over two MEG sessions in 16 healthy adults. MMNm amplitudes and latencies were obtained at both sensor- and source-level using a cortically-constrained minimum-norm approach. Intraclass correlations (ICC) were derived to assess stability of MEG responses over time. In addition, signal-to-noise ratios (SNR) and within-subject statistics were obtained in order to determine MMNm detectability in individual participants. ICC revealed robust values at both sensor- and source-level for both duration and omission MMNm amplitudes (ICC = 0.81-0.90), in particular in the right hemisphere, while moderate to strong values were obtained for duration MMNm and omission MMNm peak latencies (ICC = 0.74-0.88). Duration MMNm was robustly identified in individual participants with high SNR, whereas omission MMNm responses were only observed in half of the participants. Our data indicate that MMNm to unexpected duration changes and omitted sounds are highly reproducible, providing support for the use of MEG-parameters in basic and clinical research

Neural surprise in somatosensory Bayesian learning

Tracking statistical regularities of the environment is important for shaping human behavior and perception. Evidence suggests that the brain learns environmental dependencies using Bayesian principles. However, much remains unknown about the employed algorithms, for somesthesis in particular. Here, we describe the cortical dynamics of the somatosensory learning system to investigate both the form of the generative model as well as its neural surprise signatures. Specifically, we recorded EEG data from 40 participants subjected to a somatosensory roving-stimulus paradigm and performed single-trial modeling across peri-stimulus time in both sensor and source space. Our Bayesian model selection procedure indicates that evoked potentials are best described by a non-hierarchical learning model that tracks transitions between observations using leaky integration. From around 70ms post-stimulus onset, secondary somatosensory cortices are found to represent confidence-corrected surprise as a measure of model inadequacy. Indications of Bayesian surprise encoding, reflecting model updating, are found in primary somatosensory cortex from around 140ms. This dissociation is compatible with the idea that early surprise signals may control subsequent model update rates. In sum, our findings support the hypothesis that early somatosensory processing reflects Bayesian perceptual learning and contribute to an understanding of its underlying mechanisms

Modeling Brain Resonance Phenomena Using a Neural Mass Model

Stimulation with rhythmic light flicker (photic driving) plays an important role in the diagnosis of schizophrenia, mood disorder, migraine, and epilepsy. In particular, the adjustment of spontaneous brain rhythms to the stimulus frequency (entrainment) is used to assess the functional flexibility of the brain. We aim to gain deeper understanding of the mechanisms underlying this technique and to predict the effects of stimulus frequency and intensity. For this purpose, a modified Jansen and Rit neural mass model (NMM) of a cortical circuit is used. This mean field model has been designed to strike a balance between mathematical simplicity and biological plausibility. We reproduced the entrainment phenomenon observed in EEG during a photic driving experiment. More generally, we demonstrate that such a single area model can already yield very complex dynamics, including chaos, for biologically plausible parameter ranges. We chart the entire parameter space by means of characteristic Lyapunov spectra and Kaplan-Yorke dimension as well as time series and power spectra. Rhythmic and chaotic brain states were found virtually next to each other, such that small parameter changes can give rise to switching from one to another. Strikingly, this characteristic pattern of unpredictability generated by the model was matched to the experimental data with reasonable accuracy. These findings confirm that the NMM is a useful model of brain dynamics during photic driving. In this context, it can be used to study the mechanisms of, for example, perception and epileptic seizure generation. In particular, it enabled us to make predictions regarding the stimulus amplitude in further experiments for improving the entrainment effect

Supported diagnosis of adhd from eeg signals based on hidden markov models and probability product kernels

Attention deficit hyperactivity disorder (ADHD), most often present in childhood, may persist in adult life, hampering personal development. However, ADHD diagnosis is a real challenge since it highly depends on the clinical observation of the patient, the parental and scholar information, and the specialist expertise. Despite demanded objective diagnosis aids from biosignals, the physiological biomarkers lack robustness and significance under the non-stationary and non-linear electroencephalographic dynamics. Therefore, this work presents a supported diagnosis methodology for ADHD from the dynamic characterization of EEG based on hidden Markov models (HMM) and probability product kernels (PPK). Based on the symptom of impulsivity, the proposed approach trains an HMM for each subject from EEG signals in failed inhibition tasks. In the first instance, PPK measures the similarity between subjects through the inner product between their trained HMMs. Then, given the computational costs, fast computation of PPK for HMM facilitates parameter tuning of kernel similarity. Finally, the Kernel Principal Component Analysis (KPCA) projects the PPK to a lower dimensional space, allowing the interpretability of the results. Thus, a support vector machine supports the diagnosis of ADHD as a classification task using PPK as the inner product operator. The methodology compared classification results on EEG signals with all channels, channels of interest (COI), and analysis in the Theta, Alpha, and Beta frequency bands. The results show an accuracy rate of 97.0% in the Beta band in COI, which supports the assumption that this frequency rhythm may be correlated to differences between ADHD and controls regarding attentional allocation during the execution of the cognitive task.El trastorno por déficit de atención e hiperactividad (TDAH), que suele presentarse en la infancia, puede persistir en la vida adulta, obstaculizando el desarrollo personal. Sin embargo, el diagnóstico del TDAH es un verdadero reto, ya que depende en gran medida de la observación clínica del paciente, de la información de los padres y de los estudiosos, y de la experiencia de los especialistas. A pesar de la demanda de ayudas para el diagnóstico objetivo a partir de bioseñales, los biomarcadores fisiológicos carecen de robustez y significación bajo la dinámica electroencefalográfica no estacionaria y no lineal. Por lo tanto, este trabajo presenta una metodología de diagnóstico apoyada para el TDAH a partir de la caracterización dinámica del EEG basada en modelos ocultos de Markov (HMM) y productos de kernel de probabilidad (PPK). Basándose en el síntoma de impulsividad, el enfoque propuesto entrena un HMM para cada sujeto a partir de las señales del EEG en tareas de inhibición fallidas. En primer lugar, el PPK mide la similitud entre los sujetos a través del producto interno entre sus HMMs entrenados. Luego, dados los costes computacionales, el cálculo rápido de PPK para los HMM facilita el ajuste de los parámetros de similitud del kernel. Por último, el Análisis de Componentes Principales del Kernel (KPCA) proyecta el PPK a un espacio de menor dimensión, lo que permite la interpretabilidad de los resultados. Así, una máquina de vectores de apoyo apoya el diagnóstico del TDAH como una tarea de clasificación utilizando el PPK como operador de producto interno. La metodología comparó los resultados de clasificación en señales de EEG con todos los canales, canales de interés (COI), y análisis en las bandas de frecuencia Theta, Alpha, y Beta. Los resultados muestran una tasa de precisión del 97,0% en la banda Beta en COI, lo que apoya la suposición de que este ritmo de frecuencia puede estar correlacionado con las diferencias entre el TDAH y los controles en cuanto a la asignación atencional durante la ejecución de la tarea cognitiva.MaestríaMagíster en Ingeniería EléctricaContents 1 List of Symbols and Abbreviations 5 1.1 Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2 Abbrevations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Introduction 7 2.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Justification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 State of the art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.4 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4.1 General objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4.2 Specific objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3 Develop a multichannel time series classification methodology taking into account signal dynamics 13 3.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.1.1 Similarity between time series . . . . . . . . . . . . . . . . . . . . . . 13 3.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2.1 EEG Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2.2 HMM training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.3 Parameter tuning and Classification . . . . . . . . . . . . . . . . . . . 17 3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 4 Develop a time series classification methodology that takes into account spectral information and reduces the computational cost of training. 21 4.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.1.1 Fast computation of PPK for HMM . . . . . . . . . . . . . . . . . . . 22 4.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.2.1 Synthetic Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.2.2 Training and Parameter tuning and classification . . . . . . . . . . . 24 4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1 CONTENTS 4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 5 Develop a methodology for visualizing stochastic representations to facilitate the interpretability of inference machines 32 5.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5.1.1 Model interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5.1.2 Low-dimensional HMM visualization . . . . . . . . . . . . . . . . . . 33 5.1.3 Low-dimensional state visualization . . . . . . . . . . . . . . . . . . 34 5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 6 Conclusions 4

Methods for Analyzing Multi-Subject Resting-State Neuroimaging Time Series Data

University of Minnesota Ph.D. dissertation. May 2019. Major: Biostatistics. Advisors: Mark Fiecas, Lynn Eberly. 1 computer file (PDF); xv, 107 pages.Resting-state neuroimaging modalities such as electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) collect data in the form of time series which represent the activity in the brain at rest. This resting-state behavior can be analyzed in different ways to address different research questions and is thought to represent the intrinsic activity of the brain. We discuss three potential avenues of analysis. First, we propose a permutation-based method which tests the longitudinal functional connectivity of fMRI data collected from cognitively normal participants and Alzheimer’s patients. Next, we propose a Bayesian nonparametric model to jointly perform spectral time series analysis on EEG data from 1,116 twins from the Minnesota Twin Family Study (MTFS) and discuss a novel heritability estimator for features of the estimated spectral density curves. Finally, we propose another Bayesian nonparametric model to perform EEG microstate analysis of the MTFS data at the twin pair level. Each method discussed views the resting-state time series data from a different angle. Additionally, in each of these scenarios, we jointly analyze data collected from many different participants while accounting for the design of the study in which the data was collected. Regardless of the analysis method chosen, accounting for the within and between-participant dependence structure yields improved results

Mismatch responses: Probing probabilistic inference in the brain

Sensory signals are governed by statistical regularities and carry valuable information about the unfolding of environmental events. The brain is thought to capitalize on the probabilistic nature of sequential inputs to infer on the underlying (hidden) dynamics driving sensory stimulation. Mis-match responses (MMRs) such as the mismatch negativity (MMN) and the P3 constitute prominent neuronal signatures which are increasingly interpreted as reflecting a mismatch between the current sensory input and the brain’s generative model of incoming stimuli. As such, MMRs might be viewed as signatures of probabilistic inference in the brain and their response dynamics can provide insights into the underlying computational principles. However, given the dominance of the auditory modality in MMR research, the specifics of brain responses to probabilistic sequences across sensory modalities and especially in the somatosensory domain are not well characterized. The work presented here investigates MMRs across the auditory, visual and somatosensory modality by means of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI). We designed probabilistic stimulus sequences to elicit and characterize MMRs and employed computational modeling of response dynamics to inspect different aspects of the brain’s generative model of the sensory environment. In the first study, we used a volatile roving stimulus paradigm to elicit somatosensory MMRs and performed single-trial modeling of EEG signals in sensor and source space. Model comparison suggested that responses reflect Bayesian inference based on the estimation of transition probability and limited information integration of the recent past in order to adapt to a changing environment. The results indicated that somatosensory MMRs reflect an initial mismatch between sensory input and model beliefs represented by confidence-corrected surprise (CS) followed by model adjustment dynamics represented by Bayesian surprise (BS). For the second and third study we designed a tri-modal roving stimulus paradigm to delineate modality specific and modality general features of mismatch processing. Computational modeling of EEG signals in study 2 suggested that single-trial dynamics reflect Bayesian inference based on estimation of uni-modal transition probabilities as well as cross-modal conditional dependencies. While early mismatch processing around the MMN tended to reflect CS, later MMRs around the P3 rather reflect BS, in correspondence to the somatosensory study. Finally, the fMRI results of study 3 showed that MMRs are generated by an interaction of modality specific regions in higher order sensory cortices and a modality general fronto-parietal network. Inferior parietal regions in particular were sensitive to expectation violations with respect to the cross-modal contingencies in the stimulus sequences. Overall, our results indicate that MMRs across the senses reflect processes of probabilistic inference in a complex and inherently multi-modal environment.Sensorische Signale sind durch statistische Regularitäten bestimmt und beinhalten wertvolle Informationen über die Entwicklung von Umweltereignissen. Es wird angenommen, dass das Gehirn die Wahrscheinlichkeitseigenschaften sequenzieller Reize nutzt um auf die zugrundeliegenden (verborgenen) Dynamiken zu schließen, welche sensorische Stimulation verursachen. Diskrepanz-Reaktionen ("Mismatch responses"; MMRs) wie die "mismatch negativity" (MMN) und die P3 sind bekannte neuronale Signaturen die vermehrt als Signale einer Diskrepanz zwischen der momentanen sensorischen Einspeisung und dem generativen Modell, welches das Gehirn von den eingehenden Reizen erstellt angesehen werden. Als solche können MMRs als Signaturen von wahrscheinlichkeitsbasierter Inferenz im Gehirn betrachtet werden und ihre Reaktionsdynamiken können Einblicke in die zugrundeliegenden komputationalen Prinzipien geben. Angesichts der Dominanz der auditorischen Modalität in der MMR-Forschung, sind allerdings die spezifischen Eigenschaften von Hirn-Reaktionen auf Wahrscheinlichkeitssequenzen über sensorische Modalitäten hinweg und vor allem in der somatosensorischen Modalität nicht gut charakterisiert. Die hier vorgestellte Arbeit untersucht MMRs über die auditorische, visuelle und somatosensorische Modalität hinweg anhand von Elektroenzephalographie (EEG) und funktioneller Magnetresonanztomographie (fMRT). Wir gestalteten wahrscheinlichkeitsbasierte Reizsequenzen, um MMRs auszulösen und zu charakterisieren und verwendeten komputationale Modellierung der Reaktionsdynamiken, um verschiedene Aspekte des generativen Modells des Gehirns von der sensorischen Umwelt zu untersuchen. In der ersten Studie verwendeten wir ein volatiles "Roving-Stimulus"-Paradigma, um somatosensorische MMRs auszulösen und modellierten die Einzel-Proben der EEG-Signale im sensorischen und Quell-Raum. Modellvergleiche legten nahe, dass die Reaktionen Bayes’sche Inferenz abbilden, basierend auf der Schätzung von Transitionswahrscheinlichkeiten und limitierter Integration von Information der jüngsten Vergangenheit, welche eine Anpassung an Umweltänderungen ermöglicht. Die Ergebnisse legen nahe, dass somatosen-sorische MMRs eine initiale Diskrepanz zwischen sensorischer Einspeisung und Modellüberzeugung reflektieren welche durch "confidence-corrected surprise" (CS) repräsentiert ist, gefolgt von Modelanpassungsdynamiken repräsentiert von "Bayesian surprise" (BS). Für die zweite und dritte Studie haben wir ein Tri-Modales "Roving-Stimulus"-Paradigma gestaltet, um modalitätsspezifische und modalitätsübergreifende Eigenschaften von Diskrepanzprozessierung zu umreißen. Komputationale Modellierung von EEG-Signalen in Studie 2 legte nahe, dass Einzel-Proben Dynamiken Bayes’sche Inferenz abbilden, basierend auf der Schätzung von unimodalen Transitionswahrscheinlichkeiten sowie modalitätsübergreifenden bedingten Abhängigkeiten. Während frühe Diskrepanzprozessierung um die MMN dazu tendierten CS zu reflektieren, so reflektierten spätere MMRs um die P3 eher BS, in Übereinstimmung mit der somatosensorischen Studie. Abschließend zeigten die fMRT-Ergebnisse der Studie 3 dass MMRs durch eine Interaktion von modalitätsspezifischen Regionen in sensorischen Kortizes höherer Ordnung mit einem modalitätsübergreifenden fronto-parietalen Netzwerk generiert werden. Inferior parietale Regionen im Speziellen waren sensitiv gegenüber Erwartungsverstoß in Bezug auf die modalitätsübergreifenden Wahrscheinlichkeiten in den Reizsequenzen. Insgesamt weisen unsere Ergebnisse darauf hin, dass MMRs über die Sinne hinweg Prozesse von wahrscheinlichkeitsbasierter Inferenz in einer komplexen und inhärent multi-modalen Umwelt darstellen