Strength is in numbers: Can concordant artificial listeners improve prediction of emotion from speech?

Humans can communicate their emotions by modulating facial expressions or the tone of their voice. Albeit numerous applications exist that enable machines to read facial emotions and recognize the content of verbal messages, methods for speech emotion recognition are still in their infancy. Yet, fast and reliable applications for emotion recognition are the obvious advancement of present 'intelligent personal assistants', and may have countless applications in diagnostics, rehabilitation and research. Taking inspiration from the dynamics of human group decision-making, we devised a novel speech emotion recognition system that applies, for the first time, a semi-supervised prediction model based on consensus. Three tests were carried out to compare this algorithm with traditional approaches. Labeling performances relative to a public database of spontaneous speeches are reported. The novel system appears to be fast, robust and less computationally demanding than traditional methods, allowing for easier implementation in portable voice-analyzers (as used in rehabilitation, research, industry, etc.) and for applications in the research domain (such as real-time pairing of stimuli to participants' emotional state, selective/differential data collection based on emotional content, etc.)

Automatic speaker recognition: modelling, feature extraction and effects of clinical environment

Speaker recognition is the task of establishing identity of an individual based on his/her voice. It has a significant potential as a convenient biometric method for telephony applications and does not require sophisticated or dedicated hardware. The Speaker Recognition task is typically achieved by two-stage signal processing: training and testing. The training process calculates speaker-specific feature parameters from the speech. The features are used to generate statistical models of different speakers. In the testing phase, speech samples from unknown speakers are compared with the models and classified. Current state of the art speaker recognition systems use the Gaussian mixture model (GMM) technique in combination with the Expectation Maximization (EM) algorithm to build the speaker models. The most frequently used features are the Mel Frequency Cepstral Coefficients (MFCC). This thesis investigated areas of possible improvements in the field of speaker recognition. The identified drawbacks of the current speaker recognition systems included: slow convergence rates of the modelling techniques and feature’s sensitivity to changes due aging of speakers, use of alcohol and drugs, changing health conditions and mental state. The thesis proposed a new method of deriving the Gaussian mixture model (GMM) parameters called the EM-ITVQ algorithm. The EM-ITVQ showed a significant improvement of the equal error rates and higher convergence rates when compared to the classical GMM based on the expectation maximization (EM) method. It was demonstrated that features based on the nonlinear model of speech production (TEO based features) provided better performance compare to the conventional MFCCs features. For the first time the effect of clinical depression on the speaker verification rates was tested. It was demonstrated that the speaker verification results deteriorate if the speakers are clinically depressed. The deterioration process was demonstrated using conventional (MFCC) features. The thesis also showed that when replacing the MFCC features with features based on the nonlinear model of speech production (TEO based features), the detrimental effect of the clinical depression on speaker verification rates can be reduced

Multi-stream Longitudinal Data Analysis using Deep Learning

Longitudinal healthcare data encompasses all tasks where patients information are collected at multiple follow-up times. Analyzing this data is critical in addressing many real world problems in healthcare such as disease prediction and prevention. In this thesis, technical challenges in analyzing longitudinal administrative claims data are addressed and novel deep learning based models are proposed for multi-stream data analysis and disease prediction tasks. These algorithms and frameworks are assessed mainly on substance use disorders prediction tasks and specifically designed to tackled these disorders. Substance use disorder is a public health crisis costing the US an estimated $740 billion annually in healthcare, lost workplace productivity, and crime. Early identification and engagement of individuals at risk of developing a substance use disorder is a critical unmet need in healthcare which can be achieved by producing automatic artificial intelligence based tools trained using big healthcare data. In fact, healthcare data can be harnessed together with artificial intelligence and machine learning to advance our understanding of factors that increase the propensity for developing different diseases as well as those that aid in the treatment of these disorders. Here in, a disease prediction framework is first proposed based on recurrent neural networks. This framework includes three components: 1) data pre-processing, 2) disease prediction using long short term memory models, and 3) hypothesis exploration by varying the models and the inputs. This framework is assessed using two use cases: substance use disorder prediction and mild cognitive impairment prediction. Experimental results show that this proposed model can efficiently analyze patients\u27 data and creates efficient disease prediction tools. Second, the limitationsof current deep learning models including long short term memory models in claimsdata analysis are detected and addressed, and a novel model based on the transformer models is proposed. In fact, leveraging the real-world longitudinal claims data, a novel multi-stream transformer model is proposed for predicting opioid use disorder as an important case of substance use disorders. This model is designed to simultaneously analyze multiple types of data streams, such as medications, diagnoses, procedures and demographics, by attending to segments within and across these data streams. The proposed model tested on the IBM MarketScan data showed significantly better performance than the traditional models and recently developed deep learning models

Knowledge Modelling and Learning through Cognitive Networks

One of the most promising developments in modelling knowledge is cognitive network science, which aims to investigate cognitive phenomena driven by the networked, associative organization of knowledge. For example, investigating the structure of semantic memory via semantic networks has illuminated how memory recall patterns influence phenomena such as creativity, memory search, learning, and more generally, knowledge acquisition, exploration, and exploitation. In parallel, neural network models for artificial intelligence (AI) are also becoming more widespread as inferential models for understanding which features drive language-related phenomena such as meaning reconstruction, stance detection, and emotional profiling. Whereas cognitive networks map explicitly which entities engage in associative relationships, neural networks perform an implicit mapping of correlations in cognitive data as weights, obtained after training over labelled data and whose interpretation is not immediately evident to the experimenter. This book aims to bring together quantitative, innovative research that focuses on modelling knowledge through cognitive and neural networks to gain insight into mechanisms driving cognitive processes related to knowledge structuring, exploration, and learning. The book comprises a variety of publication types, including reviews and theoretical papers, empirical research, computational modelling, and big data analysis. All papers here share a commonality: they demonstrate how the application of network science and AI can extend and broaden cognitive science in ways that traditional approaches cannot

ELUCIDATING THE NATURE AND DEVELOPMENT OF NEURAL MECHANISMS ASSOCIATED WITH ANXIOUS APPREHENSION AND ANXIOUS AROUSAL ACROSS ADOLESCENCE

Work on adult anxiety has found that anxious apprehension, marked by chronic worry, and anxious arousal, marked by elevated sympathetic hyperarousal, are instantiated in different neurobiological systems and involve different information processing dysfunctions. However, little is known regarding how these transdiagnostic types of anxiety develop. The present dissertation seeks to apply this transdiagnostic approach to anxiety to the study of adolescent neurodevelopment. Study 1 established that anxious arousal and anxious apprehension are distinguishable via self-report, supporting that these traits are meaningfully different as early as 11 years old. Neurobiologically, anxious arousal positively correlated with dmPFC-amygdala structural connectivity, interpreted as an elevated propensity to amplify anxiety responses, whereas anxious apprehension positively correlated with right iFG structural connectivity, interpreted as reflecting elevated inhibition of immediate threat processing. Evidence was not found for neural correlates of anxiolytic dysfunction in anxious arousal, nor for neural correlates of increased internal mental rehearsal in anxious apprehension. Study 2 built on Study 1 by examining how intrinsic connectivity was related to types of anxiety both cross-sectionally and longitudinally. Study 2 found no evidence that a priori defined functional pathways mapped onto types of anxiety. In contrast, a data-driven approach revealed that functional amygdala connectivity can predict variation in anxious arousal at both waves, whereas functional iFG connectivity can predict variation in anxious apprehension in wave 2. Taken together, the present dissertation establishes that anxious arousal and anxious apprehension emerge in early adolescence, and may be marked by different kinds of information processing dysfunctions. Future work needs to more rigorously test if inferences about information processing associated with neural correlates found here are valid.Doctor of Philosoph

EEG based Major Depressive disorder and Bipolar disorder detection using Neural Networks: A review

Mental disorders represent critical public health challenges as they are leading contributors to the global burden of disease and intensely influence social and financial welfare of individuals. The present comprehensive review concentrate on the two mental disorders: Major depressive Disorder (MDD) and Bipolar Disorder (BD) with noteworthy publications during the last ten years. There is a big need nowadays for phenotypic characterization of psychiatric disorders with biomarkers. Electroencephalography (EEG) signals could offer a rich signature for MDD and BD and then they could improve understanding of pathophysiological mechanisms underling these mental disorders. In this review, we focus on the literature works adopting neural networks fed by EEG signals. Among those studies using EEG and neural networks, we have discussed a variety of EEG based protocols, biomarkers and public datasets for depression and bipolar disorder detection. We conclude with a discussion and valuable recommendations that will help to improve the reliability of developed models and for more accurate and more deterministic computational intelligence based systems in psychiatry. This review will prove to be a structured and valuable initial point for the researchers working on depression and bipolar disorders recognition by using EEG signals.Comment: 29 pages,2 figures and 18 Table

Recent Applications in Graph Theory

Graph theory, being a rigorously investigated field of combinatorial mathematics, is adopted by a wide variety of disciplines addressing a plethora of real-world applications. Advances in graph algorithms and software implementations have made graph theory accessible to a larger community of interest. Ever-increasing interest in machine learning and model deployments for network data demands a coherent selection of topics rewarding a fresh, up-to-date summary of the theory and fruitful applications to probe further. This volume is a small yet unique contribution to graph theory applications and modeling with graphs. The subjects discussed include information hiding using graphs, dynamic graph-based systems to model and control cyber-physical systems, graph reconstruction, average distance neighborhood graphs, and pure and mixed-integer linear programming formulations to cluster networks

Brain-Computer Interface

Brain-computer interfacing (BCI) with the use of advanced artificial intelligence identification is a rapidly growing new technology that allows a silently commanding brain to manipulate devices ranging from smartphones to advanced articulated robotic arms when physical control is not possible. BCI can be viewed as a collaboration between the brain and a device via the direct passage of electrical signals from neurons to an external system. The book provides a comprehensive summary of conventional and novel methods for processing brain signals. The chapters cover a range of topics including noninvasive and invasive signal acquisition, signal processing methods, deep learning approaches, and implementation of BCI in experimental problems

SEARCHING NEUROIMAGING BIOMARKERS IN MENTAL DISORDERS WITH GRAPH AND MULTIMODAL FUSION ANALYSIS OF FUNCTIONAL CONNECTIVITY

Mental disorders such as schizophrenia (SZ), bipolar (BD), and major depression disorders (MDD) can cause severe symptoms and life disruption. They share some symptoms, which can pose a major clinical challenge to their differentiation. Objective biomarkers based on neuroimaging may help to improve diagnostic accuracy and facilitate optimal treatment for patients. Over the last decades, non-invasive in-vivo neuroimaging techniques such as magnetic resonance imaging (MRI) have been increasingly applied to measure structure and function in human brains. With functional MRI (fMRI) or structural MRI (sMRI), studies have identified neurophysiological deficits in patients’ brain from different perspective. Functional connectivity (FC) analysis is an approach that measures functional integration in brains. By assessing the temporal coherence of the hemodynamic activity among brain regions, FC is considered capable of characterizing the large-scale integrity of neural activity. In this work, we present two data analysis frameworks for biomarker detection on brain imaging with FC, 1) graph analysis of FC and 2) multimodal fusion analysis, to better understand the human brain. Graph analysis reveals the interaction among brain regions based on graph theory, while the multimodal fusion framework enables us to utilize the strength of different imaging modalities through joint analysis. Four applications related to FC using these frameworks were developed. First, FC was estimated using a model-based approach, and revealed altered the small-world network structure in SZ. Secondly, we applied graph analysis on functional network connectivity (FNC) to differentiate BD and MDD during resting-state. Thirdly, two functional measures, FNC and fractional amplitude of low frequency fluctuations (fALFF), were spatially overlaid to compare the FC and spatial alterations in SZ. And finally, we utilized a multimodal fusion analysis framework, multi-set canonical correlation analysis + joint independent component analysis (mCCA+jICA) to link functional and structural abnormalities in BD and MDD. We also evaluated the accuracy of predictive diagnosis through classifiers generated on the selected features. In summary, via the two frameworks, our work has made several contributions to advance FC analysis, which improves our understanding of underlying brain function and structure, and our findings may be ultimately useful for the development of biomarkers of mental disease