Brain Music : Sistema generativo para la creación de música simbólica a partir de respuestas neuronales afectivas

gráficas, tablasEsta tesis de maestría presenta una metodología de aprendizaje profundo multimodal innovadora que fusiona un modelo de clasificación de emociones con un generador musical, con el propósito de crear música a partir de señales de electroencefalografía, profundizando así en la interconexión entre emociones y música. Los resultados alcanzan tres objetivos específicos: Primero, ya que el rendimiento de los sistemas interfaz cerebro-computadora varía considerablemente entre diferentes sujetos, se introduce un enfoque basado en la transferencia de conocimiento entre sujetos para mejorar el rendimiento de individuos con dificultades en sistemas de interfaz cerebro-computadora basados en el paradigma de imaginación motora. Este enfoque combina datos de EEG etiquetados con datos estructurados, como cuestionarios psicológicos, mediante un método de "Kernel Matching CKA". Utilizamos una red neuronal profunda (Deep&Wide) para la clasificación de la imaginación motora. Los resultados destacan su potencial para mejorar las habilidades motoras en interfaces cerebro-computadora. Segundo, proponemos una técnica innovadora llamada "Labeled Correlation Alignment"(LCA) para sonificar respuestas neurales a estímulos representados en datos no estructurados, como música afectiva. Esto genera características musicales basadas en la actividad cerebral inducida por las emociones. LCA aborda la variabilidad entre sujetos y dentro de sujetos mediante el análisis de correlación, lo que permite la creación de envolventes acústicos y la distinción entre diferente información sonora. Esto convierte a LCA en una herramienta prometedora para interpretar la actividad neuronal y su reacción a estímulos auditivos. Finalmente, en otro capítulo, desarrollamos una metodología de aprendizaje profundo de extremo a extremo para generar contenido musical MIDI (datos simbólicos) a partir de señales de actividad cerebral inducidas por música con etiquetas afectivas. Esta metodología abarca el preprocesamiento de datos, el entrenamiento de modelos de extracción de características y un proceso de emparejamiento de características mediante Deep Centered Kernel Alignment, lo que permite la generación de música a partir de señales EEG. En conjunto, estos logros representan avances significativos en la comprensión de la relación entre emociones y música, así como en la aplicación de la inteligencia artificial en la generación musical a partir de señales cerebrales. Ofrecen nuevas perspectivas y herramientas para la creación musical y la investigación en neurociencia emocional. Para llevar a cabo nuestros experimentos, utilizamos bases de datos públicas como GigaScience, Affective Music Listening y Deap Dataset (Texto tomado de la fuente)This master’s thesis presents an innovative multimodal deep learning methodology that combines an emotion classification model with a music generator, aimed at creating music from electroencephalography (EEG) signals, thus delving into the interplay between emotions and music. The results achieve three specific objectives: First, since the performance of brain-computer interface systems varies significantly among different subjects, an approach based on knowledge transfer among subjects is introduced to enhance the performance of individuals facing challenges in motor imagery-based brain-computer interface systems. This approach combines labeled EEG data with structured information, such as psychological questionnaires, through a "Kernel Matching CKA"method. We employ a deep neural network (Deep&Wide) for motor imagery classification. The results underscore its potential to enhance motor skills in brain-computer interfaces. Second, we propose an innovative technique called "Labeled Correlation Alignment"(LCA) to sonify neural responses to stimuli represented in unstructured data, such as affective music. This generates musical features based on emotion-induced brain activity. LCA addresses variability among subjects and within subjects through correlation analysis, enabling the creation of acoustic envelopes and the distinction of different sound information. This makes LCA a promising tool for interpreting neural activity and its response to auditory stimuli. Finally, in another chapter, we develop an end-to-end deep learning methodology for generating MIDI music content (symbolic data) from EEG signals induced by affectively labeled music. This methodology encompasses data preprocessing, feature extraction model training, and a feature matching process using Deep Centered Kernel Alignment, enabling music generation from EEG signals. Together, these achievements represent significant advances in understanding the relationship between emotions and music, as well as in the application of artificial intelligence in musical generation from brain signals. They offer new perspectives and tools for musical creation and research in emotional neuroscience. To conduct our experiments, we utilized public databases such as GigaScience, Affective Music Listening and Deap DatasetMaestríaMagíster en Ingeniería - Automatización IndustrialInvestigación en Aprendizaje Profundo y señales BiológicasEléctrica, Electrónica, Automatización Y Telecomunicaciones.Sede Manizale

An affective computing and image retrieval approach to support diversified and emotion-aware reminiscence therapy sessions

A demência é uma das principais causas de dependência e incapacidade entre as pessoas idosas em todo o mundo. A terapia de reminiscência é uma terapia não farmacológica comummente utilizada nos cuidados com demência devido ao seu valor terapêutico para as pessoas com demência. Esta terapia é útil para criar uma comunicação envolvente entre pessoas com demência e o resto do mundo, utilizando as capacidades preservadas da memória a longo prazo, em vez de enfatizar as limitações existentes por forma a aliviar a experiência de fracasso e isolamento social. As soluções tecnológicas de assistência existentes melhoram a terapia de reminiscência ao proporcionar uma experiência mais envolvente para todos os participantes (pessoas com demência, familiares e clínicos), mas não estão livres de lacunas: a) os dados multimédia utilizados permanecem inalterados ao longo das sessões, e há uma falta de personalização para cada pessoa com demência; b) não têm em conta as emoções transmitidas pelos dados multimédia utilizados nem as reacções emocionais da pessoa com demência aos dados multimédia apresentados; c) a perspectiva dos cuidadores ainda não foi totalmente tida em consideração. Para superar estes desafios, seguimos uma abordagem de concepção centrada no utilizador através de inquéritos mundiais, entrevistas de seguimento, e grupos de discussão com cuidadores formais e informais para informar a concepção de soluções tecnológicas no âmbito dos cuidados de demência. Para cumprir com os requisitos identificados, propomos novos métodos que facilitam a inclusão de emoções no loop durante a terapia de reminiscência para personalizar e diversificar o conteúdo das sessões ao longo do tempo. As contribuições desta tese incluem: a) um conjunto de requisitos funcionais validados recolhidos com os cuidadores formais e informais, os resultados esperados com o cumprimento de cada requisito, e um modelo de arquitectura para o desenvolvimento de soluções tecnológicas de assistência para cuidados de demência; b) uma abordagem end-to-end para identificar automaticamente múltiplas informações emocionais transmitidas por imagens; c) uma abordagem para reduzir a quantidade de imagens que precisam ser anotadas pelas pessoas sem comprometer o desempenho dos modelos de reconhecimento; d) uma técnica de fusão tardia interpretável que combina dinamicamente múltiplos sistemas de recuperação de imagens com base em conteúdo para procurar eficazmente por imagens semelhantes para diversificar e personalizar o conjunto de imagens disponíveis para serem utilizadas nas sessões.Dementia is one of the major causes of dependency and disability among elderly subjects worldwide. Reminiscence therapy is an inexpensive non-pharmacological therapy commonly used within dementia care due to its therapeutic value for people with dementia. This therapy is useful to create engaging communication between people with dementia and the rest of the world by using the preserved abilities of long-term memory rather than emphasizing the existing impairments to alleviate the experience of failure and social isolation. Current assistive technological solutions improve reminiscence therapy by providing a more lively and engaging experience to all participants (people with dementia, family members, and clinicians), but they are not free of drawbacks: a) the multimedia data used remains unchanged throughout sessions, and there is a lack of customization for each person with dementia; b) they do not take into account the emotions conveyed by the multimedia data used nor the person with dementia’s emotional reactions to the multimedia presented; c) the caregivers’ perspective have not been fully taken into account yet. To overcome these challenges, we followed a usercentered design approach through worldwide surveys, follow-up interviews, and focus groups with formal and informal caregivers to inform the design of technological solutions within dementia care. To fulfil the requirements identified, we propose novel methods that facilitate the inclusion of emotions in the loop during reminiscence therapy to personalize and diversify the content of the sessions over time. Contributions from this thesis include: a) a set of validated functional requirements gathered from formal and informal caregivers, the expected outcomes with the fulfillment of each requirement, and an architecture’s template for the development of assistive technology solutions for dementia care; b) an end-to-end approach to automatically identify multiple emotional information conveyed by images; c) an approach to reduce the amount of images that need to be annotated by humans without compromising the recognition models’ performance; d) an interpretable late-fusion technique that dynamically combines multiple content-based image retrieval systems to effectively search for similar images to diversify and personalize the pool of images available to be used in sessions

Flexibly adapting to emotional cues: Examining the functional and structural correlates of emotional reactivity and emotion control in healthy and depressed individuals

The ability of emotionally significant stimuli to bias our behaviour is an evolutionarily adaptive phenomenon. However, sometimes emotions become excessive, inappropriate, and even pathological, like in major depressive disorder (MDD). Emotional flexibility includes both the neural processes involved in reacting to, or representing, emotional significance, and those involved in controlling emotional reactivity. MDD represents a potentially distinct form of emotion (in)flexibility, and therefore offers a unique perspective for understanding both the integration of conflicting emotional cues and the neural regions involved in actively controlling emotional systems. The present investigation of emotional flexibility began by considering the functional neural correlates of competing socio-emotional cues and effortful emotion regulation in MDD using both negative and positive emotions. Study 1 revealed greater amygdala activity in MDD relative to control participants when negative cues were centrally presented and task-relevant. No significant between-group differences were observed in the amygdala for peripheral task-irrelevant negative distracters. However, controls demonstrated greater recruitment of the ventrolateral (vlPFC) and dorsomedial prefrontal cortices (dmPFC) implicated in emotion control. Conversely, attenuated amygdala activity for task-relevant and irrelevant positive cues was observed in depressed participants. In Study 2, effortful emotion regulation using strategies adapted from cognitive behaviour therapy (CBT) revealed greater activity in regions of the dorsal and lateral prefrontal cortices in both MDD and control participants when attempting to either down-regulate negative or up-regulate positive emotions. During the down-regulation of negative cues, only controls displayed a significant reduction of amygdala activity. In Study 3, an individual differences approach using multiple regression revealed that while greater amygdala-vmPFC structural connectivity was associated with low trait-anxiety, greater connectivity between amygdala and regions of occipitotemporal and parietal cortices was associated with high trait-anxiety. These findings are discussed with respect to current models of emotional reactivity and emotion control derived from studies of both healthy individuals and those with emotional disorders, particularly depression. The focus is on amygdala variability in differing contexts, the role of the vmPFC in the modulation of amygdala activity via learning processes, and the modulation of emotion by attention or cognitive control mechanisms initiated by regions of frontoparietal cortices

The Constructivistly-Organised Dimensional-Appraisal (CODA) Model and Evidence for the Role of Goal-directed Processes in Emotional Episodes Induced by Music

The study of affective responses to music is a flourishing field. Advancements in the study of this phenomena have been complemented by the introduction of several music-specific models of emotion, with two of the most well-cited ones being the BRECVEMA and the Multifactorial Process Model. These two models have undoubtedly contributed to the field. However, contemporary developments in the wider affective sciences (broadly described as the ‘rise of affectivism’) have yet to be incorporated into the music emotion literature. These developments in the affective sciences may aid in addressing remaining gaps in the music literature, in particular for acknowledging individual and contextual differences. The first aim of this thesis was to outline contemporary theories from the wider affective sciences and subsequently critique current popular models of musical emotions through the lens of these advancements. The second aim was to propose a new model based on this critique: the Constructivistly-Organised Dimensional-Appraisal (CODA) model. This CODA model draws together multiple competing models into a single framework centralised around goal-directed appraisal mechanisms which are key to the wider affective sciences but are a less commonly acknowledged component of musical affect. The third aim was to empirically test some of the core hypotheses of the CODA model. In particular, examining goal-directed mechanisms, their validity in a musical context, and their ability to address individual and contextual differences in musically induced affect. Across four experiments which include exploratory and lab-based designs through to real- world applications, the results are supportive of the role of goal-directed mechanisms in musically induced emotional episodes. Experiment one presents a first test battery of multiple appraisal dimensions developed for music. The results show that several of the hypothesised appraisal dimensions are valid dimensions is a musical context. Moreover, these mechanisms cluster into goal-directed latent variables. Experiment two develops a new set of stimuli annotations relating to musical goals, showing that music can be more or less appropriate for different musical goals (functions). Experiment three, using the new stimuli set from experiment two, tests the effects of different goals with more or less appropriate music on musically induced affect. These results show that goal-directed mechanisms can change induced core-affect (valence and arousal) and intensity, even for the same piece of music. Experiment four extends the study of goal-directed mechanisms into a real-world context through an interdisciplinary and cross-cultural design. The final experiment demonstrates how goal-directed mechanisms can be manipulated through different algorithms to induce negative affect in a Colombian population. The main conclusions of this thesis are that the CODA model, more specifically goal-directed mechanisms, provide a valuable, non-reductive, and more efficient approach to addressing individual and contextual differences for musically induced emotional episodes in the new era of affectivism

Image Sentiment Analysis of Social Media Data

Often a picture is worth a thousand words, and this is a small statement that represents one of the biggest challenges in the Image Sentiment Analysis area. The main theme of this dissertation is the Image Sentiment Analysis of social media, mainly from Twitter, so that it is identified as situations that represent risks (identification of negative situations) or that become a risk (prediction of negative situations). Despite the diversity of work done in the area of image sentiment analysis, it is still a challenging task. Several factors contribute to the difficulty, both more global factors likewise sociocultural issues, and issues within the scope of the analysis of feeling in images, such as the difficulty in finding reliable and properly labeled data to be used, as well as factors faced during the classification, for example, it is normal to associate images with darker colors and low brightness to negative feelings, after all, most are like that, but some cases escape this rule, and it is these cases that affect the accuracy of the developed models. However, in order to overcome these problems faced in classification, a multitasking model was developed, which will consider the entire image information, information from the salient areas in the images, and the facial expressions of faces contained in the images, and textual information, so that each component complements the other during classification. During the experiments it was possible to observe that the use of the proposed models can bring advantages for the classification of feeling in images and even work around some problems evidenced in existing works, such as the irony of the text. Therefore, this work aims to present the state of the art and the study carried out, in order to enable the presentation and implementation of the proposed model and carrying out the experiments and discussion of the results obtained, in order to verify the effectiveness of what was proposed. Finally, conclusions about the work done and future work will be presented.Muitas vezes uma imagem vale mais que mil palavras, e esta é uma pequena afirmação que representa um dos maiores desafios da área de classificação do sentimento contido nas imagens. O principal tema desta dissertação é a realização da análise do sentimento contido em imagens das mídias sociais, principalmente do Twitter, de modo que possam ser identificadas as situações que representam riscos (identificação de situações negativas) ou as quais possam se tornar um (previsão de situações negativas). Apesar da diversidade de trabalhos feitos na área da análise de sentimento em imagens, ainda é uma tarefa desafiante. Diversos fatores contribuem para a dificuldade , tantos fatores mais globais como questões socioculturais, quanto questões do próprio âmbito de análise de sentimento em imagens, como a dificuldade em achar dados confiáveis e devidamente etiquetados para serem utilizados, quanto fatores enfrentados durante a classificação, como por exemplo, é normal associar imagens com cores mais escuras e pouco brilho à sentimentos negativos, afinal a maioria é assim, entretanto há casos que fogem dessa regra, e são esses casos que afetam a precisão dos modelos desenvolvidos. Porém, visando contornar esses problemas enfrentados na classificação, foi desenvolvido um modelo multitarefas, o qual irá considerar informações globais, áreas salientes nas imagens, expressões faciais de rostos contidos nas imagens e informação textual, de modo que cada componente se complemente durante a classificação. Durante os experimentos foi possível observar que o uso dos modelos propostos podem trazer vantagens para a classificação do sentimento em imagens e até mesmo contornar alguns problemas evidenciados nos trabalhos já existentes, como por exemplo a ironia do texto. Assim sendo, este trabalho tem como objetivo apresentar o estado da arte e o estudo realizado, de modo a possibilitar a apresentação e implementação do modelo multitarefas proposto e realização das experiências e discussão dos resultados obtidos, de forma a verificar a eficácia do método proposto. Por fim, as conclusões sobre o trabalho feito e trabalho futuro serão apresentados

Computational approaches to Explainable Artificial Intelligence: Advances in theory, applications and trends

Deep Learning (DL), a groundbreaking branch of Machine Learning (ML), has emerged as a driving force in both theoretical and applied Artificial Intelligence (AI). DL algorithms, rooted in complex and non-linear artificial neural systems, excel at extracting high-level features from data. DL has demonstrated human-level performance in real-world tasks, including clinical diagnostics, and has unlocked solutions to previously intractable problems in virtual agent design, robotics, genomics, neuroimaging, computer vision, and industrial automation. In this paper, the most relevant advances from the last few years in Artificial Intelligence (AI) and several applications to neuroscience, neuroimaging, computer vision, and robotics are presented, reviewed and discussed. In this way, we summarize the state-of-the-art in AI methods, models and applications within a collection of works presented at the 9 International Conference on the Interplay between Natural and Artificial Computation (IWINAC). The works presented in this paper are excellent examples of new scientific discoveries made in laboratories that have successfully transitioned to real-life applications

Neural correlates of emotion word processing: the interaction between emotional valence and arousal

Emotion is characterised by two-dimensions: emotional valence describes the extent to which an emotion is positive or negative, and arousal represents its intensity. Emotional content of verbal material affects cognitive processing, although research on word recognition has only recently taken emotion into account, primarily focusing on valence, while neglecting arousal. The present work aimed to disentangle the effects of valence and arousal during a lexical decision task, using reaction times (RTs), event-related potentials (ERPs) and BOLD responses in an event-related fMRI design. These methods were chosen to determine when affective features have an effect, and which neural systems are involved. The material for three experiments was based on a word corpus created by collecting ratings for emotional and lexico-semantic features. A first and novel finding was that arousal interacted with valence. Specifically, lexical decision times were slower for high-arousal positive stimuli (PH) and low-arousal negative ones (NL) compared to low-arousal positive (PL) and high arousal negative (NH) stimuli. ERPs also showed an interaction between 200-300 ms on the early posterior negativity (EPN), a component which is sensitive to emotional stimuli. At this processing stage people access their mental lexicon. Its amplitude was greater for PH and NL words, suggesting a higher processing load for conflicting stimuli. Positive valence and low arousal elicit an approach schema, whereas negative valence and high arousal elicit an avoidance schema (Robinson, Storbeck, Meier & Kirkeby, 2004). BOLD responses showed a similar interaction in the insula bilaterally, with increased activation for PH and NL words. This region is associated with integration of information on visceral states with higher-order cognitive and emotional processing, suggesting higher difficulty in integrating conflicting stimuli. Taken together, these studies indicate that emotion affects word processing during lexical access, and models of word recognition need to take into account both valence and arousal