A PROBABILISTIC APPROACH TO THE CONSTRUCTION OF A MULTIMODAL AFFECT SPACE

Understanding affective signals from others is crucial for both human-human and human-agent interaction. The automatic analysis of emotion is by and large addressed as a pattern recognition problem which grounds in early psychological theories of emotion. Suitable features are first extracted and then used as input to classification (discrete emotion recognition) or regression (continuous affect detection). In this thesis, differently from many computational models in the literature, we draw on a simulationist approach to the analysis of facially displayed emotions - e.g., in the course of a face-to-face interaction between an expresser and an observer. At the heart of such perspective lies the enactment of the perceived emotion in the observer. We propose a probabilistic framework based on a deep latent representation of a continuous affect space, which can be exploited for both the estimation and the enactment of affective states in a multimodal space. Namely, we consider the observed facial expression together with physiological activations driven by internal autonomic activity. The rationale behind the approach lies in the large body of evidence from affective neuroscience showing that when we observe emotional facial expressions, we react with congruent facial mimicry. Further, in more complex situations, affect understanding is likely to rely on a comprehensive representation grounding the reconstruction of the state of the body associated with the displayed emotion. We show that our approach can address such problems in a unified and principled perspective, thus avoiding ad hoc heuristics while minimising learning efforts. Moreover, our model improves the inferred belief through the adoption of an inner loop of measurements and predictions within the central affect state-space, that realise the dynamics of the affect enactment. Results so far achieved have been obtained by adopting two publicly available multimodal corpora

From Photography to fMRI

Hysteria, a mysterious disease known since antiquity, is said to have ceased to exist. Challenging this commonly held view, this is the first cross-disciplinary study to examine the current functional neuroimaging research into hysteria and compare it to the nineteenth-century image-based research into the same disorder. Paula Muhr's central argument is that, both in the nineteenth-century and the current neurobiological research on hysteria, images have enabled researchers to generate new medical insights. Through detailed case studies, Muhr traces how different images, from photography to functional brain scans, have reshaped the historically situated medical understanding of this disorder that defies the mind-body dualism

From Photography to fMRI

Hysteria, a mysterious disease known since antiquity, is said to have ceased to exist. Challenging this commonly held view, this is the first cross-disciplinary study to examine the current functional neuroimaging research into hysteria and compare it to the nineteenth-century image-based research into the same disorder. Paula Muhr's central argument is that, both in the nineteenth-century and the current neurobiological research on hysteria, images have enabled researchers to generate new medical insights. Through detailed case studies, Muhr traces how different images, from photography to functional brain scans, have reshaped the historically situated medical understanding of this disorder that defies the mind-body dualism

From Photography to fMRI: Epistemic Functions of Images in Medical Research on Hysteria

Hysteria, a mysterious disease known since antiquity, is said to have ceased to exist. Challenging this commonly held view, this is the first cross-disciplinary study to examine the current functional neuroimaging research into hysteria and compare it to the nineteenth-century image-based research into the same disorder. Paula Muhr's central argument is that, both in the nineteenth-century and the current neurobiological research on hysteria, images have enabled researchers to generate new medical insights. Through detailed case studies, Muhr traces how different images, from photography to functional brain scans, have reshaped the historically situated medical understanding of this disorder that defies the mind-body dualism

Affective Communication for Socially Assistive Robots (SARs) for Children with Autism Spectrum Disorder: A Systematic Review

Research on affective communication for socially assistive robots has been conducted to enable physical robots to perceive, express, and respond emotionally. However, the use of affective computing in social robots has been limited, especially when social robots are designed for children, and especially those with autism spectrum disorder (ASD). Social robots are based on cognitiveaffective models, which allow them to communicate with people following social behaviors and rules. However, interactions between a child and a robot may change or be different compared to those with an adult or when the child has an emotional deficit. In this study, we systematically reviewed studies related to computational models of emotions for children with ASD. We used the Scopus, WoS, Springer, and IEEE-Xplore databases to answer different research questions related to the definition, interaction, and design of computational models supported by theoretical psychology approaches from 1997 to 2021. Our review found 46 articles; not all the studies considered children or those with ASD.This research was funded by VRIEA-PUCV, grant number 039.358/202

Integration of psychological models in the design of artificial creatures

Artificial creatures form an increasingly important component of interactive computer games. Examples of such creatures exist which can interact with each other and the game player and learn from their experiences. However, we argue, the design of the underlying architecture and algorithms has to a large extent overlooked knowledge from psychology and cognitive sciences. We explore the integration of observations from studies of motivational systems and emotional behaviour into the design of artificial creatures. An initial implementation of our ideas using the “sim agent” toolkit illustrates that physiological models can be used as the basis for creatures with animal like behaviour attributes. The current aim of this research is to increase the “realism” of artificial creatures in interactive game-play, but it may have wider implications for the development of AI

Affect-based information retrieval

One of the main challenges Information Retrieval (IR) systems face nowadays originates from the semantic gap problem: the semantic difference between a user’s query representation and the internal representation of an information item in a collection. The gap is further widened when the user is driven by an ill-defined information need, often the result of an anomaly in his/her current state of knowledge. The formulated search queries, which are submitted to the retrieval systems to locate relevant items, produce poor results that do not address the users’ information needs. To deal with information need uncertainty IR systems have employed in the past a range of feedback techniques, which vary from explicit to implicit. The first category of feedback techniques necessitates the communication of explicit relevance judgments, in return for better query reformulations and recommendations of relevant results. However, the latter happens at the expense of users’ cognitive resources and, furthermore, introduces an additional layer of complexity to the search process. On the other hand, implicit feedback techniques make inferences on what is relevant based on observations of user search behaviour. By doing so, they disengage users from the cognitive burden of document rating and relevance assessments. However, both categories of RF techniques determine topical relevance with respect to the cognitive and situational levels of interaction, failing to acknowledge the importance of emotions in cognition and decision making. In this thesis I investigate the role of emotions in the information seeking process and develop affective feedback techniques for interactive IR. This novel feedback framework aims to aid the search process and facilitate a more natural and meaningful interaction. I develop affective models that determine topical relevance based on information gathered from various sensory channels, and enhance their performance using personalisation techniques. Furthermore, I present an operational video retrieval system that employs affective feedback to enrich user profiles and offers meaningful recommendations of unseen videos. The use of affective feedback as a surrogate for the information need is formalised as the Affective Model of Browsing. This is a cognitive model that motivates the use of evidence extracted from the psycho-somatic mobilisation that occurs during cognitive appraisal. Finally, I address some of the ethical and privacy issues that arise from the social-emotional interaction between users and computer systems. This study involves questionnaire data gathered over three user studies, from 74 participants of different educational background, ethnicity and search experience. The results show that affective feedback is a promising area of research and it can improve many aspects of the information seeking process, such as indexing, ranking and recommendation. Eventually, it may be that relevance inferences obtained from affective models will provide a more robust and personalised form of feedback, which will allow us to deal more effectively with issues such as the semantic gap

From Photography to fMRI

Hysteria, a mysterious disease known since antiquity, is said to have ceased to exist. Challenging this commonly held view, this is the first cross-disciplinary study to examine the current functional neuroimaging research into hysteria and compare it to the nineteenth-century image-based research into the same disorder. Paula Muhr's central argument is that, both in the nineteenth-century and the current neurobiological research on hysteria, images have enabled researchers to generate new medical insights. Through detailed case studies, Muhr traces how different images, from photography to functional brain scans, have reshaped the historically situated medical understanding of this disorder that defies the mind-body dualism