Towards emotional interaction: using movies to automatically learn users’ emotional states

The HCI community is actively seeking novel methodologies to gain insight into the user's experience during interaction with both the application and the content. We propose an emotional recognition engine capable of automatically recognizing a set of human emotional states using psychophysiological measures of the autonomous nervous system, including galvanic skin response, respiration, and heart rate. A novel pattern recognition system, based on discriminant analysis and support vector machine classifiers is trained using movies' scenes selected to induce emotions ranging from the positive to the negative valence dimension, including happiness, anger, disgust, sadness, and fear. In this paper we introduce an emotion recognition system and evaluate its accuracy by presenting the results of an experiment conducted with three physiologic sensors.info:eu-repo/semantics/publishedVersio

A Connotative Space for Supporting Movie Affective Recommendation

The problem of relating media content to users’affective responses is here addressed. Previous work suggests that a direct mapping of audio-visual properties into emotion categories elicited by films is rather difficult, due to the high variability of individual reactions. To reduce the gap between the objective level of video features and the subjective sphere of emotions, we propose to shift the representation towards the connotative properties of movies, in a space inter-subjectively shared among users. Consequently, the connotative space allows to define, relate and compare affective descriptions of film videos on equal footing. An extensive test involving a significant number of users watching famous movie scenes, suggests that the connotative space can be related to affective categories of a single user. We apply this finding to reach high performance in meeting user’s emotional preferences

Movies and meaning: from low-level features to mind reading

When dealing with movies, closing the tremendous discontinuity between low-level features and the richness of semantics in the viewers' cognitive processes, requires a variety of approaches and different perspectives. For instance when attempting to relate movie content to users' affective responses, previous work suggests that a direct mapping of audio-visual properties into elicited emotions is difficult, due to the high variability of individual reactions. To reduce the gap between the objective level of features and the subjective sphere of emotions, we exploit the intermediate representation of the connotative properties of movies: the set of shooting and editing conventions that help in transmitting meaning to the audience. One of these stylistic feature, the shot scale, i.e. the distance of the camera from the subject, effectively regulates theory of mind, indicating that increasing spatial proximity to the character triggers higher occurrence of mental state references in viewers' story descriptions. Movies are also becoming an important stimuli employed in neural decoding, an ambitious line of research within contemporary neuroscience aiming at "mindreading". In this field we address the challenge of producing decoding models for the reconstruction of perceptual contents by combining fMRI data and deep features in a hybrid model able to predict specific video object classes

Multimedia interaction and access based on emotions:automating video elicited emotions recognition and visualization

Tese de doutoramento, Informática (Engenharia Informática), Universidade de Lisboa, Faculdade de Ciências, 2013Films are an excellent form of art that exploit our affective, perceptual and intellectual abilities. Technological developments and the trends for media convergence are turning video into a dominant and pervasive medium, and online video is becoming a growing entertainment activity on the web. Alongside, physiological measures are making it possible to study additional ways to identify and use emotions in human-machine interactions, multimedia retrieval and information visualization. The work described in this thesis has two main objectives: to develop an Emotions Recognition and Classification mechanism for video induced emotions; and to enable Emotional Movie Access and Exploration. Regarding the first objective, we explore recognition and classification mechanisms, in order to allow video classification based on emotions, and to identify each user’s emotional states providing different access mechanisms. We aim to provide video classification and indexing based on emotions, felt by the users while watching movies. In what concerns the second objective, we focus on emotional movie access and exploration mechanisms to find ways to access and visualize videos based on their emotional properties and users’ emotions and profiles. In this context, we designed a set of methods to access and watch the movies, both at the level of the whole movie collection, and at the individual movies level. The automatic recognition mechanism developed in this work allows for the detection of physiologic patterns, indeed providing valid individual information about users emotion while they were watching a specific movie; in addition, the user interface representations and exploration mechanisms proposed and evaluated in this thesis, show that more perceptive, satisfactory and useful visual representations influenced positively the exploration of emotional information in movies.Fundação para a Ciência e a Tecnologia (FCT, PROTEC SFRH/BD/49475/2009, LASIGE Multiannual Funding e VIRUS projecto (PTDC/EIAEIA/101012/2008

Stress and heart rate: significant parameters and their variations

The aim of this paper is to identify heart rate parameters with higher significant values when a set of people are performing a task under stress condition. In order to accomplish this, one computer application with arithmetic and memory activities which lets drive the subjects to different stages of activity and stress has been designed. Tests are formed by initial and final rest periods and three task phases with incremental stressful level. Electrocardiogram is measured in each state and parameters are extracted from it. A statistical study using analysis of variance (ANOVA) is done to see which ones are the most significant. It is concluded that the median of RR segments is the parameter to best determine the state of stress.Regional Government of Andalusia (p08-TIC-3631

An Ambient Multimedia User Experience Feedback Framework Based on User Tagging and EEG Biosignals

Multimedia is increasingly accessed online and within social networks; however, users are typically limited to visual/auditory stimulus through media presented onscreen with accompanying audio over speakers. Whilst recent research studying additional ambient sensory multimedia effects recorded numerical scores of perceptual quality, the users’ time-varying emotional response to the ambient sensory feedback is not considered. This paper thus introduces a framework to evaluate user ambient quality of multimedia experience and discover users’ time-varying emotional responses through explicit user tagging and implicit EEG biosignal analysis. In the proposed framework, users interact with the media via discrete tagging activities whilst their EEG biosignal emotional feedback is continuously monitored in-between user tagging events with emotional states correlated with media content and tags

Implicit emotional tagging of multimedia using EEG signals and brain computer interface

In multimedia content sharing social networks, tags assigned to content play an important role in search and retrieval. In other words, by annotating multimedia content, users can associate a word or a phrase (tag) with that resource such that it can be searched for efficiently. Implicit tagging refers to assigning tags by observing subjects behavior during consumption of multimedia content. This is an alternative to traditional explicit tagging which requires an explicit action by subjects. In this paper we propose a brain-computer interface (BCI) system based on P300 evoked potential, for implicit emotional tagging of multimedia content. We show that our system can successfully perform implicit emotional tagging and naïve subjects who have not participated in training of the system can also use it efficiently. Moreover, we introduce a subjective metric called “emotional taggability” to analyze the recognition performance of the system, given the degree of ambiguity that exists in terms of emotional values associated with a multimedia content

Methods for Affective Content Analysis and Recognition in Film

The research presented in this thesis resulted from the growing attention on the effects of emotion on users, raising questions about their potential application to computational systems. This research investigates the best methods for determining affective scoring for video content, specifically films. This resulted in the affective video system (AVS) framework, AVS dataset and AVS systems being developed, leading to several contributions to knowledge about the best affective methods and systems. This work presents the necessary theory to understand the subject area. It builds as the thesis matures, laying a pathway in the form of a methodology framework for viewing affective problems and systems, moving into a subsequent study reviewing the well-recognised affective methods such as the International Affective Picture System (IAPS) and how its well-defined processes and procedures could be adapted for a more modern approach using video content. The research then studies the most critical perceivable features from video clips for users, which were analysed using the repertory grid approach. This led to the above contributions being combined to create the AVS system and database, which is a unique database comprising the affective scores for various film clips. This research concluded with the presentation of the best regression methods resulting from this research and its datasets and a summary of this performance, and discussions of the results in terms of other research in this area

Using Movies to Probe the Neurobiology of Anxiety

Over the past century, research has helped us build a fundamental understanding of the neurobiological underpinnings of anxiety. Specifically, anxiety engages a broad range of cortico-subcortical neural circuitry. Core to this is a ‘defensive response network’ which includes an amygdala-prefrontal circuit that is hypothesized to drive attentional amplification of threat-relevant stimuli in the environment. In order to help prepare the body for defensive behaviors to threat, anxiety also engages peripheral physiological systems. However, our theoretical frameworks of the neurobiology of anxiety are built mostly on the foundations of tightly-controlled experiments, such as task-based fMRI. Whether these findings generalize to more naturalistic settings is unknown. To address this shortcoming, movie-watching paradigms offer an effective tool at the intersection of tightly controlled and entirely naturalistic experiments. Particularly, using suspenseful movies presents a novel and effective means to induce and study anxiety. In this thesis, I demonstrate the potential of movie-watching paradigms in the study of how trait and state anxiety impact the ‘defensive response network’ in the brain, as well as peripheral physiology. The key findings reveal that trait anxiety is associated with differing amygdala-prefrontal responses to suspenseful movies; specific trait anxiety symptoms are linked to altered states of anxiety during suspenseful movies; and states of anxiety during movies impact brain-body communication. Notably, my results frequently diverged from those of conventional task-based experiments. Taken together, the insights gathered from this thesis underscore the utility of movie-watching paradigms for a more nuanced understanding of how anxiety impacts the brain and peripheral physiology. These outcomes provide compelling evidence that further integration of naturalistic methods will be beneficial in the study of the neurobiology of anxiety