151 research outputs found
Modeling Multimodal Cues in a Deep Learning-based Framework for Emotion Recognition in the Wild
In this paper, we propose a multimodal deep learning architecture for emotion recognition in video regarding our participation to the audio-video based sub-challenge of the Emotion Recognition in the Wild 2017 challenge. Our model combines cues from multiple video modalities, including static facial features, motion patterns related to the evolution of the human expression over time, and audio information. Specifically, it is composed of three sub-networks trained separately: the first and second ones extract static visual features and dynamic patterns through 2D and 3D Convolutional Neural Networks (CNN), while the third one consists in a pretrained audio network which is used to extract useful deep acoustic signals from video. In the audio branch, we also apply Long Short Term Memory (LSTM) networks in order to capture the temporal evolution of the audio features. To identify and exploit possible relationships among different modalities, we propose a fusion network that merges cues from the different modalities in one representation. The proposed architecture outperforms the challenge baselines (38.81% and 40.47%): we achieve an accuracy of 50.39% and 49.92% respectively on the validation and the testing data
A Literature Review on Emotion Recognition Using Various Methods
Emotion Recognition is an important area of work to improve the interaction between human and machine. Complexity of emotion makes the acquisition task more difficult. Quondam works are proposed to capture emotion through unimodal mechanism such as only facial expressions or only vocal input. More recently, inception to the idea of multimodal emotion recognition has increased the accuracy rate of the detection of the machine. Moreover, deep learning technique with neural network extended the success ratio of machine in respect of emotion recognition. Recent works with deep learning technique has been performed with different kinds of input of human behavior such as audio-visual inputs, facial expressions, body gestures, EEG signal and related brainwaves. Still many aspects in this area to work on to improve and make a robust system will detect and classify emotions more accurately. In this paper, we tried to explore the relevant significant works, their techniques, and the effectiveness of the methods and the scope of the improvement of the results
MIMAMO Net: Integrating Micro- and Macro-motion for Video Emotion Recognition
Spatial-temporal feature learning is of vital importance for video emotion
recognition. Previous deep network structures often focused on macro-motion
which extends over long time scales, e.g., on the order of seconds. We believe
integrating structures capturing information about both micro- and macro-motion
will benefit emotion prediction, because human perceive both micro- and
macro-expressions. In this paper, we propose to combine micro- and macro-motion
features to improve video emotion recognition with a two-stream recurrent
network, named MIMAMO (Micro-Macro-Motion) Net. Specifically, smaller and
shorter micro-motions are analyzed by a two-stream network, while larger and
more sustained macro-motions can be well captured by a subsequent recurrent
network. Assigning specific interpretations to the roles of different parts of
the network enables us to make choice of parameters based on prior knowledge:
choices that turn out to be optimal. One of the important innovations in our
model is the use of interframe phase differences rather than optical flow as
input to the temporal stream. Compared with the optical flow, phase differences
require less computation and are more robust to illumination changes. Our
proposed network achieves state of the art performance on two video emotion
datasets, the OMG emotion dataset and the Aff-Wild dataset. The most
significant gains are for arousal prediction, for which motion information is
intuitively more informative. Source code is available at
https://github.com/wtomin/MIMAMO-Net.Comment: Accepted by AAAI 202
- …