Dynamic facial expression recognition (FER) databases provide important data
support for affective computing and applications. However, most FER databases
are annotated with several basic mutually exclusive emotional categories and
contain only one modality, e.g., videos. The monotonous labels and modality
cannot accurately imitate human emotions and fulfill applications in the real
world. In this paper, we propose MAFW, a large-scale multi-modal compound
affective database with 10,045 video-audio clips in the wild. Each clip is
annotated with a compound emotional category and a couple of sentences that
describe the subjects' affective behaviors in the clip. For the compound
emotion annotation, each clip is categorized into one or more of the 11
widely-used emotions, i.e., anger, disgust, fear, happiness, neutral, sadness,
surprise, contempt, anxiety, helplessness, and disappointment. To ensure high
quality of the labels, we filter out the unreliable annotations by an
Expectation Maximization (EM) algorithm, and then obtain 11 single-label
emotion categories and 32 multi-label emotion categories. To the best of our
knowledge, MAFW is the first in-the-wild multi-modal database annotated with
compound emotion annotations and emotion-related captions. Additionally, we
also propose a novel Transformer-based expression snippet feature learning
method to recognize the compound emotions leveraging the expression-change
relations among different emotions and modalities. Extensive experiments on
MAFW database show the advantages of the proposed method over other
state-of-the-art methods for both uni- and multi-modal FER. Our MAFW database
is publicly available from https://mafw-database.github.io/MAFW.Comment: This paper has been accepted by ACM MM'2