25 research outputs found
How to Synthesize a Large-Scale and Trainable Micro-Expression Dataset?
This paper does not contain technical novelty but introduces our key
discoveries in a data generation protocol, a database and insights. We aim to
address the lack of large-scale datasets in micro-expression (MiE) recognition
due to the prohibitive cost of data collection, which renders large-scale
training less feasible. To this end, we develop a protocol to automatically
synthesize large scale MiE training data that allow us to train improved
recognition models for real-world test data. Specifically, we discover three
types of Action Units (AUs) that can constitute trainable MiEs. These AUs come
from real-world MiEs, early frames of macro-expression videos, and the
relationship between AUs and expression categories defined by human expert
knowledge. With these AUs, our protocol then employs large numbers of face
images of various identities and an off-the-shelf face generator for MiE
synthesis, yielding the MiE-X dataset. MiE recognition models are trained or
pre-trained on MiE-X and evaluated on real-world test sets, where very
competitive accuracy is obtained. Experimental results not only validate the
effectiveness of the discovered AUs and MiE-X dataset but also reveal some
interesting properties of MiEs: they generalize across faces, are close to
early-stage macro-expressions, and can be manually defined.Comment: European Conference on Computer Vision 202
Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition
In the recent year, state-of-the-art for facial micro-expression recognition
have been significantly advanced by deep neural networks. The robustness of
deep learning has yielded promising performance beyond that of traditional
handcrafted approaches. Most works in literature emphasized on increasing the
depth of networks and employing highly complex objective functions to learn
more features. In this paper, we design a Shallow Triple Stream
Three-dimensional CNN (STSTNet) that is computationally light whilst capable of
extracting discriminative high level features and details of micro-expressions.
The network learns from three optical flow features (i.e., optical strain,
horizontal and vertical optical flow fields) computed based on the onset and
apex frames of each video. Our experimental results demonstrate the
effectiveness of the proposed STSTNet, which obtained an unweighted average
recall rate of 0.7605 and unweighted F1-score of 0.7353 on the composite
database consisting of 442 samples from the SMIC, CASME II and SAMM databases.Comment: 5 pages, 1 figure, Accepted and published in IEEE FG 201
Time Efficient Micro-Expression Recognition using Weighted Spatio-Temporal Landmark Graphs
Micro-expressions have been shown to be effective in understanding the genuine emotions of a person. While many advances have been made in detecting micro-expressions using deep learning, previous studies in recognizing micro-expressions require pre-processing steps and the use of large feature sets resulting in large runtimes and thus have limited applicability in real-world scenarios. In this paper, we propose time-efficient end-to-end framework which uses landmark-based positional features to generate spatio-temporal graphs that can be applied to micro-expression recognition using Graph Convolutional NeuralNetworks (GCNs). We explore the importance of landmark features and propose a selective feature reduction approach to further improve efficiency. We perform experiments using the SMIC, CASMEII and SAMM datasets and demonstrate that our approach significantly speeds up predictions and delivers resultscomparable to the state-of-the-art
3D-CNN for Facial Micro- and Macro-expression Spotting on Long Video Sequences using Temporal Oriented Reference Frame
Facial expression spotting is the preliminary step for micro- and
macro-expression analysis. The task of reliably spotting such expressions in
video sequences is currently unsolved. The current best systems depend upon
optical flow methods to extract regional motion features, before categorisation
of that motion into a specific class of facial movement. Optical flow is
susceptible to drift error, which introduces a serious problem for motions with
long-term dependencies, such as high frame-rate macro-expression. We propose a
purely deep learning solution which, rather than track frame differential
motion, compares via a convolutional model, each frame with two temporally
local reference frames. Reference frames are sampled according to calculated
micro- and macro-expression durations. We show that our solution achieves
state-of-the-art performance (F1-score of 0.126) in a dataset of high
frame-rate (200 fps) long video sequences (SAMM-LV) and is competitive in a low
frame-rate (30 fps) dataset (CAS(ME)2). In this paper, we document our deep
learning model and parameters, including how we use local contrast
normalisation, which we show is critical for optimal results. We surpass a
limitation in existing methods, and advance the state of deep learning in the
domain of facial expression spotting
Data Leakage and Evaluation Issues in Micro-Expression Analysis
Micro-expressions have drawn increasing interest lately due to various
potential applications. The task is, however, difficult as it incorporates many
challenges from the fields of computer vision, machine learning and emotional
sciences. Due to the spontaneous and subtle characteristics of
micro-expressions, the available training and testing data are limited, which
make evaluation complex. We show that data leakage and fragmented evaluation
protocols are issues among the micro-expression literature. We find that fixing
data leaks can drastically reduce model performance, in some cases even making
the models perform similarly to a random classifier. To this end, we go through
common pitfalls, propose a new standardized evaluation protocol using facial
action units with over 2000 micro-expression samples, and provide an open
source library that implements the evaluation protocols in a standardized
manner. Code will be available in \url{https://github.com/tvaranka/meb}