3D-CNN for Facial Micro- and Macro-expression Spotting on Long Video Sequences using Temporal Oriented Reference Frame

Facial expression spotting is the preliminary step for micro- and macro-expression analysis. The task of reliably spotting such expressions in video sequences is currently unsolved. The current best systems depend upon optical flow methods to extract regional motion features, before categorisation of that motion into a specific class of facial movement. Optical flow is susceptible to drift error, which introduces a serious problem for motions with long-term dependencies, such as high frame-rate macro-expression. We propose a purely deep learning solution which, rather than track frame differential motion, compares via a convolutional model, each frame with two temporally local reference frames. Reference frames are sampled according to calculated micro- and macro-expression durations. We show that our solution achieves state-of-the-art performance (F1-score of 0.126) in a dataset of high frame-rate (200 fps) long video sequences (SAMM-LV) and is competitive in a low frame-rate (30 fps) dataset (CAS(ME)2). In this paper, we document our deep learning model and parameters, including how we use local contrast normalisation, which we show is critical for optimal results. We surpass a limitation in existing methods, and advance the state of deep learning in the domain of facial expression spotting

Facial Micro-Expression Recognition Based on Deep Local-Holistic Network

A micro-expression is a subtle, local and brief facial movement. It can reveal the genuine emotions that a person tries to conceal and is considered an important clue for lie detection. The micro-expression research has attracted much attention due to its promising applications in various fields. However, due to the short duration and low intensity of micro-expression movements, microexpression recognition faces great challenges, and the accuracy still demands improvement. To improve the efficiency of micro-expression feature extraction, inspired by the psychological study of attentional resource allocation for micro-expression cognition, we propose a deep local-holistic network method for micro-expression recognition. Our proposed algorithm consists of two subnetworks. The first is a Hierarchical Convolutional Recurrent Neural Network (HCRNN), which extracts the local and abundant spatio-temporal micro-expression features. The second is a Robust principal-component-analysis-based recurrent neural network (RPRNN), which extracts global and sparse features with micro-expression-specific representations. The extracted effective features are employed for micro-expression recognition through the fusion of sub-networks. We evaluate the proposed method on combined databases consisting of the four most commonly used databases, i.e., CASME, CASME II, CAS(ME)(2) , and SAMM. The experimental results show that our method achieves a reasonably good performance.</p

MESNet: A Convolutional Neural Network for Spotting Multi-Scale Micro-Expression Intervals in Long Videos

Micro-expression spotting is a fundamental step in the micro-expression analysis. This paper proposes a novel network based convolutional neural network (CNN) for spotting multi-scale spontaneous micro-expression intervals in long videos. We named the network as Micro-Expression Spotting Network (MESNet). It is composed of three modules. The first module is a 2+1D Spatiotemporal Convolutional Network, which uses 2D convolution to extract spatial features and 1D convolution to extract temporal features. The second module is a Clip Proposal Network, which gives some proposed micro-expression clips. The last module is a Classification Regression Network, which classifies the proposed clips to micro-expression or not, and further regresses their temporal boundaries. We also propose a novel evaluation metric for spotting micro-expression. Extensive experiments have been conducted on the two long video datasets: CAS(ME)(2) and SAMM, and the leave-one-subject-out cross-validation is used to evaluate the spotting performance. Results show that the proposed MESNet effectively enhances the F1-score metric. And comparative results show the proposed MESNet has achieved a good performance, which outperforms other state-of-the-art methods, especially in the SAMM dataset

Simple but Effective In-the-wild Micro-Expression Spotting Based on Head Pose Segmentation

Micro-expressions may occur in high-stake situations when people attempt to conceal or suppress their true feelings. Nowadays, intelligent micro-expression analysis has long been focused on videos captured under constrained laboratory conditions. This is due to the relatively small number of publicly available datasets. Moreover, micro-expression characteristics are subtle and brief, and thus very susceptible to interference from external factors and difficult to capture. In particular, head movement is unavoidable in unconstrained scenarios, making micro-expression spotting highly challenging. This paper proposes a simple yet effective method for avoiding the interference of head movement on micro-expression spotting in natural scenarios by considering three-dimensional space. In particular, based on the head pose, which can be mapped to two-dimensional vectors (translations and rotations) for representation, long and complex videos could be divided into short video segments that basically exclude head movement interference. Following that, segmented micro-expression spotting is realized based on an effective short-segment-based micro-expression spotting algorithm. Experimental results on in-the-wild databases demonstrate the effectiveness of our proposed method in avoiding head movement interference. Additionally, due to the simplicity of this method, it creates opportunities for spotting micro-expressions in real-world scenarios, possibly even in real-time. Furthermore, it helps alleviate the small sample size problem in micro-expression analysis by boosting the spotting performance in massive unlabeled videos.</p

"Can It Be Customized According to My Motor Abilities?": Toward Designing User-Defined Head Gestures for People with Dystonia

Recent studies proposed above-the-neck gestures for people with upper-body motor impairments interacting with mobile devices without finger touch, resulting in an appropriate user-defined gesture set. However, many gestures involve sustaining eyelids in closed or open states for a period. This is challenging for people with dystonia, who have difficulty sustaining and intermitting muscle contractions. Meanwhile, other facial parts, such as the tongue and nose, can also be used to alleviate the sustained use of eyes in the interaction. Consequently, we conducted a user study inviting 16 individuals with dystonia to design gestures based on facial muscle movements for 26 common smartphone commands. We collected 416 user-defined head gestures involving facial features and shoulders. Finally, we obtained the preferred gestures set for individuals with dystonia. Participants preferred to make the gestures with their heads and use unnoticeable gestures. Our findings provide valuable references for the universal design of natural interaction technology.</p

Spotting Micro-Expressions on Long Videos Sequences

International audienceThis paper presents two methods for the first Micro-Expression Spotting Challenge 2019 by evaluating local temporal pattern (LTP) and local binary pattern (LBP) on two most recent databases, i. e. SAMM and CAS(ME)(2). First we propose LTP-ML method as the baseline results for the challenge and then we compare the results with the LBP-chi(2) distance method. The LTP patterns are extracted by applying PCA in a temporal window on several facial local regions. The micro-expression sequences are then spotted by a local classification of LTP and a global fusion. The LBP-chi(2)-distance method is to compare the feature difference by calculating chi(2) distance of LBP in a time window, the facial movements are then detected with a threshold. The performance is evaluated by Leave-One-Subject-Out cross validation. The overlap frames are used to determine the True Positives and the metric F1-score is used to compare the spotting performance of the databases. The F1-score of LTP-ML result for SAMM and CAS(ME)(2) are 0.0316 and 0.0179, respectively. The results show our proposed LTP-ML method outperformed LBP-chi(2)-distance method in terms of F1-score on both databases

Research progress of kesterite solar cells; [铜锌锡硫基薄膜太阳电池研究进展]

National audiencePhotovoltaic technology offers a sustainable solution to the challenge of increasing energy demand. Nowadays, various high-performance solar cells are emerging. Thin-film solar cells made from inorganic materials have become one of the major categories of solar cells, showing potential in the fast-growing photovoltaic (PV) market. The production technology of Cu(In,Ga)Se2 (CIGS) solar cells and CdTe solar cells has reached a mature level. The earth-abundant and environmentally friendly kerterite Cu2ZnSn(S,Se)4 (CZTSSe) is a promising alternative to chalcopyrite CIGS and CdTe for PV applications and is considered to be a cost-effective next-generation solar cell material. The crystal structure of the CZTSSe absorber material is derived from CIGS, in which In and Ga are replaced by one group II (Zn) cation and one group IV (Sn) cation, and has a similar lattice and energy band structure with CIGS. Therefore, CZTSSe inherits the advantages of high absorption coefficient, adjustable band gap, and inherent P-type conductivity, and has the new advantages of non-toxicity and abundant element reserves. It is a new generation of thin film photovoltaic technology with high efficiency, stability, safety, environmental protection, and low price. CZTSSe PV technology has made significant progress in the past few years, reaching a maximum efficiency of 14.9%, but still far below CIGS (23.6%) and CdTe (22.1%). The undesirable back/front interface is one of the main reasons for the difficulty in improving the fill factor. The detrimental interface reaction results in a large number of secondary phases, voids and defects in absorbers, which can form abundant recombination centers and limit the minority carrier diffusion length. The thicker Mo(S,Se)2 layer at the back interface leads to carrier transport barriers and has a negative impact on the crystalline quality of the absorber; high density of interface defects, unfavorable band alignment, and structural inhomogeneity across the front interface are the main factors leading to heterojunction recombination. Meanwhile, kesterite, as one of the most complex compound semiconductors, has a more complex defect chemistry than CIGS and CdTe, making the control of intrinsic defects a major challenge. Deep limit defects, such as deep defect SnZn and associated [CuZn+SnZn] clusters, act as deep recombination centers, leading to short carrier lifetimes. In addition, a large number of defect clusters like [2CuZn+SnZn] introduce considerable potential (i.e., band or electrostatic) fluctuations. As a result, the performance of kerterite-based solar cells is currently stagnant due to low fill factor and large open-circuit voltage (VOC) deficits. In this review, the state-of-the-art strategies to improve the device performance are provided, with a particular focus on back and front-interface engineering, cation substitution, and selenization annealing, post-annealing processes and so on. These strategies have led to step-wise improvements in the power conversion efficiency (PCE) of the corresponding kesterite solar cells and are the most promising approaches to achieve further efficiency breakthroughs in kesterite solar cells. This paper reviews the recent research progress around these pathways in kesterite solar cells and, more importantly, provides a comprehensive understanding of the mechanisms at play and an outlook on the future development of kesterite solar cells. © 2023 Chinese Academy of Sciences. All rights reserved

Spontaneous Facial Expressions and Micro-expressions Coding: From Brain to Face

Facial expressions are a vital way for humans to show their perceived emotions. It is convenient for detecting and recognizing expressions or micro-expressions by annotating a lot of data in deep learning. However, the study of video-based expressions or micro-expressions requires that coders have professional knowledge and be familiar with action unit (AU) coding, leading to considerable difficulties. This paper aims to alleviate this situation. We deconstruct facial muscle movements from the motor cortex and systematically sort out the relationship among facial muscles, AU, and emotion to make more people understand coding from the basic principles: We derived the relationship between AU and emotion based on a data-driven analysis of 5,000 images from the RAF-AU database, along with the experience of professional coders.We discussed the complex facial motor cortical network system that generates facial movement properties, detailing the facial nucleus and the motor system associated with facial expressions.The supporting physiological theory for AU labeling of emotions is obtained by adding facial muscle movements patterns.We present the detailed process of emotion labeling and the detection and recognition of AU.Based on the above research, the video&#39;s coding of spontaneous expressions and micro-expressions is concluded and prospected.</p

Extreme Learning Machine for Multi-Label Classification

Extreme learning machine (ELM) techniques have received considerable attention in the computational intelligence and machine learning communities because of the significantly low computational time required for training new classifiers. ELM provides solutions for regression, clustering, binary classification, multiclass classifications and so on, but not for multi-label learning. Multi-label learning deals with objects having multiple labels simultaneously, which widely exist in real-world applications. Therefore, a thresholding method-based ELM is proposed in this paper to adapt ELM to multi-label classification, called extreme learning machine for multi-label classification (ELM-ML). ELM-ML outperforms other multi-label classification methods in several standard data sets in most cases, especially for applications which only have a small labeled data set