Identity-adaptive Facial Expression Recognition Through Expression Regeneration Using Conditional Generative Adversarial Networks

Subject variation is a challenging issue for facial expression recognition, especially when handling unseen subjects with small-scale labeled facial expression databases. Although transfer learning has been widely used to tackle the problem, the performance degrades on new data. In this paper, we present a novel approach (so-called IA-gen) to alleviate the issue of subject variations by regenerating expressions from any input facial images. First of all, we train conditional generative models to generate six prototypic facial expressions from any given query face image while keeping the identity related information unchanged. Generative Adversarial Networks are employed to train the conditional generative models, and each of them is designed to generate one of the prototypic facial expression images. Second, a regular CNN (FER-Net) is fine- tuned for expression classification. After the corresponding prototypic facial expressions are regenerated from each facial image, we output the last FC layer of FER-Net as features for both the input image and the generated images. Based on the minimum distance between the input image and the generated expression images in the feature space, the input image is classified as one of the prototypic expressions consequently. Our proposed method can not only alleviate the influence of inter-subject variations but will also be flexible enough to integrate with any other FER CNNs for person-independent facial expression recognition. Our method has been evaluated on CK+, Oulu-CASIA, BU-3DFE and BU-4DFE databases, and the results demonstrate the effectiveness of our proposed method

Ion Transport in the Freshwater Bivalve, Corbicula Fluminea (Muller).

Corbicula fluminea (Muller) maintains hydromineral balance in dilute media by active transepithelial transport of Na and Cl. Cl uptake depended on external (Cl) and displayed saturable kinetics. The transport capacity in pondwater(PW)-acclimated animals was 7.00 $\pm$ 0.51 $\mu$eq(g dry tissue $\cdot$ h)\sp{-1} and the transport affinity 0.21 $\pm$ 0.08 mM. Prolonged salt depletion increased Cl transport capacity without changing the affinity. In PW-acclimated C. fluminea, Na and Cl transport were independent, and were stimulated by serotonin. In salt depleted (SD)animals, Na transport was partially Cl-dependent but Cl transport was Na-independent. Acetazolamide increased Na and Cl efflux. Both serotonin and acetazolamide promoted the loss of titratable base. N, N\sp\prime-dicyclohexylcarbodiimide (DCCD) inhibited Na and Cl transport. The inhibition of Na transport by DCCD was Cl-dependent, but the inhibition of Cl transport was Na-independent. DCCD increased the loss of titratable base in Cl-free PW medium. 4,4\sp\prime-diisothiocyanostilbene-2,2\sp\prime-disulfonic acid decreased Cl influx and Na net flux. Furosemide inhibited Na and Cl transport in PW-acclimated animals. Exposure of C. fluminea to hyperosmotic nonelectrolytes resulted in an elevation of blood solutes due to dehydration followed by a precipitous decrease in blood Na and the accumulation of nonelectrolytes. Lanthanum was rarely observed to penetrate the paracellular spaces without hyperosmotic stress, but was observed in the paracellular junctional complexes between gill epithelial cells under hyperosmotic conditions. Longer exposure resulted in greater amounts of lanthanum precipitation in more locations compared to shorter exposure. Carunculina texasensis was able to maintain normal blood ion concentration for 8 h with minimal dehydration under hyperosmotic condition. Longer exposure caused a precipitous decrease in most blood solutes and an extensive accumulation of nonelectrolytes. More lanthanum was observed in the paracellular spaces of C. texasensis compared to C. fluminea for identical treatments. This study suggests that Na and Cl transport in C. fluminea are energized by a proton-pump. The entrance of Na is likely via an apical Na-conductive channel and Cl via an apical Cl/HCO\sb3 exchange pathway. It also suggests that under normal conditions the epithelium of C. fluminea is relatively tight and the paracellular pathway for solutes is insignificant

Interlayer electric multipoles induced by in-plane field from quantum geometric origins

We show that interlayer charge transfer in 2D materials can be driven by an in-plane electric field, giving rise to electrical multipole generation in linear and second order of in-plane field. The linear and nonlinear effects have quantum geometric origins in the Berry curvature and quantum metric respectively, defined in extended parameter spaces characteristic of layered materials. We elucidate their symmetry characters, and demonstrate sizable dipole and quadrupole polarizations respectively in twisted bilayers and trilayers of transition metal dichalcogenides. Furthermore, we show that the effect is strongly enhanced during the topological phase transition tuned by interlayer translation. The effects point to a new electric control on layer quantum degree of freedom.Comment: 13 pages, 4 figure

Multi-modality Empowered Network For Facial Action Unit Detection

This paper presents a new thermal empowered multi-task network (TEMT-Net) to improve facial action unit detection. Our primary goal is to leverage the situation that the training set has multi-modality data while the application scenario only has one modality. Thermal images are robust to illumination and face color. In the proposed multi-task framework, we utilize both modality data. Action unit detection and facial landmark detection are correlated tasks. To utilize the advantage and the correlation of different modalities and different tasks, we propose a novel thermal empowered multi-task deep neural network learning approach for action unit detection, facial landmark detection and thermal image reconstruction simultaneously. The thermal image generator and facial landmark detection provide regularization on the learned features with shared factors as the input color images. Extensive experiments are conducted on the BP4D and MMSE databases, with the comparison to the state-of-the-art methods. The experiments show that the multi-modality framework improves the AU detection significantly

Adaptive Multimodal Fusion For Facial Action Units Recognition

Multimodal facial action units (AU) recognition aims to build models that are capable of processing, correlating, and integrating information from multiple modalities (i.e., 2D images from a visual sensor, 3D geometry from 3D imaging, and thermal images from an infrared sensor). Although the multimodal data can provide rich information, there are two challenges that have to be addressed when learning from multimodal data: 1) the model must capture the complex cross-modal interactions in order to utilize the additional and mutual information effectively; 2) the model must be robust enough in the circumstance of unexpected data corruptions during testing, in case of a certain modality missing or being noisy. In this paper, we propose a novel Adaptive Multimodal Fusion method (AMF) for AU detection, which learns to select the most relevant feature representations from different modalities by a re-sampling procedure conditioned on a feature scoring module. The feature scoring module is designed to allow for evaluating the quality of features learned from multiple modalities. As a result, AMF is able to adaptively select more discriminative features, thus increasing the robustness to missing or corrupted modalities. In addition, to alleviate the over-fitting problem and make the model generalize better on the testing data, a cut-switch multimodal data augmentation method is designed, by which a random block is cut and switched across multiple modalities. We have conducted a thorough investigation on two public multimodal AU datasets, BP4D and BP4D+, and the results demonstrate the effectiveness of the proposed method. Ablation studies on various circumstances also show that our method remains robust to missing or noisy modalities during tests

The 2nd 3D Face Alignment In The Wild Challenge (3DFAW-video): Dense Reconstruction From Video

3D face alignment approaches have strong advantages over 2D with respect to representational power and robustness to illumination and pose. Over the past few years, a number of research groups have made rapid advances in dense 3D alignment from 2D video and obtained impressive results. How these various methods compare is relatively unknown. Previous benchmarks addressed sparse 3D alignment and single image 3D reconstruction. No commonly accepted evaluation protocol exists for dense 3D face reconstruction from video with which to compare them. The 2nd 3D Face Alignment in the Wild from Videos (3DFAW-Video) Challenge extends the previous 3DFAW 2016 competition to the estimation of dense 3D facial structure from video. It presented a new large corpora of profile-to-profile face videos recorded under different imaging conditions and annotated with corresponding high-resolution 3D ground truth meshes. In this paper we outline the evaluation protocol, the data used, and the results. 3DFAW-Video is to be held in conjunction with the 2019 International Conference on Computer Vision, in Seoul, Korea

EmbeddingTree: Hierarchical Exploration of Entity Features in Embedding

Embedding learning transforms discrete data entities into continuous numerical representations, encoding features/properties of the entities. Despite the outstanding performance reported from different embedding learning algorithms, few efforts were devoted to structurally interpreting how features are encoded in the learned embedding space. This work proposes EmbeddingTree, a hierarchical embedding exploration algorithm that relates the semantics of entity features with the less-interpretable embedding vectors. An interactive visualization tool is also developed based on EmbeddingTree to explore high-dimensional embeddings. The tool helps users discover nuance features of data entities, perform feature denoising/injecting in embedding training, and generate embeddings for unseen entities. We demonstrate the efficacy of EmbeddingTree and our visualization tool through embeddings generated for industry-scale merchant data and the public 30Music listening/playlists dataset.Comment: 5 pages, 3 figures, accepted by PacificVis 202

TinyKG: Memory-Efficient Training Framework for Knowledge Graph Neural Recommender Systems

There has been an explosion of interest in designing various Knowledge Graph Neural Networks (KGNNs), which achieve state-of-the-art performance and provide great explainability for recommendation. The promising performance is mainly resulting from their capability of capturing high-order proximity messages over the knowledge graphs. However, training KGNNs at scale is challenging due to the high memory usage. In the forward pass, the automatic differentiation engines (\textsl{e.g.}, TensorFlow/PyTorch) generally need to cache all intermediate activation maps in order to compute gradients in the backward pass, which leads to a large GPU memory footprint. Existing work solves this problem by utilizing multi-GPU distributed frameworks. Nonetheless, this poses a practical challenge when seeking to deploy KGNNs in memory-constrained environments, especially for industry-scale graphs. Here we present TinyKG, a memory-efficient GPU-based training framework for KGNNs for the tasks of recommendation. Specifically, TinyKG uses exact activations in the forward pass while storing a quantized version of activations in the GPU buffers. During the backward pass, these low-precision activations are dequantized back to full-precision tensors, in order to compute gradients. To reduce the quantization errors, TinyKG applies a simple yet effective quantization algorithm to compress the activations, which ensures unbiasedness with low variance. As such, the training memory footprint of KGNNs is largely reduced with negligible accuracy loss. To evaluate the performance of our TinyKG, we conduct comprehensive experiments on real-world datasets. We found that our TinyKG with INT2 quantization aggressively reduces the memory footprint of activation maps with $7 \times$, only with $2\%$ loss in accuracy, allowing us to deploy KGNNs on memory-constrained devices