Metric representations for shape analysis and synthesis

2D and 3D geometric shapes are ubiquitous in computer graphics, computer animation, and computer-aided design and manufacturing. Two of the fundamental research challenges that underline these applications are the analysis and synthesis of shapes, with the former aiming to extract semantically meaningful knowledge of shapes and the latter focusing on generating plausible-looking shapes based on user inputs. Traditionally, shape analysis and synthesis are based on representations such as meshes, parameterisations, and Laplacians, which lead to mostly hand-crafted computation rules that are either suboptimal or treat related tasks separately. In this work, we propose to represent a 2D/3D shape as a square symmetric matrix that correlates every pair of geometric points on the shape, which allows us to formulate shape analysis and synthesis problems as principled optimisation problems that can be globally optimised. To demonstrate the usefulness of our new metric representation for shape analysis, we first address 3D mesh saliency detection by representing a shape as a pairwise feature distance matrix, whose principal eigenvector is experimentally shown to outperform the traditional saliency detection rules for capturing ground truth saliency annotations. Following this work, we then unify saliency detection and nonrigid shape matching via a jointly learned metric representation, which is shown to improve the accuracy of both tasks on the existing saliency detection and shape matching benchmarks. To also demonstrate the usefulness of our metric representation for shape synthesis, we address 2D facial shape beautification in images by representing a facial shape as the orthogonal projection matrix onto 2D facial landmarks, which is shown to improve the attractiveness of both frontal-neutral and non-frontal-non-neutral faces in the user studies. Finally, we show that adversarially learning the distributions of human shapes and poses in a hidden space produces higher quality human samples than in the geometry space. Together, these results show that our metric representation benefits both the analysis and synthesis of shapes, with the potential of unifying more diverse tasks such as part segmentation and labelling in the future work

DSPP: Deep Shape and Pose Priors of Humans

The prior knowledge of real human body shapes and poses is fundamentalin computer games and animation (e.g. performance capture). Linear subspaces such as the popular SMPL model have a limited capacity to represent the large geometric variations of human shapes and poses. What is worse is that random sampling from them often produces non-realistic humans because the distribution of real humans is more likely to concentrate on a non-linear manifold instead of the full subspace. Towards this problem, we propose to learn human shape and pose manifolds using a more powerful deep generator network, which is trained to produce samples that cannot be distinguished from real humans by a deep discriminator network. In contrast to previous work that learn both the generator and discriminator in the original geometry spaces, we learn them in the more representative latent spaces discovered by a shape and a pose auto-encoder network respectively. Random sampling from our priors produces higher-quality human shapes and poses. The capacity of our priors is best applied to applications such as virtual human synthesis in games

Efficient Spatial Reasoning for Human Pose Estimation

Human pose estimation from single images has made significant progress in the past but still faces fundamental challenges from the occlusion and overlapping of joints in many cases. This is partly due to the limitation of the traditional paradigm for this problem, which attempts to locate human body joints solely and as a result can fail to resolve the spatial connections among joints that are critical for the identification of the whole pose. To overcome this shortcoming, we propose to explicitly incorporate spatial reasoning into pose estimation by formulating it as a structured graph learning problem, in which each image pixel is a candidate graph node with every two nodes connected via an edge that captures their affinity. The advantage of this representation is that it allows us to learn feature embeddings for both the nodes and edges, thereby providing a sufficient capacity to delineate correct human body joints and their connecting bones. To facilitate efficient learning and inference, we exploit self-attention transformer architectures that fuse node and edge learning pathways, which can save parameter numbers and permit fast computation. Experiments on the popular MS-COCO Human pose estimation benchmark show that our method outperforms representative methods

Facial reshaping operator for controllable face beautification

Posting attractive facial photos is part of everyday life in the social media era. Motivated by the demand, we propose a lightweight method to automatically and efficiently beautify the shapes of both portrait and non-portrait faces in photos, while allowing users to customize the beautification of individual facial features. Previous methods focus on the beautification of mostly frontal and neutral faces, without incorporating user controllability in the beautification process. To address these restrictions, we propose the Facial Reshaping Operator representation, which is affine-invariant, captures the pairwise geometric configuration of facial landmarks, and allows for efficient face beautification with the user-specified weights of individual facial parts. We also propose an unsupervised beautification method in the operator space of faces, where an input face is iteratively pulled towards a local nearby density mode with improved attractiveness. Our method distinguishes itself from the commercial beautification tools in that it mildly enhances facial shapes without altering makeups or complexions, which complements these tools that lack fine-grained control on the attractiveness of facial shapes for users. The experimental results show that our method improves facial shape attractiveness for a large range of poses and expressions, demonstrating the potential of applicability to photos seen on the social media such as Facebook and Instagram everyday

MetaMHC: a meta approach to predict peptides binding to MHC molecules

As antigenic peptides binding to major histocompatibility complex (MHC) molecules is the prerequisite of cellular immune responses, an accurate computational predictor will be of great benefit to biologists and immunologists for understanding the underlying mechanism of immune recognition as well as facilitating the process of epitope mapping and vaccine design. Although various computational approaches have been developed, recent experimental results on benchmark data sets show that the development of improved predictors is needed, especially for MHC Class II peptide binding. To make the most of current methods and achieve a higher predictive performance, we developed a new web server, MetaMHC, to integrate the outputs of leading predictors by several popular ensemble strategies. MetaMHC consists of two components: MetaMHCI and MetaMHCII for MHC Class I peptide and MHC Class II peptide binding predictions, respectively. Experimental results by both cross-validation and using an independent data set show that the ensemble approaches outperform individual predictors, being statistically significant. MetaMHC is freely available at http://www.biokdd.fudan.edu.cn/Service/MetaMHC.html

Real-time Monitoring for the Next Core-Collapse Supernova in JUNO

Core-collapse supernova (CCSN) is one of the most energetic astrophysical events in the Universe. The early and prompt detection of neutrinos before (pre-SN) and during the SN burst is a unique opportunity to realize the multi-messenger observation of the CCSN events. In this work, we describe the monitoring concept and present the sensitivity of the system to the pre-SN and SN neutrinos at the Jiangmen Underground Neutrino Observatory (JUNO), which is a 20 kton liquid scintillator detector under construction in South China. The real-time monitoring system is designed with both the prompt monitors on the electronic board and online monitors at the data acquisition stage, in order to ensure both the alert speed and alert coverage of progenitor stars. By assuming a false alert rate of 1 per year, this monitoring system can be sensitive to the pre-SN neutrinos up to the distance of about 1.6 (0.9) kpc and SN neutrinos up to about 370 (360) kpc for a progenitor mass of 30$M_{\odot}$ for the case of normal (inverted) mass ordering. The pointing ability of the CCSN is evaluated by using the accumulated event anisotropy of the inverse beta decay interactions from pre-SN or SN neutrinos, which, along with the early alert, can play important roles for the followup multi-messenger observations of the next Galactic or nearby extragalactic CCSN.Comment: 24 pages, 9 figure

Detection of the Diffuse Supernova Neutrino Background with JUNO

As an underground multi-purpose neutrino detector with 20 kton liquid scintillator, Jiangmen Underground Neutrino Observatory (JUNO) is competitive with and complementary to the water-Cherenkov detectors on the search for the diffuse supernova neutrino background (DSNB). Typical supernova models predict 2-4 events per year within the optimal observation window in the JUNO detector. The dominant background is from the neutral-current (NC) interaction of atmospheric neutrinos with 12C nuclei, which surpasses the DSNB by more than one order of magnitude. We evaluated the systematic uncertainty of NC background from the spread of a variety of data-driven models and further developed a method to determine NC background within 15\% with {\it{in}} {\it{situ}} measurements after ten years of running. Besides, the NC-like backgrounds can be effectively suppressed by the intrinsic pulse-shape discrimination (PSD) capabilities of liquid scintillators. In this talk, I will present in detail the improvements on NC background uncertainty evaluation, PSD discriminator development, and finally, the potential of DSNB sensitivity in JUNO

Potential of Core-Collapse Supernova Neutrino Detection at JUNO

JUNO is an underground neutrino observatory under construction in Jiangmen, China. It uses 20kton liquid scintillator as target, which enables it to detect supernova burst neutrinos of a large statistics for the next galactic core-collapse supernova (CCSN) and also pre-supernova neutrinos from the nearby CCSN progenitors. All flavors of supernova burst neutrinos can be detected by JUNO via several interaction channels, including inverse beta decay, elastic scattering on electron and proton, interactions on C12 nuclei, etc. This retains the possibility for JUNO to reconstruct the energy spectra of supernova burst neutrinos of all flavors. The real time monitoring systems based on FPGA and DAQ are under development in JUNO, which allow prompt alert and trigger-less data acquisition of CCSN events. The alert performances of both monitoring systems have been thoroughly studied using simulations. Moreover, once a CCSN is tagged, the system can give fast characterizations, such as directionality and light curve

A Dual-Stream Recurrent Neural Network for Student Feedback Prediction using Kinect

Convenience internet access and ubiquitous computing have opened up new avenues for learning and teaching. They are now no longer confined to the classroom walls, but are available to anyone connected to the internet. E-learning has opened massive opportunities for learners who otherwise would have been constrained due to geographical distances, time and/or cost factors. It has revolutionized the learning methods and represents a paradigm shift from traditional learning methods. However, despite all its advantages, e-learning is not without its own shortcomings. Understanding the effectiveness of a teaching strategy through learner feedback has been a key performance measure and decision making criteria to fine tune the teaching strategy. However, traditional methods of collecting learner feedback are inadequate in a geographically distributed, virtual setup of the e-learning environment. Innovative and novel learner feedback collection mechanism is hence the need of the hour. In this work, we design and develop a deep learning based student feedback prediction system by recognizing the subtle facial motions during a student’s learning activity. This allows the system to infer the needs of the learners as if it is a real human teacher in order to provide the appropriate feedback. We propose a recurrent convolutional neural network structure to understand the color and depth streams of video taken by an RGB-D camera. Experimental results have shown that our system achieve high accuracy in estimating the feedback labels. While we demonstrate the proposed framework in an e-learning setup, it can be adapted to other applications such as in-house patient monitoring and rehabilitation training