A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation

Body language (BL) refers to the non-verbal communication expressed through physical movements, gestures, facial expressions, and postures. It is a form of communication that conveys information, emotions, attitudes, and intentions without the use of spoken or written words. It plays a crucial role in interpersonal interactions and can complement or even override verbal communication. Deep multi-modal learning techniques have shown promise in understanding and analyzing these diverse aspects of BL. The survey emphasizes their applications to BL generation and recognition. Several common BLs are considered i.e., Sign Language (SL), Cued Speech (CS), Co-speech (CoS), and Talking Head (TH), and we have conducted an analysis and established the connections among these four BL for the first time. Their generation and recognition often involve multi-modal approaches. Benchmark datasets for BL research are well collected and organized, along with the evaluation of SOTA methods on these datasets. The survey highlights challenges such as limited labeled data, multi-modal learning, and the need for domain adaptation to generalize models to unseen speakers or languages. Future research directions are presented, including exploring self-supervised learning techniques, integrating contextual information from other modalities, and exploiting large-scale pre-trained multi-modal models. In summary, this survey paper provides a comprehensive understanding of deep multi-modal learning for various BL generations and recognitions for the first time. By analyzing advancements, challenges, and future directions, it serves as a valuable resource for researchers and practitioners in advancing this field. n addition, we maintain a continuously updated paper list for deep multi-modal learning for BL recognition and generation: https://github.com/wentaoL86/awesome-body-language

Gaussian Differential Privacy on Riemannian Manifolds

We develop an advanced approach for extending Gaussian Differential Privacy (GDP) to general Riemannian manifolds. The concept of GDP stands out as a prominent privacy definition that strongly warrants extension to manifold settings, due to its central limit properties. By harnessing the power of the renowned Bishop-Gromov theorem in geometric analysis, we propose a Riemannian Gaussian distribution that integrates the Riemannian distance, allowing us to achieve GDP in Riemannian manifolds with bounded Ricci curvature. To the best of our knowledge, this work marks the first instance of extending the GDP framework to accommodate general Riemannian manifolds, encompassing curved spaces, and circumventing the reliance on tangent space summaries. We provide a simple algorithm to evaluate the privacy budget $\mu$ on any one-dimensional manifold and introduce a versatile Markov Chain Monte Carlo (MCMC)-based algorithm to calculate $\mu$ on any Riemannian manifold with constant curvature. Through simulations on one of the most prevalent manifolds in statistics, the unit sphere $S^d$, we demonstrate the superior utility of our Riemannian Gaussian mechanism in comparison to the previously proposed Riemannian Laplace mechanism for implementing GDP

Characterising User Transfer Amid Industrial Resource Variation: A Bayesian Nonparametric Approach

In a multitude of industrial fields, a key objective entails optimising resource management whilst satisfying user requirements. Resource management by industrial practitioners can result in a passive transfer of user loads across resource providers, a phenomenon whose accurate characterisation is both challenging and crucial. This research reveals the existence of user clusters, which capture macro-level user transfer patterns amid resource variation. We then propose CLUSTER, an interpretable hierarchical Bayesian nonparametric model capable of automating cluster identification, and thereby predicting user transfer in response to resource variation. Furthermore, CLUSTER facilitates uncertainty quantification for further reliable decision-making. Our method enables privacy protection by functioning independently of personally identifiable information. Experiments with simulated and real-world data from the communications industry reveal a pronounced alignment between prediction results and empirical observations across a spectrum of resource management scenarios. This research establishes a solid groundwork for advancing resource management strategy development

Sampling Through the Lens of Sequential Decision Making

Sampling is ubiquitous in machine learning methodologies. Due to the growth of large datasets and model complexity, we want to learn and adapt the sampling process while training a representation. Towards achieving this grand goal, a variety of sampling techniques have been proposed. However, most of them either use a fixed sampling scheme or adjust the sampling scheme based on simple heuristics. They cannot choose the best sample for model training in different stages. Inspired by "Think, Fast and Slow" (System 1 and System 2) in cognitive science, we propose a reward-guided sampling strategy called Adaptive Sample with Reward (ASR) to tackle this challenge. To the best of our knowledge, this is the first work utilizing reinforcement learning (RL) to address the sampling problem in representation learning. Our approach optimally adjusts the sampling process to achieve optimal performance. We explore geographical relationships among samples by distance-based sampling to maximize overall cumulative reward. We apply ASR to the long-standing sampling problems in similarity-based loss functions. Empirical results in information retrieval and clustering demonstrate ASR's superb performance across different datasets. We also discuss an engrossing phenomenon which we name as "ASR gravity well" in experiments

Aspheric optical surface profiling based on laser scanning and auto-collimation

Nowadays the utilization of aspheric lens has become more and more popular, enabling highly increased degree of freedom for optical design and simultaneously improving the performance of optical system. Hence this also entails a stringent requirement for fast and accurate measuring the shape of these aspheric components. In this paper, the instrument is greatly developed to satisfy the growing need to test axially symmetric aspheric surface, which is implemented by converting the pose of reflective mirror in optical path to the coordinate of reflection point on the surface when laser rapidly scans . At each movement position managed by grating-rule sensor, the rotating angle of reflective mirror is defined using position sensitive detector based on the laser auto-collimating and beam center-fitting principle. Testing a convex and a concave surfaces with highly reproducible results, including coefficient of determination better than 0.999 and RMSE less than =10, validates the feasibility of this method. In comparison to the conventional computer-generated hologram tester or interferometer, the present instrument—essentially builds on the pure geometrical optics technology—is a powerful tool to measure the aspheric surface quickly and accurately with simple structure and algorithm

Relationships of beans intake with chronic kidney disease in rural adults: A large-scale cross-sectional study

Background and aimsDietary factors play an important role in the development of chronic kidney disease (CKD). However, evidence on the relationship of beans consumption with CKD remains limited and inconclusive, especially in the middle-and low-income populations. The current study aimed to investigate the relationships of beans intake with indicators of kidney injury and CKD prevalence in rural adults.MethodsA total of 20,733 rural adults from the Henan Rural Cohort Study in 2018–2022 were included. The total beans intake was collected using a validated food frequency questionnaire. Indicators of kidney injury and CKD was determined by the estimated glomerular filtration rate and the urinary albumin to creatinine ratio. Generalized linear regression and logistic regression models were applied to estimate the relationship of beans intake with continuous and dichotomized indicators of renal function, respectively.ResultsOf the 20,733 participants, 2,676 (12.91%) subjects were identified as CKD patients. After adjusting for potential confounders, participants in the higher quartiles of beans intake had a lower prevalence of CKD (odds ratio and 95% confidence interval, OR (95%CI); Q2: 0.968(0.866–1.082); Q3: 0.836(0.744–0.939); Q4: 0.854(0.751–0.970)) and albuminuria (Q2: 0.982(0.875–1.102); Q3: 0.846(0.750–0.954); Q4: 0.852 (0.746–0.973)), compared with the Q1. Per 50 g/day increment in beans intake was significantly associated with a 5 and 4% decreased prevalence of albuminuria and CKD, respectively. These inverse relationships were also significant in the subgroups of men, elder, and high-income participants (p < 0.05).ConclusionDietary beans intake was inversely associated with the prevalence of albuminuria and CKD in rural adults, suggesting that promoting soy food intake might help reduce the occurrence of CKD in rural adults