DanceMeld: Unraveling Dance Phrases with Hierarchical Latent Codes for Music-to-Dance Synthesis

In the realm of 3D digital human applications, music-to-dance presents a challenging task. Given the one-to-many relationship between music and dance, previous methods have been limited in their approach, relying solely on matching and generating corresponding dance movements based on music rhythm. In the professional field of choreography, a dance phrase consists of several dance poses and dance movements. Dance poses composed of a series of basic meaningful body postures, while dance movements can reflect dynamic changes such as the rhythm, melody, and style of dance. Taking inspiration from these concepts, we introduce an innovative dance generation pipeline called DanceMeld, which comprising two stages, i.e., the dance decouple stage and the dance generation stage. In the decouple stage, a hierarchical VQ-VAE is used to disentangle dance poses and dance movements in different feature space levels, where the bottom code represents dance poses, and the top code represents dance movements. In the generation stage, we utilize a diffusion model as a prior to model the distribution and generate latent codes conditioned on music features. We have experimentally demonstrated the representational capabilities of top code and bottom code, enabling the explicit decoupling expression of dance poses and dance movements. This disentanglement not only provides control over motion details, styles, and rhythm but also facilitates applications such as dance style transfer and dance unit editing. Our approach has undergone qualitative and quantitative experiments on the AIST++ dataset, demonstrating its superiority over other methods.Comment: 10 pages, 8 figure

Cloth2Tex: A Customized Cloth Texture Generation Pipeline for 3D Virtual Try-On

Fabricating and designing 3D garments has become extremely demanding with the increasing need for synthesizing realistic dressed persons for a variety of applications, e.g. 3D virtual try-on, digitalization of 2D clothes into 3D apparel, and cloth animation. It thus necessitates a simple and straightforward pipeline to obtain high-quality texture from simple input, such as 2D reference images. Since traditional warping-based texture generation methods require a significant number of control points to be manually selected for each type of garment, which can be a time-consuming and tedious process. We propose a novel method, called Cloth2Tex, which eliminates the human burden in this process. Cloth2Tex is a self-supervised method that generates texture maps with reasonable layout and structural consistency. Another key feature of Cloth2Tex is that it can be used to support high-fidelity texture inpainting. This is done by combining Cloth2Tex with a prevailing latent diffusion model. We evaluate our approach both qualitatively and quantitatively and demonstrate that Cloth2Tex can generate high-quality texture maps and achieve the best visual effects in comparison to other methods. Project page: tomguluson92.github.io/projects/cloth2tex/Comment: 15 pages, 15 figure

ScratchDet: Training Single-Shot Object Detectors from Scratch

Current state-of-the-art object objectors are fine-tuned from the off-the-shelf networks pretrained on large-scale classification dataset ImageNet, which incurs some additional problems: 1) The classification and detection have different degrees of sensitivity to translation, resulting in the learning objective bias; 2) The architecture is limited by the classification network, leading to the inconvenience of modification. To cope with these problems, training detectors from scratch is a feasible solution. However, the detectors trained from scratch generally perform worse than the pretrained ones, even suffer from the convergence issue in training. In this paper, we explore to train object detectors from scratch robustly. By analysing the previous work on optimization landscape, we find that one of the overlooked points in current trained-from-scratch detector is the BatchNorm. Resorting to the stable and predictable gradient brought by BatchNorm, detectors can be trained from scratch stably while keeping the favourable performance independent to the network architecture. Taking this advantage, we are able to explore various types of networks for object detection, without suffering from the poor convergence. By extensive experiments and analyses on downsampling factor, we propose the Root-ResNet backbone network, which makes full use of the information from original images. Our ScratchDet achieves the state-of-the-art accuracy on PASCAL VOC 2007, 2012 and MS COCO among all the train-from-scratch detectors and even performs better than several one-stage pretrained methods. Codes will be made publicly available at https://github.com/KimSoybean/ScratchDet.Comment: CVPR2019 Oral Presentation. Camera Ready Versio

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

Audio-driven talking head generation has drawn much attention in recent years, and many efforts have been made in lip-sync, expressive facial expressions, natural head pose generation, and high video quality. However, no model has yet led or tied on all these metrics due to the one-to-many mapping between audio and motion. In this paper, we propose VividTalk, a two-stage generic framework that supports generating high-visual quality talking head videos with all the above properties. Specifically, in the first stage, we map the audio to mesh by learning two motions, including non-rigid expression motion and rigid head motion. For expression motion, both blendshape and vertex are adopted as the intermediate representation to maximize the representation ability of the model. For natural head motion, a novel learnable head pose codebook with a two-phase training mechanism is proposed. In the second stage, we proposed a dual branch motion-vae and a generator to transform the meshes into dense motion and synthesize high-quality video frame-by-frame. Extensive experiments show that the proposed VividTalk can generate high-visual quality talking head videos with lip-sync and realistic enhanced by a large margin, and outperforms previous state-of-the-art works in objective and subjective comparisons.Comment: 10 pages, 8 figure

One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field

Talking head generation aims to generate faces that maintain the identity information of the source image and imitate the motion of the driving image. Most pioneering methods rely primarily on 2D representations and thus will inevitably suffer from face distortion when large head rotations are encountered. Recent works instead employ explicit 3D structural representations or implicit neural rendering to improve performance under large pose changes. Nevertheless, the fidelity of identity and expression is not so desirable, especially for novel-view synthesis. In this paper, we propose HiDe-NeRF, which achieves high-fidelity and free-view talking-head synthesis. Drawing on the recently proposed Deformable Neural Radiance Fields, HiDe-NeRF represents the 3D dynamic scene into a canonical appearance field and an implicit deformation field, where the former comprises the canonical source face and the latter models the driving pose and expression. In particular, we improve fidelity from two aspects: (i) to enhance identity expressiveness, we design a generalized appearance module that leverages multi-scale volume features to preserve face shape and details; (ii) to improve expression preciseness, we propose a lightweight deformation module that explicitly decouples the pose and expression to enable precise expression modeling. Extensive experiments demonstrate that our proposed approach can generate better results than previous works. Project page: https://www.waytron.net/hidenerf/Comment: Accepted by CVPR 202

Salidroside Inhibits HMGB1 Acetylation and Release through Upregulation of SirT1 during Inflammation

HMGB1, a highly conserved nonhistone DNA-binding protein, plays an important role in inflammatory diseases. Once released to the extracellular space, HMGB1 acts as a proinflammatory cytokine that triggers inflammatory reaction. Our previous study showed that salidroside exerts anti-inflammatory effect via inhibiting the JAK2-STAT3 signalling pathway. However, whether salidroside inhibits the release of HMGB1 is still unclear. In this study, we aim to study the effects of salidroside on HMGB1 release and then investigate the potential molecular mechanisms. In an experimental rat model of sepsis caused by CLP, salidroside administration significantly attenuated lung injury and reduced the serum HMGB1 level. In RAW264.7 cells, we investigated the effects of salidroside on LPS-induced HMGB1 release and then explored the underlying molecular mechanisms. We found that salidroside significantly inhibited LPS-induced HMGB1 release, and the inhibitory effect was correlated with the HMGB1 acetylation levels. Mechanismly, salidroside inhibits HMGB1 acetylation through the AMPK-SirT1 pathway. In addition, SirT1 overexpression attenuated LPS-induced HMGB1 acetylation and nucleocytoplasmic translocation. Furthermore, in SirT1 shRNA plasmid-transfected cells, salidroside treatment enhanced SirT1 expression and reduced LPS-activated HMGB1 acetylation and nucleocytoplasmic translocation. Collectively, these results demonstrated that salidroside might reduce HMGB1 release through the AMPK-SirT1 signalling pathway and suppress HMGB1 acetylation and nucleocytoplasmic translocation

Fe-Chlorophyllin Promotes the Growth of Wheat Roots Associated with Nitric Oxide Generation

Effects of Fe-chlorophyllin on the growth of wheat root were investigated in this study. We found that Fe-chlorophyllin can promote root growth. The production of nitric oxide in wheat root was detected using DAF-2DA fluorescent emission. The intensity of fluorescent in the presence of 0.1 mg/L Fe-chlorophyllin was near to that observed with the positive control of sodium nitroprusside (SNP), the nitric oxide donor. IAA oxidase activity decreased with all treatments of Fe-chlorophyllin from 0.01 to 10 mg/L. At the relatively lower Fe-chlorophyllin concentration of 0.1 mg/L, the activity of IAA oxidase displayed a remarkable decrease, being 40.1% lower than the control. Meanwhile, Fe-chlorophyllin treatment could increase the activities of reactive oxygen scavenging enzymes, such as superoxide dismutase (SOD) and peroxidase (POD), as determined using non-denaturing polyacrylamide gel electrophoresis. These results indicate that Fe-chlorophyllin contributes to the growth of wheat root associated with nitric oxide generation