Deep Generative Models on 3D Representations: A Survey

Generative models, as an important family of statistical modeling, target learning the observed data distribution via generating new instances. Along with the rise of neural networks, deep generative models, such as variational autoencoders (VAEs) and generative adversarial network (GANs), have made tremendous progress in 2D image synthesis. Recently, researchers switch their attentions from the 2D space to the 3D space considering that 3D data better aligns with our physical world and hence enjoys great potential in practice. However, unlike a 2D image, which owns an efficient representation (i.e., pixel grid) by nature, representing 3D data could face far more challenges. Concretely, we would expect an ideal 3D representation to be capable enough to model shapes and appearances in details, and to be highly efficient so as to model high-resolution data with fast speed and low memory cost. However, existing 3D representations, such as point clouds, meshes, and recent neural fields, usually fail to meet the above requirements simultaneously. In this survey, we make a thorough review of the development of 3D generation, including 3D shape generation and 3D-aware image synthesis, from the perspectives of both algorithms and more importantly representations. We hope that our discussion could help the community track the evolution of this field and further spark some innovative ideas to advance this challenging task

Learning Human Mesh Recovery in 3D Scenes

We present a novel method for recovering the absolute pose and shape of a human in a pre-scanned scene given a single image. Unlike previous methods that perform sceneaware mesh optimization, we propose to first estimate absolute position and dense scene contacts with a sparse 3D CNN, and later enhance a pretrained human mesh recovery network by cross-attention with the derived 3D scene cues. Joint learning on images and scene geometry enables our method to reduce the ambiguity caused by depth and occlusion, resulting in more reasonable global postures and contacts. Encoding scene-aware cues in the network also allows the proposed method to be optimization-free, and opens up the opportunity for real-time applications. The experiments show that the proposed network is capable of recovering accurate and physically-plausible meshes by a single forward pass and outperforms state-of-the-art methods in terms of both accuracy and speed.Comment: Accepted to CVPR 2023. Project page: https://zju3dv.github.io/sahmr

4K4D: Real-Time 4D View Synthesis at 4K Resolution

This paper targets high-fidelity and real-time view synthesis of dynamic 3D scenes at 4K resolution. Recently, some methods on dynamic view synthesis have shown impressive rendering quality. However, their speed is still limited when rendering high-resolution images. To overcome this problem, we propose 4K4D, a 4D point cloud representation that supports hardware rasterization and enables unprecedented rendering speed. Our representation is built on a 4D feature grid so that the points are naturally regularized and can be robustly optimized. In addition, we design a novel hybrid appearance model that significantly boosts the rendering quality while preserving efficiency. Moreover, we develop a differentiable depth peeling algorithm to effectively learn the proposed model from RGB videos. Experiments show that our representation can be rendered at over 400 FPS on the DNA-Rendering dataset at 1080p resolution and 80 FPS on the ENeRF-Outdoor dataset at 4K resolution using an RTX 4090 GPU, which is 30x faster than previous methods and achieves the state-of-the-art rendering quality. Our project page is available at https://zju3dv.github.io/4k4d/.Comment: Project Page: https://zju3dv.github.io/4k4

Topological Magnetoresistance of Magnetic Skyrmionic Bubbles

Magnetic skyrmions offer promising prospects for constructing future energy-efficient and high-density information technology, leading to extensive explorations of new skyrmionic materials recently. The topological Hall effect has been widely adopted as a distinctive marker of skyrmion emergence. Alternately, here we propose a novel signature of skyrmion state by quantitatively investigating the magnetoresistance (MR) induced by skyrmionic bubbles in CeMn2Ge2. An intriguing finding was revealed: the anomalous MR measured at different temperatures can be normalized into a single curve, regardless of sample thickness. This behavior can be accurately reproduced by the recent chiral spin textures MR model. Further analysis of the MR anomaly allowed us to quantitatively examine the effective magnetic fields of various scattering channels. Remarkably, the analyses, combined with the Lorentz transmission electronic microscopy results, indicate that the in-plane scattering channel with triplet exchange interactions predominantly governs the magnetotransport in the Bloch-type skyrmionic bubble state. Our results not only provide insights into the quantum correction on MR induced by skyrmionic bubble phase, but also present an electrical probing method for studying chiral spin texture formation, evolution and their topological properties, which opens up exciting possibilities for identifying new skyrmionic materials and advancing the methodology for studying chiral spin textures.Comment: 17 pages,5 figures,submitte

Somatic SF3B1 hotspot mutation in prolactinomas.

The genetic basis and corresponding clinical relevance of prolactinomas remain poorly understood. Here, we perform whole genome sequencing (WGS) on 21 patients with prolactinomas to detect somatic mutations and then validate the mutations with digital polymerase chain reaction (PCR) analysis of tissue samples from 227 prolactinomas. We identify the same hotspot somatic mutation in splicing factor 3 subunit B1 (SF3B1R625H) in 19.8% of prolactinomas. These patients with mutant prolactinomas display higher prolactin (PRL) levels (p = 0.02) and shorter progression-free survival (PFS) (p = 0.02) compared to patients without the mutation. Moreover, we identify that the SF3B1R625H mutation causes aberrant splicing of estrogen related receptor gamma (ESRRG), which results in stronger binding of pituitary-specific positive transcription factor 1 (Pit-1), leading to excessive PRL secretion. Thus our study validates an important mutation and elucidates a potential mechanism underlying the pathogenesis of prolactinomas that may lead to the development of targeted therapeutics

Chromatic Dispersion Equalization FIR Digital Filter for Coherent Receiver

Chromatic dispersion equalization (CDE) in coherent optical communication systems is extremely critical for subsequent digital signal processing (such as frequency offset estimation and carrier phase recovery). Various methods mentioned in the published literature are not satisfactory when the signal bandwidth is limited. This paper proposes a way of using singular value decomposition least square (SVDLS) to obtain the optimal tap weight of the CDE filter and a method to introduce the adaptive mutation particle swarm optimizer (AMPSO) algorithm into the CDE. We show that the two proposed approaches are based on the best approximation of the frequency domain response of the designed and ideal CDE filter. Compared with the traditional CDE method, which needs to be implemented in the full frequency band, the two methods can be implemented in the narrow frequency band. The simulation shows that the effective bandwidth of the baseband signal is limited by squared-root-raised-cosine (SRRC) pulse shaping with a roll-off factor of 0.25 in different modulation formats (DP-QPSK, DP-16 QAM, DP-64 QAM) when the number of taps of the filter is 131, which is 37.5% less than the full frequency band. The designed filter is superior to the existing filter in terms of filtering effect and implementation complexity

Low-Complexity Chromatic Dispersion Equalization FIR Digital Filter for Coherent Receiver

This paper proposes a novel and efficient low-complexity chromatic dispersion equalizer (CDE) based on finite impulse response (FIR) filter architecture for polarization-multiplexed coherent optical communication systems. The FIR filter coefficients are optimized by weights to reduce the energy leakage caused by the truncation effect, and then quantization is used uniformly to reduce the number of real number additions and real number multiplications by utilizing the diversity of the quantized coefficients. Using Optisystem 15 to build a coherent optical communication system for simulation and experimental demonstration, the results show that after the filter coefficients are optimized by weights. Compared with the time-domain chromatic dispersion equalizer (TD-CDE), the proposed design has a lower bit error rate (BER) and better equalization effect. When the transmission distance is 4000 km and the system quantization stages M = 16, the multiplication operation and addition operations reduce computing resources by 99% and 43%, and the BER only increases by 5%. Compared with frequency-domain chromatic dispersion equalizer (FD-CDE), widely used in long-distance communication, the multiplication operation reduces computing resources by 30%. The proposed method provides a new idea for high-performance CDE in long-distance coherent optical communication systems