16 research outputs found
3D Representation Learning for Shape Reconstruction and Understanding
The real world we are living in is inherently composed of multiple 3D objects. However, most of the existing works in computer vision traditionally either focus on images or videos where the 3D information inevitably gets lost due to the camera projection. Traditional methods typically rely on hand-crafted algorithms and features with many constraints and geometric priors to understand the real world. However, following the trend of deep learning, there has been an exponential growth in the number of research works based on deep neural networks to learn 3D representations for complex shapes and scenes, which lead to many cutting-edged applications in augmented reality (AR), virtual reality (VR) and robotics as one of the most important directions for computer vision and computer graphics.
This thesis aims to build an intelligent system with dynamic 3D representations that can change over time to understand and recover the real world with semantic, instance and geometric information and eventually bridge the gap between the real world and the digital world. As the first step towards the challenges, this thesis explores both explicit representations and implicit representations by explicitly addressing the existing open problems in these areas. This thesis starts from neural implicit representation learning on 3D scene representation learning and understanding and moves to a parametric model based explicit 3D reconstruction method. Extensive experimentation over various benchmarks on various domains demonstrates the superiority of our method against previous state-of-the-art approaches, enabling many applications in the real world. Based on the proposed methods and current observations of open problems, this thesis finally presents a comprehensive conclusion with potential future research directions
deformation in SCFTs and integrable supersymmetric theories
We calculate the -multiplets for two-dimensional Euclidean
and superconformal field theories
under the deformation at leading order of perturbation theory
in the deformation coupling. Then, from these deformed
multiplets, we calculate two- and three-point correlators. We show the
chiral ring's elements do not flow under the
deformation. For the case of , we show the
twisted chiral ring and chiral ring cease to exist simultaneously. Specializing
to integrable supersymmetric seed theories, such as
Landau-Ginzburg models, we use the thermodynamic Bethe ansatz to study the
S-matrices and ground state energies. From both an S-matrix perspective and
Melzer's folding prescription, we show that the deformed ground state energy
obeys the inviscid Burgers' equation. Finally, we show that several indices
independent of -term perturbations including the Witten index,
Cecotti-Fendley-Intriligator-Vafa index and elliptic genus do not flow under
the deformation.Comment: 46 page
SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark
In this paper, we present SignAvatars, the first large-scale multi-prompt 3D
sign language (SL) motion dataset designed to bridge the communication gap for
hearing-impaired individuals. While there has been an exponentially growing
number of research regarding digital communication, the majority of existing
communication technologies primarily cater to spoken or written languages,
instead of SL, the essential communication method for hearing-impaired
communities. Existing SL datasets, dictionaries, and sign language production
(SLP) methods are typically limited to 2D as the annotating 3D models and
avatars for SL is usually an entirely manual and labor-intensive process
conducted by SL experts, often resulting in unnatural avatars. In response to
these challenges, we compile and curate the SignAvatars dataset, which
comprises 70,000 videos from 153 signers, totaling 8.34 million frames,
covering both isolated signs and continuous, co-articulated signs, with
multiple prompts including HamNoSys, spoken language, and words. To yield 3D
holistic annotations, including meshes and biomechanically-valid poses of body,
hands, and face, as well as 2D and 3D keypoints, we introduce an automated
annotation pipeline operating on our large corpus of SL videos. SignAvatars
facilitates various tasks such as 3D sign language recognition (SLR) and the
novel 3D SL production (SLP) from diverse inputs like text scripts, individual
words, and HamNoSys notation. Hence, to evaluate the potential of SignAvatars,
we further propose a unified benchmark of 3D SL holistic motion production. We
believe that this work is a significant step forward towards bringing the
digital world to the hearing-impaired communities. Our project page is at
https://signavatars.github.io/Comment: 9 pages; Project page available at https://signavatars.github.io
Decomposed Human Motion Prior for Video Pose Estimation via Adversarial Training
Estimating human pose from video is a task that receives considerable
attention due to its applicability in numerous 3D fields. The complexity of
prior knowledge of human body movements poses a challenge to neural network
models in the task of regressing keypoints. In this paper, we address this
problem by incorporating motion prior in an adversarial way. Different from
previous methods, we propose to decompose holistic motion prior to joint motion
prior, making it easier for neural networks to learn from prior knowledge
thereby boosting the performance on the task. We also utilize a novel
regularization loss to balance accuracy and smoothness introduced by motion
prior. Our method achieves 9\% lower PA-MPJPE and 29\% lower acceleration error
than previous methods tested on 3DPW. The estimator proves its robustness by
achieving impressive performance on in-the-wild dataset
U3DS: Unsupervised 3D Semantic Scene Segmentation
Contemporary point cloud segmentation approaches largely rely on richly
annotated 3D training data. However, it is both time-consuming and challenging
to obtain consistently accurate annotations for such 3D scene data. Moreover,
there is still a lack of investigation into fully unsupervised scene
segmentation for point clouds, especially for holistic 3D scenes. This paper
presents U3DS, as a step towards completely unsupervised point cloud
segmentation for any holistic 3D scenes. To achieve this, U3DS leverages a
generalized unsupervised segmentation method for both object and background
across both indoor and outdoor static 3D point clouds with no requirement for
model pre-training, by leveraging only the inherent information of the point
cloud to achieve full 3D scene segmentation. The initial step of our proposed
approach involves generating superpoints based on the geometric characteristics
of each scene. Subsequently, it undergoes a learning process through a spatial
clustering-based methodology, followed by iterative training using
pseudo-labels generated in accordance with the cluster centroids. Moreover, by
leveraging the invariance and equivariance of the volumetric representations,
we apply the geometric transformation on voxelized features to provide two sets
of descriptors for robust representation learning. Finally, our evaluation
provides state-of-the-art results on the ScanNet and SemanticKITTI, and
competitive results on the S3DIS, benchmark datasets.Comment: 10 Pages, 4 figures, accepted to IEEE/CVF Winter Conference on
Applications of Computer Vision (WACV) 202
The Hitchhiker's Guide to 4d Superconformal Field Theories
Superconformal field theory with supersymmetry in four
dimensional spacetime provides a prime playground to study strongly coupled
phenomena in quantum field theory. Its rigid structure ensures valuable
analytic control over non-perturbative effects, yet the theory is still
flexible enough to incorporate a large landscape of quantum systems. Here we
aim to offer a guidebook to fundamental features of the 4d
superconformal field theories and basic tools to construct them in
string/M-/F-theory. The content is based on a series of lectures at the Quantum
Field Theories and Geometry School
(https://sites.google.com/view/qftandgeometrysummerschool/home) in July 2020.Comment: v3: Improved discussion, fixed typos, added references v2: Typos
fixed and added references. v1: 96 pages. Based on a series of lectures at
the Quantum Field Theories and Geometry School in July 202
Recommended from our members
TT¯ deformation in SCFTs and integrable supersymmetric theories
We calculate the S-multiplets for two-dimensional Euclidean N = (0, 2) and N = (2, 2) superconformal field theories under the TT¯ deformation at leading order of perturbation theory in the deformation coupling. Then, from these N = (0, 2) deformed multiplets, we calculate two- and three-point correlators. We show the N = (0, 2) chiral ring’s elements do not flow under the TT¯ deformation. Specializing to integrable supersymmetric seed theories, such as N = (2, 2) Landau-Ginzburg models, we use the thermodynamic Bethe ansatz to study the S-matrices and ground state energies. From both an S-matrix perspective and Melzer’s folding prescription, we show that the deformed ground state energy obeys the inviscid Burgers’ equation. Finally, we show that several indices independent of D-term perturbations including the Witten index, Cecotti-Fendley-Intriligator-Vafa index and elliptic genus do not flow under the TT¯ deformation
Recommended from our members
TT¯-deformed free energy of the Airy model
Sharpening the correspondence of Jackiw-Teitelboim (JT) gravity and its dual matrix model description at a finite radial cutoff λ through the TT¯ deformation is of interest. To proceed, we simplify the problem by considering the Airy model and deform Airy correlators in the same way as in TT¯ -deformed JT gravity. We use those correlators to compute the annealed and quenched free energies for both λ > 0 and λ < 0 from an integral representation of the replica trick. At the leading order in λ and low temperatures, we confirm that the genus-zero quenched free energy monotonically decreases as a function of temperature when perturbation theory is valid. We then study the all-genus quenched free energy at low temperatures, where we discover and discuss subtleties due to non-perturbative effects in the Airy model, as well as the contributions from the non-perturbative branch under the TT¯ deformation
Recommended from our members
in JT Gravity and BF Gauge Theory
JT gravity has a first-order formulation as a two-dimensional BF
theory, which can be viewed as the dimensional reduction of the
Chern-Simons description of 3d3d
gravity. We consider {T\overbar{T}}TT¯-type
deformations of the (0+1)(0+1)-dimensional
dual to this 2d2d
BF theory and interpret the deformation as a modification of the BF
theory boundary conditions. The fundamental observables in this deformed
BF theory, and in its 3d3d
Chern-Simons lift, are Wilson lines and loops. In the
3d3d
Chern-Simons setting, we study modifications to correlators involving
boundary-anchored Wilson lines which are induced by a
{T\overbar{T}}TT¯
deformation on the 2d2d
boundary; results are presented at both the classical level (using
modified boundary conditions) and the quantum-mechanical level (using
conformal perturbation theory). Finally, we calculate the analogous
deformed Wilson line correlators in 2d2d
BF theory below the Hagedorn temperature where the principal series
dominates over the discrete series