132 research outputs found
Differentiable surface splatting for point-based geometry processing
We propose Differentiable Surface Splatting (DSS), a high-fidelity differentiable renderer for point clouds. Gradients for point locations and normals are carefully designed to handle discontinuities of the rendering function. Regularization terms are introduced to ensure uniform distribution of the points on the underlying surface. We demonstrate applications of DSS to inverse rendering for geometry synthesis and denoising, where large scale topological changes, as well as small scale detail modifications, are accurately and robustly handled without requiring explicit connectivity, outperforming state-of-the-art techniques. The data and code are at https://github.com/yifita/DSS.</jats:p
Drivable 3D Gaussian Avatars
We present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable
model for human bodies rendered with Gaussian splats. Current photorealistic
drivable avatars require either accurate 3D registrations during training,
dense input images during testing, or both. The ones based on neural radiance
fields also tend to be prohibitively slow for telepresence applications. This
work uses the recently presented 3D Gaussian Splatting (3DGS) technique to
render realistic humans at real-time framerates, using dense calibrated
multi-view videos as input. To deform those primitives, we depart from the
commonly used point deformation method of linear blend skinning (LBS) and use a
classic volumetric deformation method: cage deformations. Given their smaller
size, we drive these deformations with joint angles and keypoints, which are
more suitable for communication applications. Our experiments on nine subjects
with varied body shapes, clothes, and motions obtain higher-quality results
than state-of-the-art methods when using the same training and test data.Comment: Website: https://zielon.github.io/d3ga
3D Gaussian Splatting for Real-Time Radiance Field Rendering
Radiance Field methods have recently revolutionized novel-view synthesis of
scenes captured with multiple photos or videos. However, achieving high visual
quality still requires neural networks that are costly to train and render,
while recent faster methods inevitably trade off speed for quality. For
unbounded and complete scenes (rather than isolated objects) and 1080p
resolution rendering, no current method can achieve real-time display rates. We
introduce three key elements that allow us to achieve state-of-the-art visual
quality while maintaining competitive training times and importantly allow
high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution.
First, starting from sparse points produced during camera calibration, we
represent the scene with 3D Gaussians that preserve desirable properties of
continuous volumetric radiance fields for scene optimization while avoiding
unnecessary computation in empty space; Second, we perform interleaved
optimization/density control of the 3D Gaussians, notably optimizing
anisotropic covariance to achieve an accurate representation of the scene;
Third, we develop a fast visibility-aware rendering algorithm that supports
anisotropic splatting and both accelerates training and allows realtime
rendering. We demonstrate state-of-the-art visual quality and real-time
rendering on several established datasets.Comment: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting
PAPR: Proximity Attention Point Rendering
Learning accurate and parsimonious point cloud representations of scene
surfaces from scratch remains a challenge in 3D representation learning.
Existing point-based methods often suffer from the vanishing gradient problem
or require a large number of points to accurately model scene geometry and
texture. To address these limitations, we propose Proximity Attention Point
Rendering (PAPR), a novel method that consists of a point-based scene
representation and a differentiable renderer. Our scene representation uses a
point cloud where each point is characterized by its spatial position,
foreground score, and view-independent feature vector. The renderer selects the
relevant points for each ray and produces accurate colours using their
associated features. PAPR effectively learns point cloud positions to represent
the correct scene geometry, even when the initialization drastically differs
from the target geometry. Notably, our method captures fine texture details
while using only a parsimonious set of points. We also demonstrate four
practical applications of our method: geometry editing, object manipulation,
texture transfer, and exposure control. More results and code are available on
our project website at https://zvict.github.io/papr/
4K4D: Real-Time 4D View Synthesis at 4K Resolution
This paper targets high-fidelity and real-time view synthesis of dynamic 3D
scenes at 4K resolution. Recently, some methods on dynamic view synthesis have
shown impressive rendering quality. However, their speed is still limited when
rendering high-resolution images. To overcome this problem, we propose 4K4D, a
4D point cloud representation that supports hardware rasterization and enables
unprecedented rendering speed. Our representation is built on a 4D feature grid
so that the points are naturally regularized and can be robustly optimized. In
addition, we design a novel hybrid appearance model that significantly boosts
the rendering quality while preserving efficiency. Moreover, we develop a
differentiable depth peeling algorithm to effectively learn the proposed model
from RGB videos. Experiments show that our representation can be rendered at
over 400 FPS on the DNA-Rendering dataset at 1080p resolution and 80 FPS on the
ENeRF-Outdoor dataset at 4K resolution using an RTX 4090 GPU, which is 30x
faster than previous methods and achieves the state-of-the-art rendering
quality. Our project page is available at https://zju3dv.github.io/4k4d/.Comment: Project Page: https://zju3dv.github.io/4k4
Boosting Point Clouds Rendering via Radiance Mapping
Recent years we have witnessed rapid development in NeRF-based image
rendering due to its high quality. However, point clouds rendering is somehow
less explored. Compared to NeRF-based rendering which suffers from dense
spatial sampling, point clouds rendering is naturally less computation
intensive, which enables its deployment in mobile computing device. In this
work, we focus on boosting the image quality of point clouds rendering with a
compact model design. We first analyze the adaption of the volume rendering
formulation on point clouds. Based on the analysis, we simplify the NeRF
representation to a spatial mapping function which only requires single
evaluation per pixel. Further, motivated by ray marching, we rectify the the
noisy raw point clouds to the estimated intersection between rays and surfaces
as queried coordinates, which could avoid spatial frequency collapse and
neighbor point disturbance. Composed of rasterization, spatial mapping and the
refinement stages, our method achieves the state-of-the-art performance on
point clouds rendering, outperforming prior works by notable margins, with a
smaller model size. We obtain a PSNR of 31.74 on NeRF-Synthetic, 25.88 on
ScanNet and 30.81 on DTU. Code and data would be released soon
Text-to-3D using Gaussian Splatting
In this paper, we present Gaussian Splatting based text-to-3D generation
(GSGEN), a novel approach for generating high-quality 3D objects. Previous
methods suffer from inaccurate geometry and limited fidelity due to the absence
of 3D prior and proper representation. We leverage 3D Gaussian Splatting, a
recent state-of-the-art representation, to address existing shortcomings by
exploiting the explicit nature that enables the incorporation of 3D prior.
Specifically, our method adopts a progressive optimization strategy, which
includes a geometry optimization stage and an appearance refinement stage. In
geometry optimization, a coarse representation is established under a 3D
geometry prior along with the ordinary 2D SDS loss, ensuring a sensible and
3D-consistent rough shape. Subsequently, the obtained Gaussians undergo an
iterative refinement to enrich details. In this stage, we increase the number
of Gaussians by compactness-based densification to enhance continuity and
improve fidelity. With these designs, our approach can generate 3D content with
delicate details and more accurate geometry. Extensive evaluations demonstrate
the effectiveness of our method, especially for capturing high-frequency
components. Video results are provided at https://gsgen3d.github.io. Our code
is available at https://github.com/gsgen3d/gsgenComment: Project page: https://gsgen3d.github.io. Code:
https://github.com/gsgen3d/gsge
VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for Analysis-by-Synthesis
Differentiable rendering allows the application of computer graphics onvision tasks, e.g. object pose and shape fitting, via analysis-by-synthesis,where gradients at occluded regions are important when inverting the renderingprocess. To obtain those gradients, state-of-the-art (SoTA) differentiablerenderers use rasterization to collect a set of nearest components for eachpixel and aggregate them based on the viewing distance. In this paper, wepropose VoGE, which uses ray tracing to capture nearest components with theirvolume density distributions on the rays and aggregates via integral of thevolume densities based on Gaussian ellipsoids, which brings more efficient andstable gradients. To efficiently render via VoGE, we propose an approximateclose-form solution for the volume density aggregation and a coarse-to-finerendering strategy. Finally, we provide a CUDA implementation of VoGE, whichgives a competitive rendering speed in comparison to PyTorch3D. Quantitativeand qualitative experiment results show VoGE outperforms SoTA counterparts whenapplied to various vision tasks,e.g., object pose estimation, shape/texturefitting, and occlusion reasoning. The VoGE library and demos are available athttps://github.com/Angtian/VoGE.<br
- …