182 research outputs found
Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction
The ultimate goal of many image-based modeling systems is to render
photo-realistic novel views of a scene without visible artifacts. Existing
evaluation metrics and benchmarks focus mainly on the geometric accuracy of the
reconstructed model, which is, however, a poor predictor of visual accuracy.
Furthermore, using only geometric accuracy by itself does not allow evaluating
systems that either lack a geometric scene representation or utilize coarse
proxy geometry. Examples include light field or image-based rendering systems.
We propose a unified evaluation approach based on novel view prediction error
that is able to analyze the visual quality of any method that can render novel
views from input images. One of the key advantages of this approach is that it
does not require ground truth geometry. This dramatically simplifies the
creation of test datasets and benchmarks. It also allows us to evaluate the
quality of an unknown scene during the acquisition and reconstruction process,
which is useful for acquisition planning. We evaluate our approach on a range
of methods including standard geometry-plus-texture pipelines as well as
image-based rendering techniques, compare it to existing geometry-based
benchmarks, and demonstrate its utility for a range of use cases.Comment: 10 pages, 12 figures, paper was submitted to ACM Transactions on
Graphics for revie
Stereoscopic Seam Carving With Temporal Consistency
In this paper, we present a novel technique for seam carving of stereoscopic video. It removes seams of pixels in areas that are most likely not noticed by the viewer. When applying seam carving to stereoscopic video rather than monoscopic still images, new challenges arise. The detected seams must be consistent between the left and the right view, so that no depth information is destroyed. When removing seams in two consecutive frames, temporal consistency between the removed seams must be established to avoid flicker in the resulting video. By making certain assumptions, the available depth information can be harnessed to improve the quality achieved by seam carving. Assuming that closer pixels are more important, the algorithm can focus on removing distant pixels first. Furthermore, we assume that coherent pixels belonging to the same object have similar depth. By avoiding to cut through edges in the depth map, we can thus avoid cutting through object boundaries
Capturing and viewing gigapixel images
We present a system to capture and view "Gigapixel images": very high resolution, high dynamic range, and wide angle imagery consisting of several billion pixels each. A specialized camera mount, in combination with an automated pipeline for alignment, exposure compensation, and stitching, provide the means to acquire Gigapixel images with a standard camera and lens. More importantly, our novel viewer enables exploration of such images at interactive rates over a network, while dynamically and smoothly interpolating the projection between perspective and curved projections, and simultaneously modifying the tone-mapping to ensure an optimal view of the portion of the scene being viewed.publishe
Single-Image 3D Human Digitization with Shape-Guided Diffusion
We present an approach to generate a 360-degree view of a person with a
consistent, high-resolution appearance from a single input image. NeRF and its
variants typically require videos or images from different viewpoints. Most
existing approaches taking monocular input either rely on ground-truth 3D scans
for supervision or lack 3D consistency. While recent 3D generative models show
promise of 3D consistent human digitization, these approaches do not generalize
well to diverse clothing appearances, and the results lack photorealism. Unlike
existing work, we utilize high-capacity 2D diffusion models pretrained for
general image synthesis tasks as an appearance prior of clothed humans. To
achieve better 3D consistency while retaining the input identity, we
progressively synthesize multiple views of the human in the input image by
inpainting missing regions with shape-guided diffusion conditioned on
silhouette and surface normal. We then fuse these synthesized multi-view images
via inverse rendering to obtain a fully textured high-resolution 3D mesh of the
given person. Experiments show that our approach outperforms prior methods and
achieves photorealistic 360-degree synthesis of a wide range of clothed humans
with complex textures from a single image.Comment: SIGGRAPH Asia 2023. Project website: https://human-sgd.github.io
HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
Volumetric scene representations enable photorealistic view synthesis for
static scenes and form the basis of several existing 6-DoF video techniques.
However, the volume rendering procedures that drive these representations
necessitate careful trade-offs in terms of quality, rendering speed, and memory
efficiency. In particular, existing methods fail to simultaneously achieve
real-time performance, small memory footprint, and high-quality rendering for
challenging real-world scenes. To address these issues, we present HyperReel --
a novel 6-DoF video representation. The two core components of HyperReel are:
(1) a ray-conditioned sample prediction network that enables high-fidelity,
high frame rate rendering at high resolutions and (2) a compact and
memory-efficient dynamic volume representation. Our 6-DoF video pipeline
achieves the best performance compared to prior and contemporary approaches in
terms of visual quality with small memory requirements, while also rendering at
up to 18 frames-per-second at megapixel resolution without any custom CUDA
code.Comment: Project page: https://hyperreel.github.io
Adapting the randomised controlled trial (RCT) for precision medicine: introducing the nested-precision RCT (npRCT)
Adaptations to the gold standard randomised controlled trial (RCT) have been introduced to decrease trial costs and avoid high sample sizes. To facilitate development of precision medicine algorithms that aim to optimise treatment allocation for individual patients, we propose a new RCT adaptation termed the nested-precision RCT (npRCT). The npRCT combines a traditional RCT (intervention A versus B) with a precision RCT (stratified versus randomised allocation to A or B). This combination allows online development of a precision algorithm, thus providing an integrated platform for algorithm development and its testing. Moreover, as both the traditional and the precision RCT include participants randomised to interventions of interest, data from these participants can be jointly analysed to determine the comparative effectiveness of intervention A versus B, thus increasing statistical power. We quantify savings of the npRCT compared to two independent RCTs by highlighting sample size requirements for different target effect sizes and by introducing an open-source power calculation app. We describe important practical considerations such as blinding issues and potential biases that need to be considered when designing an npRCT. We also highlight limitations and research contexts that are less suited for an npRCT. In conclusion, we introduce the npRCT as a novel precision medicine trial design strategy which may provide one opportunity to efficiently combine traditional and precision RCTs
- …