4,090 research outputs found
Calipso: Physics-based Image and Video Editing through CAD Model Proxies
We present Calipso, an interactive method for editing images and videos in a
physically-coherent manner. Our main idea is to realize physics-based
manipulations by running a full physics simulation on proxy geometries given by
non-rigidly aligned CAD models. Running these simulations allows us to apply
new, unseen forces to move or deform selected objects, change physical
parameters such as mass or elasticity, or even add entire new objects that
interact with the rest of the underlying scene. In Calipso, the user makes
edits directly in 3D; these edits are processed by the simulation and then
transfered to the target 2D content using shape-to-image correspondences in a
photo-realistic rendering process. To align the CAD models, we introduce an
efficient CAD-to-image alignment procedure that jointly minimizes for rigid and
non-rigid alignment while preserving the high-level structure of the input
shape. Moreover, the user can choose to exploit image flow to estimate scene
motion, producing coherent physical behavior with ambient dynamics. We
demonstrate Calipso's physics-based editing on a wide range of examples
producing myriad physical behavior while preserving geometric and visual
consistency.Comment: 11 page
Robust Temporally Coherent Laplacian Protrusion Segmentation of 3D Articulated Bodies
In motion analysis and understanding it is important to be able to fit a
suitable model or structure to the temporal series of observed data, in order
to describe motion patterns in a compact way, and to discriminate between them.
In an unsupervised context, i.e., no prior model of the moving object(s) is
available, such a structure has to be learned from the data in a bottom-up
fashion. In recent times, volumetric approaches in which the motion is captured
from a number of cameras and a voxel-set representation of the body is built
from the camera views, have gained ground due to attractive features such as
inherent view-invariance and robustness to occlusions. Automatic, unsupervised
segmentation of moving bodies along entire sequences, in a temporally-coherent
and robust way, has the potential to provide a means of constructing a
bottom-up model of the moving body, and track motion cues that may be later
exploited for motion classification. Spectral methods such as locally linear
embedding (LLE) can be useful in this context, as they preserve "protrusions",
i.e., high-curvature regions of the 3D volume, of articulated shapes, while
improving their separation in a lower dimensional space, making them in this
way easier to cluster. In this paper we therefore propose a spectral approach
to unsupervised and temporally-coherent body-protrusion segmentation along time
sequences. Volumetric shapes are clustered in an embedding space, clusters are
propagated in time to ensure coherence, and merged or split to accommodate
changes in the body's topology. Experiments on both synthetic and real
sequences of dense voxel-set data are shown. This supports the ability of the
proposed method to cluster body-parts consistently over time in a totally
unsupervised fashion, its robustness to sampling density and shape quality, and
its potential for bottom-up model constructionComment: 31 pages, 26 figure
Scene relighting and editing for improved object insertion
Abstract. The goal of this thesis is to develop a scene relighting and object insertion pipeline using Neural Radiance Fields (NeRF) to incorporate one or more objects into an outdoor environment scene. The output is a 3D mesh that embodies decomposed bidirectional reflectance distribution function (BRDF) characteristics, which interact with varying light source positions and strengths. To achieve this objective, the thesis is divided into two sub-tasks.
The first sub-task involves extracting visual information about the outdoor environment from a sparse set of corresponding images. A neural representation is constructed, providing a comprehensive understanding of the constituent elements, such as materials, geometry, illumination, and shadows. The second sub-task involves generating a neural representation of the inserted object using either real-world images or synthetic data.
To accomplish these objectives, the thesis draws on existing literature in computer vision and computer graphics. Different approaches are assessed to identify their advantages and disadvantages, with detailed descriptions of the chosen techniques provided, highlighting their functioning to produce the ultimate outcome.
Overall, this thesis aims to provide a framework for compositing and relighting that is grounded in NeRF and allows for the seamless integration of objects into outdoor environments. The outcome of this work has potential applications in various domains, such as visual effects, gaming, and virtual reality
An interactive 3D medical visualization system based on a light field display
This paper presents a prototype medical data visualization system exploiting a light field display and custom direct volume rendering techniques to enhance understanding of massive volumetric data, such as CT, MRI, and PET scans. The system can be integrated with standard medical image archives and extends the capabilities of current radiology workstations by supporting real-time rendering of volumes of potentially unlimited size on light field displays generating dynamic observer-independent light fields. The system allows multiple untracked naked-eye users in a sufficiently large interaction area to coherently perceive rendered volumes as real objects, with stereo and motion parallax cues. In this way, an effective collaborative analysis of volumetric data can be achieved. Evaluation tests demonstrate the usefulness of the generated depth cues and the improved performance in understanding complex spatial structures with respect to standard techniques.883-893Pubblicat
Dynamic Mesh-Aware Radiance Fields
Embedding polygonal mesh assets within photorealistic Neural Radience Fields
(NeRF) volumes, such that they can be rendered and their dynamics simulated in
a physically consistent manner with the NeRF, is under-explored from the system
perspective of integrating NeRF into the traditional graphics pipeline. This
paper designs a two-way coupling between mesh and NeRF during rendering and
simulation. We first review the light transport equations for both mesh and
NeRF, then distill them into an efficient algorithm for updating radiance and
throughput along a cast ray with an arbitrary number of bounces. To resolve the
discrepancy between the linear color space that the path tracer assumes and the
sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range
(HDR) images. We also present a strategy to estimate light sources and cast
shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric
formulation can be efficiently integrated with a high-performance physics
simulator that supports cloth, rigid and soft bodies. The full rendering and
simulation system can be run on a GPU at interactive rates. We show that a
hybrid system approach outperforms alternatives in visual realism for mesh
insertion, because it allows realistic light transport from volumetric NeRF
media onto surfaces, which affects the appearance of reflective/refractive
surfaces and illumination of diffuse surfaces informed by the dynamic scene.Comment: ICCV 202
NeRFs: The Search for the Best 3D Representation
Neural Radiance Fields or NeRFs have become the representation of choice for
problems in view synthesis or image-based rendering, as well as in many other
applications across computer graphics and vision, and beyond. At their core,
NeRFs describe a new representation of 3D scenes or 3D geometry. Instead of
meshes, disparity maps, multiplane images or even voxel grids, they represent
the scene as a continuous volume, with volumetric parameters like
view-dependent radiance and volume density obtained by querying a neural
network. The NeRF representation has now been widely used, with thousands of
papers extending or building on it every year, multiple authors and websites
providing overviews and surveys, and numerous industrial applications and
startup companies. In this article, we briefly review the NeRF representation,
and describe the three decades-long quest to find the best 3D representation
for view synthesis and related problems, culminating in the NeRF papers. We
then describe new developments in terms of NeRF representations and make some
observations and insights regarding the future of 3D representations.Comment: Updated based on feedback in-person and via e-mail at SIGGRAPH 2023.
In particular, I have added references and discussion of seminal SIGGRAPH
image-based rendering papers, and better put the recent Kerbl et al. work in
context, with more reference
Artistic Path Space Editing of Physically Based Light Transport
Die Erzeugung realistischer Bilder ist ein wichtiges Ziel der Computergrafik, mit Anwendungen u.a. in der Spielfilmindustrie, Architektur und Medizin. Die physikalisch basierte Bildsynthese, welche in letzter Zeit anwendungsübergreifend weiten Anklang findet, bedient sich der numerischen Simulation des Lichttransports entlang durch die geometrische Optik vorgegebener Ausbreitungspfade; ein Modell, welches für übliche Szenen ausreicht, Photorealismus zu erzielen.
Insgesamt gesehen ist heute das computergestützte Verfassen von Bildern und Animationen mit wohlgestalteter und theoretisch fundierter Schattierung stark vereinfacht. Allerdings ist bei der praktischen Umsetzung auch die Rücksichtnahme auf Details wie die Struktur des Ausgabegeräts wichtig und z.B. das Teilproblem der effizienten physikalisch basierten Bildsynthese in partizipierenden Medien ist noch weit davon entfernt, als gelöst zu gelten.
Weiterhin ist die Bildsynthese als Teil eines weiteren Kontextes zu sehen: der effektiven Kommunikation von Ideen und Informationen. Seien es nun Form und Funktion eines Gebäudes, die medizinische Visualisierung einer Computertomografie oder aber die Stimmung einer Filmsequenz -- Botschaften in Form digitaler Bilder sind heutzutage omnipräsent. Leider hat die Verbreitung der -- auf Simulation ausgelegten -- Methodik der physikalisch basierten Bildsynthese generell zu einem Verlust intuitiver, feingestalteter und lokaler künstlerischer Kontrolle des finalen Bildinhalts geführt, welche in vorherigen, weniger strikten Paradigmen vorhanden war.
Die Beiträge dieser Dissertation decken unterschiedliche Aspekte der Bildsynthese ab. Dies sind zunächst einmal die grundlegende Subpixel-Bildsynthese sowie effiziente Bildsyntheseverfahren für partizipierende Medien. Im Mittelpunkt der Arbeit stehen jedoch Ansätze zum effektiven visuellen Verständnis der Lichtausbreitung, die eine lokale künstlerische Einflussnahme ermöglichen und gleichzeitig auf globaler Ebene konsistente und glaubwürdige Ergebnisse erzielen. Hierbei ist die Kernidee, Visualisierung und Bearbeitung des Lichts direkt im alle möglichen Lichtpfade einschließenden "Pfadraum" durchzuführen. Dies steht im Gegensatz zu Verfahren nach Stand der Forschung, die entweder im Bildraum arbeiten oder auf bestimmte, isolierte Beleuchtungseffekte wie perfekte Spiegelungen, Schatten oder Kaustiken zugeschnitten sind. Die Erprobung der vorgestellten Verfahren hat gezeigt, dass mit ihnen real existierende Probleme der Bilderzeugung für Filmproduktionen gelöst werden können
Virtual finger boosts three-dimensional imaging and microsurgery as well as terabyte volume image visualization and analysis
Three-dimensional (3D) bioimaging, visualization and data analysis are in strong need of powerful 3D exploration techniques. We develop virtual finger (VF) to generate 3D curves, points and regions-of-interest in the 3D space of a volumetric image with a single finger operation, such as a computer mouse stroke, or click or zoom from the 2D-projection plane of an image as visualized with a computer. VF provides efficient methods for acquisition, visualization and analysis of 3D images for roundworm, fruitfly, dragonfly, mouse, rat and human. Specifically, VF enables instant 3D optical zoom-in imaging, 3D free-form optical microsurgery, and 3D visualization and annotation of terabytes of whole-brain image volumes. VF also leads to orders of magnitude better efficiency of automated 3D reconstruction of neurons and similar biostructures over our previous systems. We use VF to generate from images of 1,107 Drosophila GAL4 lines a projectome of a Drosophila brain. © 2014 Macmillan Publishers Limited. All rights reserved
Efficient Many-Light Rendering of Scenes with Participating Media
We present several approaches based on virtual lights that aim at capturing the light transport without compromising quality, and while preserving the elegance and efficiency of many-light rendering. By reformulating the integration scheme, we obtain two numerically efficient techniques; one tailored specifically for interactive, high-quality lighting on surfaces, and one for handling scenes with participating media
KiloNeuS: A Versatile Neural Implicit Surface Representation for Real-Time Rendering
NeRF-based techniques fit wide and deep multi-layer perceptrons (MLPs) to a
continuous radiance field that can be rendered from any unseen viewpoint.
However, the lack of surface and normals definition and high rendering times
limit their usage in typical computer graphics applications. Such limitations
have recently been overcome separately, but solving them together remains an
open problem. We present KiloNeuS, a neural representation reconstructing an
implicit surface represented as a signed distance function (SDF) from
multi-view images and enabling real-time rendering by partitioning the space
into thousands of tiny MLPs fast to inference. As we learn the implicit surface
locally using independent models, resulting in a globally coherent geometry is
non-trivial and needs to be addressed during training. We evaluate rendering
performance on a GPU-accelerated ray-caster with in-shader neural network
inference, resulting in an average of 46 FPS at high resolution, proving a
satisfying tradeoff between storage costs and rendering quality. In fact, our
evaluation for rendering quality and surface recovery shows that KiloNeuS
outperforms its single-MLP counterpart. Finally, to exhibit the versatility of
KiloNeuS, we integrate it into an interactive path-tracer taking full advantage
of its surface normals. We consider our work a crucial first step toward
real-time rendering of implicit neural representations under global
illumination.Comment: 9 pages, 8 figure
- …