2 research outputs found
Incremental Scene Synthesis
We present a method to incrementally generate complete 2D or 3D scenes with
the following properties: (a) it is globally consistent at each step according
to a learned scene prior, (b) real observations of a scene can be incorporated
while observing global consistency, (c) unobserved regions can be hallucinated
locally in consistence with previous observations, hallucinations and global
priors, and (d) hallucinations are statistical in nature, i.e., different
scenes can be generated from the same observations. To achieve this, we model
the virtual scene, where an active agent at each step can either perceive an
observed part of the scene or generate a local hallucination. The latter can be
interpreted as the agent's expectation at this step through the scene and can
be applied to autonomous navigation. In the limit of observing real data at
each point, our method converges to solving the SLAM problem. It can otherwise
sample entirely imagined scenes from prior distributions. Besides autonomous
agents, applications include problems where large data is required for building
robust real-world applications, but few samples are available. We demonstrate
efficacy on various 2D as well as 3D data
SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation
In this paper we propose a neural message passing approach to augment an
input 3D indoor scene with new objects matching their surroundings. Given an
input, potentially incomplete, 3D scene and a query location, our method
predicts a probability distribution over object types that fit well in that
location. Our distribution is predicted though passing learned messages in a
dense graph whose nodes represent objects in the input scene and edges
represent spatial and structural relationships. By weighting messages through
an attention mechanism, our method learns to focus on the most relevant
surrounding scene context to predict new scene objects. We found that our
method significantly outperforms state-of-the-art approaches in terms of
correctly predicting objects missing in a scene based on our experiments in the
SUNCG dataset. We also demonstrate other applications of our method, including
context-based 3D object recognition and iterative scene generation.Comment: 8 pages, 8 figures, to appear in ICCV 201