77 research outputs found
End-to-End Optimization of Scene Layout
We propose an end-to-end variational generative model for scene layout
synthesis conditioned on scene graphs. Unlike unconditional scene layout
generation, we use scene graphs as an abstract but general representation to
guide the synthesis of diverse scene layouts that satisfy relationships
included in the scene graph. This gives rise to more flexible control over the
synthesis process, allowing various forms of inputs such as scene layouts
extracted from sentences or inferred from a single color image. Using our
conditional layout synthesizer, we can generate various layouts that share the
same structure of the input example. In addition to this conditional generation
design, we also integrate a differentiable rendering module that enables layout
refinement using only 2D projections of the scene. Given a depth and a
semantics map, the differentiable rendering module enables optimizing over the
synthesized layout to fit the given input in an analysis-by-synthesis fashion.
Experiments suggest that our model achieves higher accuracy and diversity in
conditional scene synthesis and allows exemplar-based scene generation from
various input forms.Comment: CVPR 2020 (Oral). Project page: http://3dsln.csail.mit.edu
CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graphs
Controllable scene synthesis aims to create interactive environments for
various industrial use cases. Scene graphs provide a highly suitable interface
to facilitate these applications by abstracting the scene context in a compact
manner. Existing methods, reliant on retrieval from extensive databases or
pre-trained shape embeddings, often overlook scene-object and object-object
relationships, leading to inconsistent results due to their limited generation
capacity. To address this issue, we present CommonScenes, a fully generative
model that converts scene graphs into corresponding controllable 3D scenes,
which are semantically realistic and conform to commonsense. Our pipeline
consists of two branches, one predicting the overall scene layout via a
variational auto-encoder and the other generating compatible shapes via latent
diffusion, capturing global scene-object and local inter-object relationships
while preserving shape diversity. The generated scenes can be manipulated by
editing the input scene graph and sampling the noise in the diffusion model.
Due to lacking a scene graph dataset offering high-quality object-level meshes
with relations, we also construct SG-FRONT, enriching the off-the-shelf indoor
dataset 3D-FRONT with additional scene graph labels. Extensive experiments are
conducted on SG-FRONT where CommonScenes shows clear advantages over other
methods regarding generation consistency, quality, and diversity. Codes and the
dataset will be released upon acceptance
Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Art
Procedural content generation (PCG) can be applied to a wide variety of tasks
in games, from narratives, levels and sounds, to trees and weapons. A large
amount of game content is comprised of graphical assets, such as clouds,
buildings or vegetation, that do not require gameplay function considerations.
There is also a breadth of literature examining the procedural generation of
such elements for purposes outside of games. The body of research, focused on
specific methods for generating specific assets, provides a narrow view of the
available possibilities. Hence, it is difficult to have a clear picture of all
approaches and possibilities, with no guide for interested parties to discover
possible methods and approaches for their needs, and no facility to guide them
through each technique or approach to map out the process of using them.
Therefore, a systematic literature review has been conducted, yielding 200
accepted papers. This paper explores state-of-the-art approaches to graphical
asset generation, examining research from a wide range of applications, inside
and outside of games. Informed by the literature, a conceptual framework has
been derived to address the aforementioned gaps
- …