'Institute of Electrical and Electronics Engineers (IEEE)'
Abstract
We consider the problem of editing 3D objects and scenes
based on open-ended language instructions. A common approach to this
problem is to use a 2D image generator or editor to guide the 3D editing
process, obviating the need for 3D data. However, this process is often
inefficient due to the need for iterative updates of costly 3D representations, such as neural radiance fields, either through individual view edits
or score distillation sampling. A major disadvantage of this approach
is the slow convergence caused by aggregating inconsistent information
across views, as the guidance from 2D models is not multi-view consistent. We thus introduce the Direct Gaussian Editor (DGE), a method
that addresses these issues in two stages. First, we modify a given highquality image editor like InstructPix2Pix to be multi-view consistent. To
do so, we propose a training-free approach that integrates cues from the
3D geometry of the underlying scene. Second, given a multi-view consistent edited sequence of images, we directly and efficiently optimize the
3D representation, which is based on 3D Gaussian Splatting. Because it
avoids incremental and iterative edits, DGE is significantly more accurate and efficient than existing approaches and offers additional benefits,
such as enabling selective editing of parts of the scene