8 research outputs found
PaletteNeRF: Palette-based Color Editing for NeRFs
Neural Radiance Field (NeRF) is a powerful tool to faithfully generate novel
views for scenes with only sparse captured images. Despite its strong
capability for representing 3D scenes and their appearance, its editing ability
is very limited. In this paper, we propose a simple but effective extension of
vanilla NeRF, named PaletteNeRF, to enable efficient color editing on
NeRF-represented scenes. Motivated by recent palette-based image decomposition
works, we approximate each pixel color as a sum of palette colors modulated by
additive weights. Instead of predicting pixel colors as in vanilla NeRFs, our
method predicts additive weights. The underlying NeRF backbone could also be
replaced with more recent NeRF models such as KiloNeRF to achieve real-time
editing. Experimental results demonstrate that our method achieves efficient,
view-consistent, and artifact-free color editing on a wide range of
NeRF-represented scenes.Comment: 12 pages, 10 figure
RecolorNeRF: Layer Decomposed Radiance Fields for Efficient Color Editing of 3D Scenes
Radiance fields have gradually become a main representation of media.
Although its appearance editing has been studied, how to achieve
view-consistent recoloring in an efficient manner is still under explored. We
present RecolorNeRF, a novel user-friendly color editing approach for the
neural radiance fields. Our key idea is to decompose the scene into a set of
pure-colored layers, forming a palette. By this means, color manipulation can
be conducted by altering the color components of the palette directly. To
support efficient palette-based editing, the color of each layer needs to be as
representative as possible. In the end, the problem is formulated as an
optimization problem, where the layers and their blending weights are jointly
optimized with the NeRF itself. Extensive experiments show that our
jointly-optimized layer decomposition can be used against multiple backbones
and produce photo-realistic recolored novel-view renderings. We demonstrate
that RecolorNeRF outperforms baseline methods both quantitatively and
qualitatively for color editing even in complex real-world scenes.Comment: To appear in ACM Multimedia 2023. Project website is accessible at
https://sites.google.com/view/recolorner
Real-time Global Illumination Decomposition of Videos
We propose the first approach for the decomposition of a monocular color
video into direct and indirect illumination components in real time. We
retrieve, in separate layers, the contribution made to the scene appearance by
the scene reflectance, the light sources and the reflections from various
coherent scene regions to one another. Existing techniques that invert global
light transport require image capture under multiplexed controlled lighting, or
only enable the decomposition of a single image at slow off-line frame rates.
In contrast, our approach works for regular videos and produces temporally
coherent decomposition layers at real-time frame rates. At the core of our
approach are several sparsity priors that enable the estimation of the
per-pixel direct and indirect illumination layers based on a small set of
jointly estimated base reflectance colors. The resulting variational
decomposition problem uses a new formulation based on sparse and dense sets of
non-linear equations that we solve efficiently using a novel alternating
data-parallel optimization strategy. We evaluate our approach qualitatively and
quantitatively, and show improvements over the state of the art in this field,
in both quality and runtime. In addition, we demonstrate various real-time
appearance editing applications for videos with consistent illumination
Text-guided Image-and-Shape Editing and Generation: A Short Survey
Image and shape editing are ubiquitous among digital artworks. Graphics
algorithms facilitate artists and designers to achieve desired editing intents
without going through manually tedious retouching. In the recent advance of
machine learning, artists' editing intents can even be driven by text, using a
variety of well-trained neural networks. They have seen to be receiving an
extensive success on such as generating photorealistic images, artworks and
human poses, stylizing meshes from text, or auto-completion given image and
shape priors. In this short survey, we provide an overview over 50 papers on
state-of-the-art (text-guided) image-and-shape generation techniques. We start
with an overview on recent editing algorithms in the introduction. Then, we
provide a comprehensive review on text-guided editing techniques for 2D and 3D
independently, where each of its sub-section begins with a brief background
introduction. We also contextualize editing algorithms under recent implicit
neural representations. Finally, we conclude the survey with the discussion
over existing methods and potential research ideas.Comment: 10 page
Fast Accurate and Automatic Brushstroke Extraction
Brushstrokes are viewed as the artist’s “handwriting” in a painting. In many applications such as style learning and transfer, mimicking painting, and painting authentication, it is highly desired to quantitatively and accurately identify brushstroke characteristics from old masters’ pieces using computer programs. However, due to the nature of hundreds or thousands of intermingling brushstrokes in the painting, it still remains challenging. This article proposes an efficient algorithm for brush Stroke extraction based on a Deep neural network, i.e., DStroke. Compared to the state-of-the-art research, the main merit of the proposed DStroke is to automatically and rapidly extract brushstrokes from a painting without manual annotation, while accurately approximating the real brushstrokes with high reliability. Herein, recovering the faithful soft transitions between brushstrokes is often ignored by the other methods. In fact, the details of brushstrokes in a master piece of painting (e.g., shapes, colors, texture, overlaps) are highly desired by artists since they hold promise to enhance and extend the artists’ powers, just like microscopes extend biologists’ powers. To demonstrate the high efficiency of the proposed DStroke, we perform it on a set of real scans of paintings and a set of synthetic paintings, respectively. Experiments show that the proposed DStroke is noticeably faster and more accurate at identifying and extracting brushstrokes, outperforming the other methods
Image Color Correction, Enhancement, and Editing
This thesis presents methods and approaches to image color correction, color enhancement, and color editing. To begin, we study the color correction problem from the standpoint of the camera's image signal processor (ISP). A camera's ISP is hardware that applies a series of in-camera image processing and color manipulation steps, many of which are nonlinear in nature, to render the initial sensor image to its final photo-finished representation saved in the 8-bit standard RGB (sRGB) color space. As white balance (WB) is one of the major procedures applied by the ISP for color correction, this thesis presents two different methods for ISP white balancing. Afterwards, we discuss another scenario of correcting and editing image colors, where we present a set of methods to correct and edit WB settings for images that have been improperly white-balanced by the ISP. Then, we explore another factor that has a significant impact on the quality of camera-rendered colors, in which we outline two different methods to correct exposure errors in camera-rendered images. Lastly, we discuss post-capture auto color editing and manipulation. In particular, we propose auto image recoloring methods to generate different realistic versions of the same camera-rendered image with new colors. Through extensive evaluations, we demonstrate that our methods provide superior solutions compared to existing alternatives targeting color correction, color enhancement, and color editing