13 research outputs found
Content Masked Loss: Human-Like Brush Stroke Planning in a Reinforcement Learning Painting Agent
The objective of most Reinforcement Learning painting agents is to minimize
the loss between a target image and the paint canvas. Human painter artistry
emphasizes important features of the target image rather than simply
reproducing it (DiPaola 2007). Using adversarial or L2 losses in the RL
painting models, although its final output is generally a work of finesse,
produces a stroke sequence that is vastly different from that which a human
would produce since the model does not have knowledge about the abstract
features in the target image. In order to increase the human-like planning of
the model without the use of expensive human data, we introduce a new loss
function for use with the model's reward function: Content Masked Loss. In the
context of robot painting, Content Masked Loss employs an object detection
model to extract features which are used to assign higher weight to regions of
the canvas that a human would find important for recognizing content. The
results, based on 332 human evaluators, show that the digital paintings
produced by our Content Masked model show detectable subject matter earlier in
the stroke sequence than existing methods without compromising on the quality
of the final painting. Our code is available at
https://github.com/pschaldenbrand/ContentMaskedLoss
Painting Many Pasts: Synthesizing Time Lapse Videos of Paintings
We introduce a new video synthesis task: synthesizing time lapse videos
depicting how a given painting might have been created. Artists paint using
unique combinations of brushes, strokes, and colors. There are often many
possible ways to create a given painting. Our goal is to learn to capture this
rich range of possibilities.
Creating distributions of long-term videos is a challenge for learning-based
video synthesis methods. We present a probabilistic model that, given a single
image of a completed painting, recurrently synthesizes steps of the painting
process. We implement this model as a convolutional neural network, and
introduce a novel training scheme to enable learning from a limited dataset of
painting time lapses. We demonstrate that this model can be used to sample many
time steps, enabling long-term stochastic video synthesis. We evaluate our
method on digital and watercolor paintings collected from video websites, and
show that human raters find our synthetic videos to be similar to time lapse
videos produced by real artists. Our code is available at
https://xamyzhao.github.io/timecraft.Comment: 10 pages, CVPR 202