354 research outputs found
Image inpainting with gradient attention
We present a novel modification of context encoder loss function, which results in more accurate and plausible inpainting. For this purpose, we introduce gradient attention loss component of loss function, to suppress the common problem of inconsistency in shapes and edges between the inpainted region and its context. To this end, the mean absolute error is computed not only for the input and output images, but also for their derivatives. Therefore, model concentrates on areas with larger gradient, which are crucial for accurate reconstruction. The positive effects on inpainting results are observed both for fully-connected and fully-convolutional models tested on MNIST and CelebA datasets
CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training
Recent image inpainting methods have made great progress but often struggle
to generate plausible image structures when dealing with large holes in complex
images. This is partially due to the lack of effective network structures that
can capture both the long-range dependency and high-level semantics of an
image. To address these problems, we propose cascaded modulation GAN (CM-GAN),
a new network design consisting of an encoder with Fourier convolution blocks
that extract multi-scale feature representations from the input image with
holes and a StyleGAN-like decoder with a novel cascaded global-spatial
modulation block at each scale level. In each decoder block, global modulation
is first applied to perform coarse semantic-aware structure synthesis, then
spatial modulation is applied on the output of global modulation to further
adjust the feature map in a spatially adaptive fashion. In addition, we design
an object-aware training scheme to prevent the network from hallucinating new
objects inside holes, fulfilling the needs of object removal tasks in
real-world scenarios. Extensive experiments are conducted to show that our
method significantly outperforms existing methods in both quantitative and
qualitative evaluation.Comment: 32 pages, 18 figure
- …