1,180 research outputs found
Patch-based Texture Synthesis for Image Inpainting
Image inpaiting is an important task in image processing and vision. In this
paper, we develop a general method for patch-based image inpainting by
synthesizing new textures from existing one. A novel framework is introduced to
find several optimal candidate patches and generate a new texture patch in the
process. We form it as an optimization problem that identifies the potential
patches for synthesis from an coarse-to-fine manner. We use the texture
descriptor as a clue in searching for matching patches from the known region.
To ensure the structure faithful to the original image, a geometric constraint
metric is formally defined that is applied directly to the patch synthesis
procedure. We extensively conducted our experiments on a wide range of testing
images on various scenarios and contents by arbitrarily specifying the target
the regions for inference followed by using existing evaluation metrics to
verify its texture coherency and structural consistency. Our results
demonstrate the high accuracy and desirable output that can be potentially used
for numerous applications: object removal, background subtraction, and image
retrieval.Comment: in Computer Science and Applications, 201
Unsupervised Deep Context Prediction for Background Foreground Separation
In many advanced video based applications background modeling is a
pre-processing step to eliminate redundant data, for instance in tracking or
video surveillance applications. Over the past years background subtraction is
usually based on low level or hand-crafted features such as raw color
components, gradients, or local binary patterns. The background subtraction
algorithms performance suffer in the presence of various challenges such as
dynamic backgrounds, photometric variations, camera jitters, and shadows. To
handle these challenges for the purpose of accurate background modeling we
propose a unified framework based on the algorithm of image inpainting. It is
an unsupervised visual feature learning hybrid Generative Adversarial algorithm
based on context prediction. We have also presented the solution of random
region inpainting by the fusion of center region inpaiting and random region
inpainting with the help of poisson blending technique. Furthermore we also
evaluated foreground object detection with the fusion of our proposed method
and morphological operations. The comparison of our proposed method with 12
state-of-the-art methods shows its stability in the application of background
estimation and foreground detection.Comment: 17 page
High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling
Existing image inpainting methods often produce artifacts when dealing with
large holes in real applications. To address this challenge, we propose an
iterative inpainting method with a feedback mechanism. Specifically, we
introduce a deep generative model which not only outputs an inpainting result
but also a corresponding confidence map. Using this map as feedback, it
progressively fills the hole by trusting only high-confidence pixels inside the
hole at each iteration and focuses on the remaining pixels in the next
iteration. As it reuses partial predictions from the previous iterations as
known pixels, this process gradually improves the result. In addition, we
propose a guided upsampling network to enable generation of high-resolution
inpainting results. We achieve this by extending the Contextual Attention
module to borrow high-resolution feature patches in the input image.
Furthermore, to mimic real object removal scenarios, we collect a large object
mask dataset and synthesize more realistic training data that better simulates
user inputs. Experiments show that our method significantly outperforms
existing methods in both quantitative and qualitative evaluations. More results
and Web APP are available at https://zengxianyu.github.io/iic
Multi-View Inpainting for RGB-D Sequence
In this work we propose a novel approach to remove undesired objects from
RGB-D sequences captured with freely moving cameras, which enables static 3D
reconstruction. Our method jointly uses existing information from multiple
frames as well as generates new one via inpainting techniques. We use balanced
rules to select source frames; local homography based image warping method for
alignment and Markov random field (MRF) based approach for combining existing
information. For the left holes, we employ exemplar based multi-view inpainting
method to deal with the color image and coherently use it as guidance to
complete the depth correspondence. Experiments show that our approach is
qualified for removing the undesired objects and inpainting the holes.Comment: 10 page
Image Inpainting using Block-wise Procedural Training with Annealed Adversarial Counterpart
Recent advances in deep generative models have shown promising potential in
image inpanting, which refers to the task of predicting missing pixel values of
an incomplete image using the known context. However, existing methods can be
slow or generate unsatisfying results with easily detectable flaws. In
addition, there is often perceivable discontinuity near the holes and require
further post-processing to blend the results. We present a new approach to
address the difficulty of training a very deep generative model to synthesize
high-quality photo-realistic inpainting. Our model uses conditional generative
adversarial networks (conditional GANs) as the backbone, and we introduce a
novel block-wise procedural training scheme to stabilize the training while we
increase the network depth. We also propose a new strategy called adversarial
loss annealing to reduce the artifacts. We further describe several losses
specifically designed for inpainting and show their effectiveness. Extensive
experiments and user-study show that our approach outperforms existing methods
in several tasks such as inpainting, face completion and image harmonization.
Finally, we show our framework can be easily used as a tool for interactive
guided inpainting, demonstrating its practical value to solve common real-world
challenges
Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space
In this paper we present an end-to-end deep learning framework to turn images
that show dynamic content, such as vehicles or pedestrians, into realistic
static frames. This objective encounters two main challenges: detecting all the
dynamic objects, and inpainting the static occluded background with plausible
imagery. The second problem is approached with a conditional generative
adversarial model that, taking as input the original dynamic image and its
dynamic/static binary mask, is capable of generating the final static image.
The former challenge is addressed by the use of a convolutional network that
learns a multi-class semantic segmentation of the image.
These generated images can be used for applications such as augmented reality
or vision-based robot localization purposes. To validate our approach, we show
both qualitative and quantitative comparisons against other state-of-the-art
inpainting methods by removing the dynamic objects and hallucinating the static
structure behind them. Furthermore, to demonstrate the potential of our
results, we carry out pilot experiments that show the benefits of our proposal
for visual place recognition.Comment: Accepted for Publication at IEEE International Conference on Robotics
and Automation (ICRA) 201
Foreground-aware Image Inpainting
Existing image inpainting methods typically fill holes by borrowing
information from surrounding pixels. They often produce unsatisfactory results
when the holes overlap with or touch foreground objects due to lack of
information about the actual extent of foreground and background regions within
the holes. These scenarios, however, are very important in practice, especially
for applications such as the removal of distracting objects. To address the
problem, we propose a foreground-aware image inpainting system that explicitly
disentangles structure inference and content completion. Specifically, our
model learns to predict the foreground contour first, and then inpaints the
missing region using the predicted contour as guidance. We show that by such
disentanglement, the contour completion model predicts reasonable contours of
objects, and further substantially improves the performance of image
inpainting. Experiments show that our method significantly outperforms existing
methods and achieves superior inpainting results on challenging cases with
complex compositions.Comment: Camera Ready version of CVPR 2019 with supplementary material
Texture Modelling with Nested High-order Markov-Gibbs Random Fields
Currently, Markov-Gibbs random field (MGRF) image models which include
high-order interactions are almost always built by modelling responses of a
stack of local linear filters. Actual interaction structure is specified
implicitly by the filter coefficients. In contrast, we learn an explicit
high-order MGRF structure by considering the learning process in terms of
general exponential family distributions nested over base models, so that
potentials added later can build on previous ones. We relatively rapidly add
new features by skipping over the costly optimisation of parameters.
We introduce the use of local binary patterns as features in MGRF texture
models, and generalise them by learning offsets to the surrounding pixels.
These prove effective as high-order features, and are fast to compute. Several
schemes for selecting high-order features by composition or search of a small
subclass are compared. Additionally we present a simple modification of the
maximum likelihood as a texture modelling-specific objective function which
aims to improve generalisation by local windowing of statistics.
The proposed method was experimentally evaluated by learning high-order MGRF
models for a broad selection of complex textures and then performing texture
synthesis, and succeeded on much of the continuum from stochastic through
irregularly structured to near-regular textures. Learning interaction structure
is very beneficial for textures with large-scale structure, although those with
complex irregular structure still provide difficulties. The texture models were
also quantitatively evaluated on two tasks and found to be competitive with
other works: grading of synthesised textures by a panel of observers; and
comparison against several recent MGRF models by evaluation on a constrained
inpainting task.Comment: Submitted to Computer Vision and Image Understandin
Patch-Based Image Inpainting with Generative Adversarial Networks
Area of image inpainting over relatively large missing regions recently
advanced substantially through adaptation of dedicated deep neural networks.
However, current network solutions still introduce undesired artifacts and
noise to the repaired regions. We present an image inpainting method that is
based on the celebrated generative adversarial network (GAN) framework. The
proposed PGGAN method includes a discriminator network that combines a global
GAN (G-GAN) architecture with a patchGAN approach. PGGAN first shares network
layers between G-GAN and patchGAN, then splits paths to produce two adversarial
losses that feed the generator network in order to capture both local
continuity of image texture and pervasive global features in images. The
proposed framework is evaluated extensively, and the results including
comparison to recent state-of-the-art demonstrate that it achieves considerable
improvements on both visual and quantitative evaluations
Shift-Net: Image Inpainting via Deep Feature Rearrangement
Deep convolutional networks (CNNs) have exhibited their potential in image
inpainting for producing plausible results. However, in most existing methods,
e.g., context encoder, the missing parts are predicted by propagating the
surrounding convolutional features through a fully connected layer, which
intends to produce semantically plausible but blurry result. In this paper, we
introduce a special shift-connection layer to the U-Net architecture, namely
Shift-Net, for filling in missing regions of any shape with sharp structures
and fine-detailed textures. To this end, the encoder feature of the known
region is shifted to serve as an estimation of the missing parts. A guidance
loss is introduced on decoder feature to minimize the distance between the
decoder feature after fully connected layer and the ground-truth encoder
feature of the missing parts. With such constraint, the decoder feature in
missing region can be used to guide the shift of encoder feature in known
region. An end-to-end learning algorithm is further developed to train the
Shift-Net. Experiments on the Paris StreetView and Places datasets demonstrate
the efficiency and effectiveness of our Shift-Net in producing sharper,
fine-detailed, and visually plausible results. The codes and pre-trained models
are available at https://github.com/Zhaoyi-Yan/Shift-Net.Comment: 25 pages, 17 figures, 1 table, main paper + supplementary materia
- …