32,681 research outputs found
A unified view on patch aggregation
Patch-based methods are widely used in various topics of image processing, such as image restoration or image editing and synthesis. Patches capture local image geometry and structure and are much easier to model than whole images: in practice, patches are small enough to be represented by simple multivariate priors. An important question arising in all patch-based methods is the one of patch aggregation. For instance, in image restoration, restored patches are usually not compatible, in the sense that two overlapping restored patches do not necessarily yield the same values to their common pixels. A standard way to overcome this difficulty is to see the values provided by different patches at a given pixel as independent estimators of a true unknown value and to aggregate these estimators. This aggregation step usually boils down to a simple average , with uniform weights or with weights depending on the trust we have on these different estimators. In this paper, we propose a probabilistic framework aiming at a better understanding of this crucial and often neglected step. The key idea is to see the aggregation of two patches as a fusion between their models rather than a fusion of estimators. The proposed fusion operation is pretty intuitive and generalizes previous aggregation methods. It also yields a novel interpretation of the Expected Patch Log Likelihood (EPLL) proposed in [40
A Unified Conditional Framework for Diffusion-based Image Restoration
Diffusion Probabilistic Models (DPMs) have recently shown remarkable
performance in image generation tasks, which are capable of generating highly
realistic images. When adopting DPMs for image restoration tasks, the crucial
aspect lies in how to integrate the conditional information to guide the DPMs
to generate accurate and natural output, which has been largely overlooked in
existing works. In this paper, we present a unified conditional framework based
on diffusion models for image restoration. We leverage a lightweight UNet to
predict initial guidance and the diffusion model to learn the residual of the
guidance. By carefully designing the basic module and integration module for
the diffusion model block, we integrate the guidance and other auxiliary
conditional information into every block of the diffusion model to achieve
spatially-adaptive generation conditioning. To handle high-resolution images,
we propose a simple yet effective inter-step patch-splitting strategy to
produce arbitrary-resolution images without grid artifacts. We evaluate our
conditional framework on three challenging tasks: extreme low-light denoising,
deblurring, and JPEG restoration, demonstrating its significant improvements in
perceptual quality and the generalization to restoration tasks
The Power of Triply Complementary Priors for Image Compressive Sensing
Recent works that utilized deep models have achieved superior results in
various image restoration applications. Such approach is typically supervised
which requires a corpus of training images with distribution similar to the
images to be recovered. On the other hand, the shallow methods which are
usually unsupervised remain promising performance in many inverse problems,
\eg, image compressive sensing (CS), as they can effectively leverage non-local
self-similarity priors of natural images. However, most of such methods are
patch-based leading to the restored images with various ringing artifacts due
to naive patch aggregation. Using either approach alone usually limits
performance and generalizability in image restoration tasks. In this paper, we
propose a joint low-rank and deep (LRD) image model, which contains a pair of
triply complementary priors, namely \textit{external} and \textit{internal},
\textit{deep} and \textit{shallow}, and \textit{local} and \textit{non-local}
priors. We then propose a novel hybrid plug-and-play (H-PnP) framework based on
the LRD model for image CS. To make the optimization tractable, a simple yet
effective algorithm is proposed to solve the proposed H-PnP based image CS
problem. Extensive experimental results demonstrate that the proposed H-PnP
algorithm significantly outperforms the state-of-the-art techniques for image
CS recovery such as SCSNet and WNNM
Super-Resolution Based Patch-Free 3D Image Segmentation with High-Frequency Guidance
High resolution (HR) 3D images are widely used nowadays, such as medical
images like Magnetic Resonance Imaging (MRI) and Computed Tomography (CT).
However, segmentation of these 3D images remains a challenge due to their high
spatial resolution and dimensionality in contrast to currently limited GPU
memory. Therefore, most existing 3D image segmentation methods use patch-based
models, which have low inference efficiency and ignore global contextual
information. To address these problems, we propose a super-resolution (SR)
based patch-free 3D image segmentation framework that can realize HR
segmentation from a global-wise low-resolution (LR) input. The framework
contains two sub-tasks, of which semantic segmentation is the main task and
super resolution is an auxiliary task aiding in rebuilding the high frequency
information from the LR input. To furthermore balance the information loss with
the LR input, we propose a High-Frequency Guidance Module (HGM), and design an
efficient selective cropping algorithm to crop an HR patch from the original
image as restoration guidance for it. In addition, we also propose a
Task-Fusion Module (TFM) to exploit the inter connections between segmentation
and SR task, realizing joint optimization of the two tasks. When predicting,
only the main segmentation task is needed, while other modules can be removed
for acceleration. The experimental results on two different datasets show that
our framework has a four times higher inference speed compared to traditional
patch-based methods, while its performance also surpasses other patch-based and
patch-free models.Comment: Version #2 uploaded in Jul 10, 202
A Bayesian Hyperprior Approach for Joint Image Denoising and Interpolation, with an Application to HDR Imaging
Recently, impressive denoising results have been achieved by Bayesian
approaches which assume Gaussian models for the image patches. This improvement
in performance can be attributed to the use of per-patch models. Unfortunately
such an approach is particularly unstable for most inverse problems beyond
denoising. In this work, we propose the use of a hyperprior to model image
patches, in order to stabilize the estimation procedure. There are two main
advantages to the proposed restoration scheme: Firstly it is adapted to
diagonal degradation matrices, and in particular to missing data problems (e.g.
inpainting of missing pixels or zooming). Secondly it can deal with signal
dependent noise models, particularly suited to digital cameras. As such, the
scheme is especially adapted to computational photography. In order to
illustrate this point, we provide an application to high dynamic range imaging
from a single image taken with a modified sensor, which shows the effectiveness
of the proposed scheme.Comment: Some figures are reduced to comply with arxiv's size constraints.
Full size images are available as HAL technical report hal-01107519v5, IEEE
Transactions on Computational Imaging, 201
Image Deblurring and Super-resolution by Adaptive Sparse Domain Selection and Adaptive Regularization
As a powerful statistical image modeling technique, sparse representation has
been successfully used in various image restoration applications. The success
of sparse representation owes to the development of l1-norm optimization
techniques, and the fact that natural images are intrinsically sparse in some
domain. The image restoration quality largely depends on whether the employed
sparse domain can represent well the underlying image. Considering that the
contents can vary significantly across different images or different patches in
a single image, we propose to learn various sets of bases from a pre-collected
dataset of example image patches, and then for a given patch to be processed,
one set of bases are adaptively selected to characterize the local sparse
domain. We further introduce two adaptive regularization terms into the sparse
representation framework. First, a set of autoregressive (AR) models are
learned from the dataset of example image patches. The best fitted AR models to
a given patch are adaptively selected to regularize the image local structures.
Second, the image non-local self-similarity is introduced as another
regularization term. In addition, the sparsity regularization parameter is
adaptively estimated for better image restoration performance. Extensive
experiments on image deblurring and super-resolution validate that by using
adaptive sparse domain selection and adaptive regularization, the proposed
method achieves much better results than many state-of-the-art algorithms in
terms of both PSNR and visual perception.Comment: 35 pages. This paper is under review in IEEE TI
- …