27 research outputs found
Learning Raw Image Denoising with Bayer Pattern Unification and Bayer Preserving Augmentation
In this paper, we present new data pre-processing and augmentation techniques
for DNN-based raw image denoising. Compared with traditional RGB image
denoising, performing this task on direct camera sensor readings presents new
challenges such as how to effectively handle various Bayer patterns from
different data sources, and subsequently how to perform valid data augmentation
with raw images. To address the first problem, we propose a Bayer pattern
unification (BayerUnify) method to unify different Bayer patterns. This allows
us to fully utilize a heterogeneous dataset to train a single denoising model
instead of training one model for each pattern. Furthermore, while it is
essential to augment the dataset to improve model generalization and
performance, we discovered that it is error-prone to modify raw images by
adapting augmentation methods designed for RGB images. Towards this end, we
present a Bayer preserving augmentation (BayerAug) method as an effective
approach for raw image augmentation. Combining these data processing technqiues
with a modified U-Net, our method achieves a PSNR of 52.11 and a SSIM of 0.9969
in NTIRE 2019 Real Image Denoising Challenge, demonstrating the
state-of-the-art performance. Our code is available at
https://github.com/Jiaming-Liu/BayerUnifyAug.Comment: Accepted by CVPRW 201
Towards practical deep learning based image restoration model
Image Restoration (IR) is a task of reconstructing the latent image from its degraded observations. It has become an important research area in computer vision and image processing, and has wide applications in the imaging industry. Conventional methods apply inverse filtering or optimization-based approaches to restore images corrupted in ideal cases. The limited restoration performance on ill-posed problems and the low-efficient iterative optimization processes prevents such algorithms from being deployed to more complicated industry applications. Recently, the advanced deep Convolutional Neural Networks (CNNs) begin to model the image restoration as learning and inferring the posterior probability in a regression model, and successfully achieved remarkable performance. However, due to the data-driven nature, the models trained with simple synthetic paired data (e.g, bicubic interpolation or Gaussian noises) cannot be well adapted to more complicated inputs from real data domains. Besides, acquiring real paired data for training such models is also very challenging.
In this dissertation, we discuss the data manipulation and model adaptability of the deep learning based image restoration tasks. Specifically, we study improving the model adaptability by understanding the domain difference between its training data and its expected testing data. We argue that the cause of image degradation can be various due to multiple imaging and transmission pipelines. Though complicated to analyze, for some specific imaging problems, we can still improve the performance of deep restoration models on unseen testing data by resolving the data domain differences implied in the image acquisition and formation pipeline. Our analysis focuses on digital image denoising, image restoration from more complicated degradation types beyond denoising and multi-image inpainting. For all tasks, the proposed training or adaptation strategies, based on the physical principle of the degradation formation or based on geometric assumption of the image, achieve a reasonable improvement on the restoration performance.
For image denoising, we discuss the influence of the Bayer pattern of the Camera Filter Array (CFA) and the image demosaicing process on the adaptability of the deep denoising models. Specifically, for the task of denoising RAW sensor observations, we find that unifying and augmenting the data Bayer pattern during training and testing is an efficient strategy to make the well-trained denoising model Bayer-invariant. Additionally, for the RGB image denoising, demosaicing the noisy RAW images with Bayer patterns will result in the spatial-correlation of pixel noises. Therefore, we propose the pixel-shuffle down-sampling approach to break down this spatial correlation, and make the Gaussian-trained denoiser more adaptive to real RGB noisy images.
Beyond denoising, we explain a more complicated degradation process involving diffraction when there are some occlusions on the imaging lens. One example is a novel imaging model called Under-Display Camera (UDC). From the perspective of optical analysis, we study the physics-based imaging processing method by deriving the forward model of the degradation, and synthesize the paired data for both conventional and deep denoising pipeline. Experiments demonstrate the effectiveness of the forward model and the deep restoration model trained with synthetic data achieves visually similar performance to the one trained with real paired images.
Last, we further discuss reference-based image inpainting to restore the missing regions in the target image by reusing contents from the source image. Due to the color and spatial misalignment between the two images, we first initialize the warping by using multi-homography registration, and then propose a content-preserving Color and Spatial Transformer (CST) to refine the misalignment and color difference. We designed the CST to be scale-robust, so it mitigates the warping problems when the model is applied to testing images with different resolution. We synthesize realistic data while training the CST, and it suggests the inpainting pipeline achieves a more robust restoration performance with the proposed CST
Joint Demosaicking and Denoising in the Wild: The Case of Training Under Ground Truth Uncertainty
Image demosaicking and denoising are the two key fundamental steps in digital
camera pipelines, aiming to reconstruct clean color images from noisy luminance
readings. In this paper, we propose and study Wild-JDD, a novel learning
framework for joint demosaicking and denoising in the wild. In contrast to
previous works which generally assume the ground truth of training data is a
perfect reflection of the reality, we consider here the more common imperfect
case of ground truth uncertainty in the wild. We first illustrate its
manifestation as various kinds of artifacts including zipper effect, color
moire and residual noise. Then we formulate a two-stage data degradation
process to capture such ground truth uncertainty, where a conjugate prior
distribution is imposed upon a base distribution. After that, we derive an
evidence lower bound (ELBO) loss to train a neural network that approximates
the parameters of the conjugate prior distribution conditioned on the degraded
input. Finally, to further enhance the performance for out-of-distribution
input, we design a simple but effective fine-tuning strategy by taking the
input as a weakly informative prior. Taking into account ground truth
uncertainty, Wild-JDD enjoys good interpretability during optimization.
Extensive experiments validate that it outperforms state-of-the-art schemes on
joint demosaicking and denoising tasks on both synthetic and realistic raw
datasets.Comment: Accepted by AAAI202
CycleISP: Real Image Restoration via Improved Data Synthesis
The availability of large-scale datasets has helped unleash the true
potential of deep convolutional neural networks (CNNs). However, for the
single-image denoising problem, capturing a real dataset is an unacceptably
expensive and cumbersome procedure. Consequently, image denoising algorithms
are mostly developed and evaluated on synthetic data that is usually generated
with a widespread assumption of additive white Gaussian noise (AWGN). While the
CNNs achieve impressive results on these synthetic datasets, they do not
perform well when applied on real camera images, as reported in recent
benchmark datasets. This is mainly because the AWGN is not adequate for
modeling the real camera noise which is signal-dependent and heavily
transformed by the camera imaging pipeline. In this paper, we present a
framework that models camera imaging pipeline in forward and reverse
directions. It allows us to produce any number of realistic image pairs for
denoising both in RAW and sRGB spaces. By training a new image denoising
network on realistic synthetic data, we achieve the state-of-the-art
performance on real camera benchmark datasets. The parameters in our model are
~5 times lesser than the previous best method for RAW denoising. Furthermore,
we demonstrate that the proposed framework generalizes beyond image denoising
problem e.g., for color matching in stereoscopic cinema. The source code and
pre-trained models are available at https://github.com/swz30/CycleISP.Comment: CVPR 2020 (Oral