Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method

Bilateral back-projection for single image super resolution

In this paper, a novel algorithm for single image super resolution is proposed. Back-projection [1] can minimize the reconstruction error with an efficient iterative procedure. Although it can produce visually appealing result, this method suffers from the chessboard effect and ringing effect, especially along strong edges. The underlining reason is that there is no edge guidance in the error correction process. Bilateral filtering can achieve edge-preserving image smoothing by adding the extra information from the feature domain. The basic idea is to do the smoothing on the pixels which are nearby both in space domain and in feature domain. The proposed bilateral back-projection algorithm strives to integrate the bilateral filtering into the back-projection method. In our approach, the back-projection process can be guided by the edge information to avoid across-edge smoothing, thus the chessboard effect and ringing effect along image edges are removed. Promising results can be obtained by the proposed bilateral back-projection method efficiently. 1

Perceptual modelling for 2D and 3D

Livrable D1.1 du projet ANR PERSEECe rapport a été réalisé dans le cadre du projet ANR PERSEE (n° ANR-09-BLAN-0170). Exactement il correspond au livrable D1.1 du projet

Learning Enriched Features for Real Image Restoration and Enhancement

With the goal of recovering high-quality image content from its degraded version, image restoration enjoys numerous applications, such as in surveillance, computational photography, medical imaging, and remote sensing. Recently, convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. Existing CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatially precise but contextually less robust results are achieved, while in the latter case, semantically reliable but spatially less accurate outputs are generated. In this paper, we present a novel architecture with the collective goals of maintaining spatially-precise high-resolution representations through the entire network and receiving strong contextual information from the low-resolution representations. The core of our approach is a multi-scale residual block containing several key elements: (a) parallel multi-resolution convolution streams for extracting multi-scale features, (b) information exchange across the multi-resolution streams, (c) spatial and channel attention mechanisms for capturing contextual information, and (d) attention based multi-scale feature aggregation. In a nutshell, our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Extensive experiments on five real image benchmark datasets demonstrate that our method, named as MIRNet, achieves state-of-the-art results for a variety of image processing tasks, including image denoising, super-resolution, and image enhancement. The source code and pre-trained models are available at https://github.com/swz30/MIRNet.Comment: Accepted for publication at ECCV 202

Defect-aware Super-resolution Thermography by Adversarial Learning

nfrared thermography is a valuable non-destructive tool for inspection of materials. It measures the surface temperature evolution, from which hidden defects may be detected. Yet, thermal cameras typically have a low native spatial resolution resulting in a blurry and low-quality thermal image sequence and videos. In this study, a novel adversarial deep learning framework, called Dual-IRT-GAN, is proposed for performing super-resolution tasks. The proposed Dual-IRT-GAN attempts to achieve the objective of improving local texture details, as well as highlighting defective regions. Technically speaking, the proposed model consists of two modules SEGnet and SRnet that carry out defect detection and super resolution tasks, respectively. By leveraging the defect information from SEGnet, SRnet is capable of generating plausible high-resolution thermal images with an enhanced focus on defect regions. The generated high-resolution images are then delivered to the discriminator for adversarial training using GAN's framework. The proposed Dual-IRT-GAN model, which is trained on an exclusive virtual dataset, is demonstrated on experimental thermographic data obtained from fiber reinforced polymers having a variety of defect types, sizes, and depths. The obtained results show its high performance in maintaining background color consistency and removing undesired noise, and in highlighting defect zones with finer detailed textures in high-resolution

Defect-aware Super-resolution Thermography by Adversarial Learning

nfrared thermography is a valuable non-destructive tool for inspection of materials. It measures the surface temperature evolution, from which hidden defects may be detected. Yet, thermal cameras typically have a low native spatial resolution resulting in a blurry and low-quality thermal image sequence and videos. In this study, a novel adversarial deep learning framework, called Dual-IRT-GAN, is proposed for performing super-resolution tasks. The proposed Dual-IRT-GAN attempts to achieve the objective of improving local texture details, as well as highlighting defective regions. Technically speaking, the proposed model consists of two modules SEGnet and SRnet that carry out defect detection and super resolution tasks, respectively. By leveraging the defect information from SEGnet, SRnet is capable of generating plausible high-resolution thermal images with an enhanced focus on defect regions. The generated high-resolution images are then delivered to the discriminator for adversarial training using GAN's framework. The proposed Dual-IRT-GAN model, which is trained on an exclusive virtual dataset, is demonstrated on experimental thermographic data obtained from fiber reinforced polymers having a variety of defect types, sizes, and depths. The obtained results show its high performance in maintaining background color consistency and removing undesired noise, and in highlighting defect zones with finer detailed textures in high-resolution