Natural and Realistic Single Image Super-Resolution with Explicit Natural Manifold Discrimination

Recently, many convolutional neural networks for single image super-resolution (SISR) have been proposed, which focus on reconstructing the high-resolution images in terms of objective distortion measures. However, the networks trained with objective loss functions generally fail to reconstruct the realistic fine textures and details that are essential for better perceptual quality. Recovering the realistic details remains a challenging problem, and only a few works have been proposed which aim at increasing the perceptual quality by generating enhanced textures. However, the generated fake details often make undesirable artifacts and the overall image looks somewhat unnatural. Therefore, in this paper, we present a new approach to reconstructing realistic super-resolved images with high perceptual quality, while maintaining the naturalness of the result. In particular, we focus on the domain prior properties of SISR problem. Specifically, we define the naturalness prior in the low-level domain and constrain the output image in the natural manifold, which eventually generates more natural and realistic images. Our results show better naturalness compared to the recent super-resolution algorithms including perception-oriented ones.Comment: Presented in CVPR 201

Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions

Generative Adversarial Networks (GANs) is a novel class of deep generative models which has recently gained significant attention. GANs learns complex and high-dimensional distributions implicitly over images, audio, and data. However, there exists major challenges in training of GANs, i.e., mode collapse, non-convergence and instability, due to inappropriate design of network architecture, use of objective function and selection of optimization algorithm. Recently, to address these challenges, several solutions for better design and optimization of GANs have been investigated based on techniques of re-engineered network architectures, new objective functions and alternative optimization algorithms. To the best of our knowledge, there is no existing survey that has particularly focused on broad and systematic developments of these solutions. In this study, we perform a comprehensive survey of the advancements in GANs design and optimization solutions proposed to handle GANs challenges. We first identify key research issues within each design and optimization technique and then propose a new taxonomy to structure solutions by key research issues. In accordance with the taxonomy, we provide a detailed discussion on different GANs variants proposed within each solution and their relationships. Finally, based on the insights gained, we present the promising research directions in this rapidly growing field.Comment: 42 pages, Figure 13, Table

Generative adversarial networks: an overview

Generative adversarial networks (GANs) provide a way to learn deep representations without extensively annotated training data. They achieve this by deriving backpropagation signals through a competitive process involving a pair of networks. The representations that can be learned by GANs may be used in a variety of applications, including image synthesis, semantic image editing, style transfer, image superresolution, and classification. The aim of this review article is to provide an overview of GANs for the signal processing community, drawing on familiar analogies and concepts where possible. In addition to identifying different methods for training and constructing GANs, we also point to remaining challenges in their theory and application

Diverse Image Generation with Very Low Resolution Conditioning

Traditionnellement, lorsqu’il s’agit de générer des images à haute résolution (HR) à partir d’images à basse résolution (LR), les méthodes proposées jusqu’à maintenant se sont principalement concentrées sur les techniques de super-résolution qui visent à récupérer l’image la plus probable à partir d’une image de basse qualité. En procédant de cette manière, on ignore le fait qu’il existe généralement de nombreuses versions valides d’images HR qui correspondent à une image LR donnée. L’objectif de ce travail est d’obtenir différentes versions d’images HR à partir d’une même image LR en utilisant un modèle adversarial génératif. On aborde ce problème sous deux angles différents. D’abord, on utilise des méthodes de super résolution, où en plus de l’image LR, le générateur peut être paramétré par une variable latente afin de produire différentes variations potentielles de l’image. Un tel conditionnement permet de moduler le générateur entre la récupération de l’image la plus proche de la vérité terrain et de variété d’images possibles. Les résultats démontrent notre supériorité en termes de reconstruction et de variété d’images hallucinées plausible par rapport à d’autres méthodes de l’état de l’art. La deuxième approche s’appuie sur les travaux de traduction d’image à image, en proposant une nouvelle approche où le modèle est conditionné sur une version LR du cible. Plus précisément, notre approche vise à transférer les détails fins d’une image source HR pour les adapter la structure générale, selon la version LR de celle-ci. On génère donc des images HR qui partagent les caractéristiques distinctives de l’image HR et qui correspond à l’image LR de la cible lors de la réduction d’échelle. Cette méthode diffère des méthodes précédentes qui se concentrent plutôt sur la traduction d’un style d’image donné en un contenu cible. Les résultats qualitatifs et quantitatifs démontrent des améliorations en termes de qualité visuelle, de diversité et de couverture par rapport aux méthodes de pointe telles que Stargan-v2.Traditionally, when it comes to generating high-resolution (HR) images from a low-resolution(LR) images, the methods proposed so far have mainly focused on super-resolution techniques that aim at recovering the most probable image from low-quality image. Doing so ignores the fact that there are usually many valid versions of HR images that match a given LR image. The objective of this work is to obtain different versions of HR images from the same LR imageusing a generative adversarial model. We approach this problem from two different angles. First, we use super-resolution methods, where in addition to the LR image, the generator can be parameterized by a latent variable to produce different potential variations of the image. Such a conditioning allows to modulate the generator between retrieving the closest image to the ground truth and a variety of possible images. The results demonstrate our superiority in terms of reconstruction and variety of plausible hallucinated images compared to other state-of-the-art methods. The second approach builds on the work of image-to-image translation, by proposing a new approach where the model is conditioned on a LR version of the target. More precisely, our approach aims at transferring the fine details of an HR source image to fit the general structure, according to the LR version of it. We therefore generate HR images that share the distinctive features of the HR image and match the LR image of the target duringdownscaling. This method differs from previous methods that focus instead on translatinga given image style into target content. Qualitative and quantitative results demonstrate improvements in visual quality, diversity, and coverage over state-of-the-art methods such asStargan-v2