Quality Aware Generative Adversarial Networks

Generative Adversarial Networks (GANs) have become a very popular tool for implicitly learning high-dimensional probability distributions. Several improvements have been made to the original GAN formulation to address some of its shortcomings like mode collapse, convergence issues, entanglement, poor visual quality etc. While a significant effort has been directed towards improving the visual quality of images generated by GANs, it is rather surprising that objective image quality metrics have neither been employed as cost functions nor as regularizers in GAN objective functions. In this work, we show how a distance metric that is a variant of the Structural SIMilarity (SSIM) index (a popular full-reference image quality assessment algorithm), and a novel quality aware discriminator gradient penalty function that is inspired by the Natural Image Quality Evaluator (NIQE, a popular no-reference image quality assessment algorithm) can each be used as excellent regularizers for GAN objective functions. Specifically, we demonstrate state-of-the-art performance using the Wasserstein GAN gradient penalty (WGAN-GP) framework over CIFAR-10, STL10 and CelebA datasets.Comment: 10 pages, NeurIPS 201

A weighted optimization for Fourier Ptychographic Microscopy

ourier ptychography can be implemented as a phase retrieval optimization algorithm that iteratively solves for high resolution spectrum from low resolution images. In prior art, all the low resolution images were considered equally in the optimization. In this paper, we propose a weighted optimization algorithm to enhance the quality of reconstruction with the same convergence speed. Our method is motivated by the observation that bright field and dark field low resolution images have significantly different pixel intensities. Therefore, we weight their estimated error differently in the optimization. Though the proposed method is both conceptually and computationally simple, it dramatically improves the quality of reconstruction. We also show that the weighted optimization algorithm converges to a lower mean squared error value compared to the conventional optimization. We validate our approach on several low resolution images from an experimental dataset

Completely Blind Quality Assessment of User Generated Video Content

In this work, we address the challenging problem of completely blind video quality assessment (BVQA) of user generated content (UGC). The challenge is twofold since the quality prediction model is oblivious of human opinion scores, and there are no well-defined distortion models for UGC content. Our solution is inspired by a recent computational neuroscience model which hypothesizes that the human visual system (HVS) transforms a natural video input to follow a straighter temporal trajectory in the perceptual domain. A bandpass filter based computational model of the lateral geniculate nucleus (LGN) and V1 regions of the HVS was used to validate the perceptual straightening hypothesis. We hypothesize that distortions in natural videos lead to loss in straightness (or increased curvature) in their transformed representations in the HVS. We provide extensive empirical evidence to validate our hypothesis. We quantify the loss in straightness as a measure of temporal quality, and show that this measure delivers acceptable quality prediction performance on its own. Further, the temporal quality measure is combined with a state-of-the-art blind spatial (image) quality metric to design a blind video quality predictor that we call STraightness Evaluation Metric (STEM). STEM is shown to deliver state-of-the-art performance over the class of BVQA algorithms on five UGC VQA datasets including KoNViD-1K, LIVE-Qualcomm, LIVE-VQC, CVD and YouTube-UGC. Importantly, our solution is completely blind i.e., training-free, generalizes very well, is explainable, has few tunable parameters, and is simple and easy to implement. © 1992-2012 IEEE

Medical Image Super Resolution- Fourier Ptychographic Microscopy

Fourier Ptychographic Micrcoscopy(FPM) is a computational imaging technique for improving the resolution of a microscope. Fourier ptychography can be implemented as a phase retrieval optimiza- tion algorithm that iteratively solves for high resolution spectrum from low resolution images. In prior art, all the low resolution images were considered equally in the optimization. In this thesis, We propose a weighted optimization algorithm to enhance the quality of reconstruction with the same convergence speed. Our method is motivated by the observation that bright field and dark field low resolution images have signifficantly different pixel intensities. Therefore, We weight their estimated error differently in the optimization. Though the proposed method is both conceptually and computationally simple, it dramatically improved the quality of reconstruction. We have also applied deep convolutional neural network for reconstructing the high resolution image from the set of low resolution images. We have showed improvement in terms of SSIM and PSNR

Quality Aware Generative Adversarial Networks

Generative Adversarial Networks (GANs) have become a very popular tool for implicitly learning high-dimensional probability distributions. Several improvements have been made to the original GAN formulation to address some of its shortcomings like mode collapse, convergence issues, entanglement, poor visual quality etc. While a significant effort has been directed towards improving the visual quality of images generated by GANs, it is rather surprising that objective image quality metrics have neither been employed as cost functions nor as regularizers in GAN objective functions. In this work, we show how a distance metric that is a variant of the Structural SIMilarity (SSIM) index (a popular full-reference image quality assessment algorithm), and a novel quality aware discriminator gradient penalty function that is inspired by the Natural Image Quality Evaluator (NIQE, a popular no-reference image quality assessment algorithm) can each be used as excellent regularizers for GAN objective functions. Specifically, we demonstrate state-of-the-art performance using the Wasserstein GAN gradient penalty (WGAN-GP) framework over CIFAR-10, STL10 and CelebA datasets

Improving the Visual Quality of Generative Adversarial Network (GAN)-Generated Images Using the Multi-Scale Structural Similarity Index

This paper presents a simple yet effective method to improve the visual quality of Generative Adversarial Network (GAN) generated images. In typical GAN architectures, the discriminator block is designed mainly to capture the class-specific content from images without explicitly imposing constraints on the visual quality of the generated images. A key insight from the image quality assessment literature is that natural scenes possess a very unique local structural and (hence) statistical signature, and that distortions affect this signature. We translate this insight into a constraint on the loss function of the discriminator in the GAN architecture with the goal of improving the visual quality of the generated images. Specifically, this constraint is based on the Multi-scale Structural Similarity (MS-SSIM) index to guarantee local structural and statistical integrity. We train GAN s (Boundary Equilibrium GANs, to be precise,) using the proposed approach on popular face and car image databases and demonstrate the improvement relative to standard training approaches both visually and quantitatively

Improving the Visual Quality of Video Frame Prediction Models Using the Perceptual Straightening Hypothesis

We present a simple and effective method to improve the visual quality of the predicted frames in a frame prediction model. A recent neuroscience study hypothesizes that the perceptual representations of a sequence of frames extracted from a natural video follow a straight temporal trajectory. The perceptual representations of a sequence of video frames are found using a computational model of the LGN and V1 areas of the human visual system. In this work, we leverage the strength of this perceptual straightening model to formulate a novel objective function for video frame prediction. In general, a frame prediction model takes past frames as input and predicts the future frame. We enforce the perceptual straightness constraint through adversarial training by introducing the proposed novel quality aware discriminator loss. Our quality aware discriminator imposes the linear relationship between the perceptual representation of the predicted frame and the perceptual representations of the past frames.Specifically, we claim that imposing a perceptual straightness constraint through the discriminator helps in predicting (i.e., generating) video frames that look more natural and therefore, having a higher perceptual quality. We demonstrate the effectiveness of our proposed objective function on two popular video datasets using three different frame prediction models. These experiments show that our solution is both consistent and stable, thereby allowing it to be integrated with other frame prediction models as well. © 1994-2012 IEEE

Generative adversarial network-based photoacoustic image reconstruction from bandlimited and limited-view data

Ultrasound transducers used in photoacoustic imaging are bandlimited and have a limited detection angle, which degrades the reconstructed image quality. One way to address this problem is to have transducers with multiple frequency bands with acquisition around the sample. This approach is expensive and it is not feasible for systems with a handheld probe using a linear transducer array. In this work, we aim to develop a deep learning method for photoacoustic reconstruction from bandlimited and limited-view data. We have developed a Generative Adversarial Networks (GANs)-based framework conditioned with a photoacoustic measurement for image reconstruction. In this way, the transducer used in the measurement can be incorporated and the generator trying to compensate for the limited data problem. We have developed the model for a handheld photoacoustic system using a linear transducer array with 128 elements having a center frequency of 7MHz and -6dB bandwidth from 4-10 MHz. We trained the network using simulated blood vessel images and tested it on in vivo measurements from the human forearm. We have compared the reconstructed images using the proposed method with the time-reversal on simulated data for detection using a bandlimited and directional transducer and compared it using the ground truth. Further, we compare our results to the in vivo images from the system which uses a delay and sum algorithm. The results from both simulations and experiments show that the proposed approach can remove bandlimited and limited view artifacts and can achieve a better image quality. © COPYRIGHT SPIE. Downloading of the abstract is permitted for personal use only