Learning to Predict Image-based Rendering Artifacts with Respect to a Hidden Reference Image

Image metrics predict the perceived per-pixel difference between a reference image and its degraded (e. g., re-rendered) version. In several important applications, the reference image is not available and image metrics cannot be applied. We devise a neural network architecture and training procedure that allows predicting the MSE, SSIM or VGG16 image difference from the distorted image alone while the reference is not observed. This is enabled by two insights: The first is to inject sufficiently many un-distorted natural image patches, which can be found in arbitrary amounts and are known to have no perceivable difference to themselves. This avoids false positives. The second is to balance the learning, where it is carefully made sure that all image errors are equally likely, avoiding false negatives. Surprisingly, we observe, that the resulting no-reference metric, subjectively, can even perform better than the reference-based one, as it had to become robust against mis-alignments. We evaluate the effectiveness of our approach in an image-based rendering context, both quantitatively and qualitatively. Finally, we demonstrate two applications which reduce light field capture time and provide guidance for interactive depth adjustment.Comment: 13 pages, 11 figure

Multi-Task Learning Approach for Natural Images' Quality Assessment

Blind image quality assessment (BIQA) is a method to predict the quality of a natural image without the presence of a reference image. Current BIQA models typically learn their prediction separately for different image distortions, ignoring the relationship between the learning tasks. As a result, a BIQA model may has great prediction performance for natural images affected by one particular type of distortion but is less effective when tested on others. In this paper, we propose to address this limitation by training our BIQA model simultaneously under different distortion conditions using multi-task learning (MTL) technique. Given a set of training images, our Multi-Task Learning based Image Quality assessment (MTL-IQ) model first extracts spatial domain BIQA features. The features are then used as an input to a trace-norm regularisation based MTL framework to learn prediction models for different distortion classes simultaneously. For a test image of a known distortion, MTL-IQ selects a specific trained model to predict the image’s quality score. For a test image of an unknown distortion, MTLIQ first estimates the amount of each distortion present in the image using a support vector classifier. The probability estimates are then used to weigh the image prediction scores from different trained models. The weighted scores are then pooled to obtain the final image quality score. Experimental results on standard image quality assessment (IQA) databases show that MTL-IQ is highly correlated with human perceptual measures of image quality. It also obtained higher prediction performance in both overall and individual distortion cases compared to current BIQA models