The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called "perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.Comment: Accepted to CVPR 2018; Code and data available at https://www.github.com/richzhang/PerceptualSimilarit

Divisive Normalization from Wilson-Cowan Dynamics

Divisive Normalization and the Wilson-Cowan equations are influential models of neural interaction and saturation [Carandini and Heeger Nat.Rev.Neurosci. 2012; Wilson and Cowan Kybernetik 1973]. However, they have not been analytically related yet. In this work we show that Divisive Normalization can be obtained from the Wilson-Cowan model. Specifically, assuming that Divisive Normalization is the steady state solution of the Wilson-Cowan differential equation, we find that the kernel that controls neural interactions in Divisive Normalization depends on the Wilson-Cowan kernel but also has a signal-dependent contribution. A standard stability analysis of a Wilson-Cowan model with the parameters obtained from our relation shows that the Divisive Normalization solution is a stable node. This stability demonstrates the consistency of our steady state assumption, and is in line with the straightforward use of Divisive Normalization with time-varying stimuli. The proposed theory provides a physiological foundation (a relation to a dynamical network with fixed wiring among neurons) for the functional suggestions that have been done on the need of signal-dependent Divisive Normalization [e.g. in Coen-Cagli et al., PLoS Comp.Biol. 2012]. Moreover, this theory explains the modifications that had to be introduced ad-hoc in Gaussian kernels of Divisive Normalization in [Martinez et al. Front. Neurosci. 2019] to reproduce contrast responses. The proposed relation implies that the Wilson-Cowan dynamics also reproduces visual masking and subjective image distortion metrics, which up to now had been mainly explained via Divisive Normalization. Finally, this relation allows to apply to Divisive Normalization the methods which up to now had been developed for dynamical systems such as Wilson-Cowan networks

Biologically Inspired Mechanisms for Adversarial Robustness

A convolutional neural network strongly robust to adversarial perturbations at reasonable computational and performance cost has not yet been demonstrated. The primate visual ventral stream seems to be robust to small perturbations in visual stimuli but the underlying mechanisms that give rise to this robust perception are not understood. In this work, we investigate the role of two biologically plausible mechanisms in adversarial robustness. We demonstrate that the non-uniform sampling performed by the primate retina and the presence of multiple receptive fields with a range of receptive field sizes at each eccentricity improve the robustness of neural networks to small adversarial perturbations. We verify that these two mechanisms do not suffer from gradient obfuscation and study their contribution to adversarial robustness through ablation studies.Comment: 25 pages, 15 figure