39 research outputs found
Laplacian-Steered Neural Style Transfer
Neural Style Transfer based on Convolutional Neural Networks (CNN) aims to
synthesize a new image that retains the high-level structure of a content
image, rendered in the low-level texture of a style image. This is achieved by
constraining the new image to have high-level CNN features similar to the
content image, and lower-level CNN features similar to the style image. However
in the traditional optimization objective, low-level features of the content
image are absent, and the low-level features of the style image dominate the
low-level detail structures of the new image. Hence in the synthesized image,
many details of the content image are lost, and a lot of inconsistent and
unpleasing artifacts appear. As a remedy, we propose to steer image synthesis
with a novel loss function: the Laplacian loss. The Laplacian matrix
("Laplacian" in short), produced by a Laplacian operator, is widely used in
computer vision to detect edges and contours. The Laplacian loss measures the
difference of the Laplacians, and correspondingly the difference of the detail
structures, between the content image and a new image. It is flexible and
compatible with the traditional style transfer constraints. By incorporating
the Laplacian loss, we obtain a new optimization objective for neural style
transfer named Lapstyle. Minimizing this objective will produce a stylized
image that better preserves the detail structures of the content image and
eliminates the artifacts. Experiments show that Lapstyle produces more
appealing stylized images with less artifacts, without compromising their
"stylishness".Comment: Accepted by the ACM Multimedia Conference (MM) 2017. 9 pages, 65
figure
Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks
It is a long standing question how biological systems transform visual inputs to robustly infer high level visual information. Research in the last decades has established that much of the underlying computations take place in a hierarchical fashion along the ventral visual pathway. However, the exact processing stages along this hierarchy are difficult to characterise. Here we present a method to generate stimuli that will allow a principled description of the processing stages along the ventral stream. We introduce a new parametric texture model based on the powerful feature spaces of convolutional neural networks optimised for object recognition. We show that constraining spatial summary statistic on feature maps suffices to synthesise high quality natural textures. Moreover we establish that our texture representations continuously disentangle high level visual information and demonstrate that the hierarchical parameterisation of the texture model naturally enables us to generate novel types of stimuli for systematically probing mid-level vision
Texture and art with deep neural networks
Although the study of biological vision and computer vision attempt to understand powerful visual information processing from different angles, they have a long history of informing each other. Recent advances in texture synthesis that were motivated by visual neuroscience have led to a substantial advance in image synthesis and manipulation in computer vision using convolutional neural networks (CNNs). Here, we review these recent advances and discuss how they can in turn inspire new research in visual perception and computational neuroscience
A Neural Algorithm of Artistic Style
In fine art, especially painting, humans have mastered the skill to create unique visual experiences by composing a complex interplay between the content and style of an image. The algorithmic basis of this process is unknown and there exists no artificial system with similar capabilities. Recently, a class of biologically inspired vision models called Deep Neural Networks have demonstrated near-human performance in complex visual tasks such as object and face recognition. Here we introduce an artificial system based on a Deep Neural Network that creates artistic images of high perceptual quality. The system can separate and recombine the content and style of arbitrary images, providing a neural algorithm for the creation of artistic images. In light of recent studies using fMRI and electrophysiology that have shown striking similarities between performance-optimised artificial neural networks and biological vision, our work offers a path towards an algorithmic understanding of how humans create and perceive artistic imagery. The algorithm introduces a novel class of stimuli that could be used to test specific computational hypotheses about the perceptual processing of artistic style
Texture Synthesis Using Convolutional Neural Networks
Here we introduce a new model of natural textures based on the feature spaces of convolutional neural networks optimised for object recognition. Samples from the model are of high perceptual quality demonstrating the generative power of neural networks trained in a purely discriminative fashion. Within the model, textures are represented by the correlations between feature maps in several layers of the network. We show that across layers the texture representations increasingly capture the statistical properties of natural images while making object information more and more explicit. The model provides a new tool to generate stimuli for neuroscience and might offer insights into the deep representations learned by convolutional neural networks
Image Style Transfer Using Convolutional Neural Networks
Rendering the semantic content of an image in different styles is a difficult image processing task. Arguably, a major limiting factor for previous approaches has been the lack of image representations that explicitly represent semantic information and, thus, allow to separate image content from style. Here we use image representations derived from Convolutional Neural Networks optimised for object recognition, which make high level image information explicit. We introduce A Neural Algorithm of Artistic Style that can separate and recombine the image content and style of natural images. The algorithm allows us to produce new images of high perceptual quality that combine the content of an arbitrary photograph with the appearance of numerous well-known artworks. Our results provide new insights into the deep image representations learned by Convolutional Neural Networks and demonstrate their potential for high level image synthesis and manipulation
Synaptic unreliability facilitates information transmission in balanced cortical populations
Cortical neurons fire in a highly irregular manner, suggesting that their input is tightly balanced and changes in presynaptic firing rate are encoded primarily in the variance of the postsynaptic currents. Here we show that such balance has a surprising effect on information transmission: Synaptic unreliability which is ubiquitous in cortex and usually thought to impair neural communication actually increases the information rate. We show that the beneficial effect of noise is based on a very general mechanism which contrary to stochastic resonance does not rely on a threshold nonlinearity
Synaptic unreliability facilitates information transmission in balanced cortical populations
Cortical neurons fire in a highly irregular manner, suggesting that their input is tightly balanced and changes in presynaptic firing rate are encoded primarily in the variance of the postsynaptic currents. Here we show that such balance has a surprising effect on information transmission: Synaptic unreliability – which is ubiquitous in cortex and usually thought to impair neural communication – actually increases the information rate. We show that the beneficial effect of noise is based on a very general mechanism which contrary to stochastic resonance does not rely on a threshold nonlinearity
Synaptic unreliability facilitates information transmission in balanced cortical populations
Synaptic unreliability is one of the major sources of biophysical noise in the brain. In the context of neural information processing, it is a central question how neural systems can afford this unreliability. Here we examined how synaptic noise affects signal transmission in cortical circuits, where excitation and inhibition are thought to be tightly balanced. Surprisingly, we found that in this balanced state synaptic response variability actually facilitates information transmission, rather than impairing it. In particular, the transmission of fast-varying signals benefits from synaptic noise, as it instantaneously increases the amount of information shared between presynaptic signal and postsynaptic current. This finding provides a parsimonious explanation why cortex can afford to operate with noisy synapses