29,150 research outputs found
Dynamic Analysis and an Eigen Initializer for Recurrent Neural Networks
In recurrent neural networks, learning long-term dependency is the main
difficulty due to the vanishing and exploding gradient problem. Many
researchers are dedicated to solving this issue and they proposed many
algorithms. Although these algorithms have achieved great success,
understanding how the information decays remains an open problem. In this
paper, we study the dynamics of the hidden state in recurrent neural networks.
We propose a new perspective to analyze the hidden state space based on an
eigen decomposition of the weight matrix. We start the analysis by linear state
space model and explain the function of preserving information in activation
functions. We provide an explanation for long-term dependency based on the
eigen analysis. We also point out the different behavior of eigenvalues for
regression tasks and classification tasks. From the observations on
well-trained recurrent neural networks, we proposed a new initialization method
for recurrent neural networks, which improves consistently performance. It can
be applied to vanilla-RNN, LSTM, and GRU. We test on many datasets, such as
Tomita Grammars, pixel-by-pixel MNIST datasets, and machine translation
datasets (Multi30k). It outperforms the Xavier initializer and kaiming
initializer as well as other RNN-only initializers like IRNN and sp-RNN in
several tasks
Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks
We propose a method for lossy image compression based on recurrent,
convolutional neural networks that outperforms BPG (4:2:0 ), WebP, JPEG2000,
and JPEG as measured by MS-SSIM. We introduce three improvements over previous
research that lead to this state-of-the-art result. First, we show that
training with a pixel-wise loss weighted by SSIM increases reconstruction
quality according to several metrics. Second, we modify the recurrent
architecture to improve spatial diffusion, which allows the network to more
effectively capture and propagate image information through the network's
hidden state. Finally, in addition to lossless entropy coding, we use a
spatially adaptive bit allocation algorithm to more efficiently use the limited
number of bits to encode visually complex image regions. We evaluate our method
on the Kodak and Tecnick image sets and compare against standard codecs as well
recently published methods based on deep neural networks
Conditional Random Fields as Recurrent Neural Networks
Pixel-level labelling tasks, such as semantic segmentation, play a central
role in image understanding. Recent approaches have attempted to harness the
capabilities of deep learning techniques for image recognition to tackle
pixel-level labelling tasks. One central issue in this methodology is the
limited capacity of deep learning techniques to delineate visual objects. To
solve this problem, we introduce a new form of convolutional neural network
that combines the strengths of Convolutional Neural Networks (CNNs) and
Conditional Random Fields (CRFs)-based probabilistic graphical modelling. To
this end, we formulate mean-field approximate inference for the Conditional
Random Fields with Gaussian pairwise potentials as Recurrent Neural Networks.
This network, called CRF-RNN, is then plugged in as a part of a CNN to obtain a
deep network that has desirable properties of both CNNs and CRFs. Importantly,
our system fully integrates CRF modelling with CNNs, making it possible to
train the whole deep network end-to-end with the usual back-propagation
algorithm, avoiding offline post-processing methods for object delineation. We
apply the proposed method to the problem of semantic image segmentation,
obtaining top results on the challenging Pascal VOC 2012 segmentation
benchmark.Comment: This paper is published in IEEE ICCV 201
Recurrent Convolutional Neural Networks for Scene Parsing
Scene parsing is a technique that consist on giving a label to all pixels in
an image according to the class they belong to. To ensure a good visual
coherence and a high class accuracy, it is essential for a scene parser to
capture image long range dependencies. In a feed-forward architecture, this can
be simply achieved by considering a sufficiently large input context patch,
around each pixel to be labeled. We propose an approach consisting of a
recurrent convolutional neural network which allows us to consider a large
input context, while limiting the capacity of the model. Contrary to most
standard approaches, our method does not rely on any segmentation methods, nor
any task-specific features. The system is trained in an end-to-end manner over
raw pixels, and models complex spatial dependencies with low inference cost. As
the context size increases with the built-in recurrence, the system identifies
and corrects its own errors. Our approach yields state-of-the-art performance
on both the Stanford Background Dataset and the SIFT Flow Dataset, while
remaining very fast at test time
Generative Image Modeling Using Spatial LSTMs
Modeling the distribution of natural images is challenging, partly because of
strong statistical dependencies which can extend over hundreds of pixels.
Recurrent neural networks have been successful in capturing long-range
dependencies in a number of problems but only recently have found their way
into generative image models. We here introduce a recurrent image model based
on multi-dimensional long short-term memory units which are particularly suited
for image modeling due to their spatial structure. Our model scales to images
of arbitrary size and its likelihood is computationally tractable. We find that
it outperforms the state of the art in quantitative comparisons on several
image datasets and produces promising results when used for texture synthesis
and inpainting
- …