92,584 research outputs found
Echo State Queueing Network: a new reservoir computing learning tool
In the last decade, a new computational paradigm was introduced in the field
of Machine Learning, under the name of Reservoir Computing (RC). RC models are
neural networks which a recurrent part (the reservoir) that does not
participate in the learning process, and the rest of the system where no
recurrence (no neural circuit) occurs. This approach has grown rapidly due to
its success in solving learning tasks and other computational applications.
Some success was also observed with another recently proposed neural network
designed using Queueing Theory, the Random Neural Network (RandNN). Both
approaches have good properties and identified drawbacks. In this paper, we
propose a new RC model called Echo State Queueing Network (ESQN), where we use
ideas coming from RandNNs for the design of the reservoir. ESQNs consist in
ESNs where the reservoir has a new dynamics inspired by recurrent RandNNs. The
paper positions ESQNs in the global Machine Learning area, and provides
examples of their use and performances. We show on largely used benchmarks that
ESQNs are very accurate tools, and we illustrate how they compare with standard
ESNs.Comment: Proceedings of the 10th IEEE Consumer Communications and Networking
Conference (CCNC), Las Vegas, USA, 201
Structured Sequence Modeling with Graph Convolutional Recurrent Networks
This paper introduces Graph Convolutional Recurrent Network (GCRN), a deep
learning model able to predict structured sequences of data. Precisely, GCRN is
a generalization of classical recurrent neural networks (RNN) to data
structured by an arbitrary graph. Such structured sequences can represent
series of frames in videos, spatio-temporal measurements on a network of
sensors, or random walks on a vocabulary graph for natural language modeling.
The proposed model combines convolutional neural networks (CNN) on graphs to
identify spatial structures and RNN to find dynamic patterns. We study two
possible architectures of GCRN, and apply the models to two practical problems:
predicting moving MNIST data, and modeling natural language with the Penn
Treebank dataset. Experiments show that exploiting simultaneously graph spatial
and dynamic information about data can improve both precision and learning
speed
Conditional Random Fields as Recurrent Neural Networks
Pixel-level labelling tasks, such as semantic segmentation, play a central
role in image understanding. Recent approaches have attempted to harness the
capabilities of deep learning techniques for image recognition to tackle
pixel-level labelling tasks. One central issue in this methodology is the
limited capacity of deep learning techniques to delineate visual objects. To
solve this problem, we introduce a new form of convolutional neural network
that combines the strengths of Convolutional Neural Networks (CNNs) and
Conditional Random Fields (CRFs)-based probabilistic graphical modelling. To
this end, we formulate mean-field approximate inference for the Conditional
Random Fields with Gaussian pairwise potentials as Recurrent Neural Networks.
This network, called CRF-RNN, is then plugged in as a part of a CNN to obtain a
deep network that has desirable properties of both CNNs and CRFs. Importantly,
our system fully integrates CRF modelling with CNNs, making it possible to
train the whole deep network end-to-end with the usual back-propagation
algorithm, avoiding offline post-processing methods for object delineation. We
apply the proposed method to the problem of semantic image segmentation,
obtaining top results on the challenging Pascal VOC 2012 segmentation
benchmark.Comment: This paper is published in IEEE ICCV 201
A mathematical analysis of the effects of Hebbian learning rules on the dynamics and structure of discrete-time random recurrent neural networks
We present a mathematical analysis of the effects of Hebbian learning in
random recurrent neural networks, with a generic Hebbian learning rule
including passive forgetting and different time scales for neuronal activity
and learning dynamics. Previous numerical works have reported that Hebbian
learning drives the system from chaos to a steady state through a sequence of
bifurcations. Here, we interpret these results mathematically and show that
these effects, involving a complex coupling between neuronal dynamics and
synaptic graph structure, can be analyzed using Jacobian matrices, which
introduce both a structural and a dynamical point of view on the neural network
evolution. Furthermore, we show that the sensitivity to a learned pattern is
maximal when the largest Lyapunov exponent is close to 0. We discuss how neural
networks may take advantage of this regime of high functional interest
Variational inference formulation for a model-free simulation of a dynamical system with unknown parameters by a recurrent neural network
We propose a recurrent neural network for a "model-free" simulation of a
dynamical system with unknown parameters without prior knowledge. The deep
learning model aims to jointly learn the nonlinear time marching operator and
the effects of the unknown parameters from a time series dataset. We assume
that the time series data set consists of an ensemble of trajectories for a
range of the parameters. The learning task is formulated as a statistical
inference problem by considering the unknown parameters as random variables. A
latent variable is introduced to model the effects of the unknown parameters,
and a variational inference method is employed to simultaneously train
probabilistic models for the time marching operator and an approximate
posterior distribution for the latent variable. Unlike the classical
variational inference, where a factorized distribution is used to approximate
the posterior, we employ a feedforward neural network supplemented by an
encoder recurrent neural network to develop a more flexible probabilistic
model. The approximate posterior distribution makes an inference on a
trajectory to identify the effects of the unknown parameters. The time marching
operator is approximated by a recurrent neural network, which takes a latent
state sampled from the approximate posterior distribution as one of the input
variables, to compute the time evolution of the probability distribution
conditioned on the latent variable. In the numerical experiments, it is shown
that the proposed variational inference model makes a more accurate simulation
compared to the standard recurrent neural networks. It is found that the
proposed deep learning model is capable of correctly identifying the dimensions
of the random parameters and learning a representation of complex time series
data
Automated segmentation on the entire cardiac cycle using a deep learning work-flow
The segmentation of the left ventricle (LV) from CINE MRI images is essential
to infer important clinical parameters. Typically, machine learning algorithms
for automated LV segmentation use annotated contours from only two cardiac
phases, diastole, and systole. In this work, we present an analysis work-flow
for fully-automated LV segmentation that learns from images acquired through
the cardiac cycle. The workflow consists of three components: first, for each
image in the sequence, we perform an automated localization and subsequent
cropping of the bounding box containing the cardiac silhouette. Second, we
identify the LV contours using a Temporal Fully Convolutional Neural Network
(T-FCNN), which extends Fully Convolutional Neural Networks (FCNN) through a
recurrent mechanism enforcing temporal coherence across consecutive frames.
Finally, we further defined the boundaries using either one of two components:
fully-connected Conditional Random Fields (CRFs) with Gaussian edge potentials
and Semantic Flow. Our initial experiments suggest that significant improvement
in performance can potentially be achieved by using a recurrent neural network
component that explicitly learns cardiac motion patterns whilst performing LV
segmentation.Comment: 6 pages, 2 figures, published on IEEE Xplor
- …