358,639 research outputs found
Group Invariance, Stability to Deformations, and Complexity of Deep Convolutional Representations
The success of deep convolutional architectures is often attributed in part
to their ability to learn multiscale and invariant representations of natural
signals. However, a precise study of these properties and how they affect
learning guarantees is still missing. In this paper, we consider deep
convolutional representations of signals; we study their invariance to
translations and to more general groups of transformations, their stability to
the action of diffeomorphisms, and their ability to preserve signal
information. This analysis is carried by introducing a multilayer kernel based
on convolutional kernel networks and by studying the geometry induced by the
kernel mapping. We then characterize the corresponding reproducing kernel
Hilbert space (RKHS), showing that it contains a large class of convolutional
neural networks with homogeneous activation functions. This analysis allows us
to separate data representation from learning, and to provide a canonical
measure of model complexity, the RKHS norm, which controls both stability and
generalization of any learned model. In addition to models in the constructed
RKHS, our stability analysis also applies to convolutional networks with
generic activations such as rectified linear units, and we discuss its
relationship with recent generalization bounds based on spectral norms
Neurogenesis Deep Learning
Neural machine learning methods, such as deep neural networks (DNN), have
achieved remarkable success in a number of complex data processing tasks. These
methods have arguably had their strongest impact on tasks such as image and
audio processing - data processing domains in which humans have long held clear
advantages over conventional algorithms. In contrast to biological neural
systems, which are capable of learning continuously, deep artificial networks
have a limited ability for incorporating new information in an already trained
network. As a result, methods for continuous learning are potentially highly
impactful in enabling the application of deep networks to dynamic data sets.
Here, inspired by the process of adult neurogenesis in the hippocampus, we
explore the potential for adding new neurons to deep layers of artificial
neural networks in order to facilitate their acquisition of novel information
while preserving previously trained data representations. Our results on the
MNIST handwritten digit dataset and the NIST SD 19 dataset, which includes
lower and upper case letters and digits, demonstrate that neurogenesis is well
suited for addressing the stability-plasticity dilemma that has long challenged
adaptive machine learning algorithms.Comment: 8 pages, 8 figures, Accepted to 2017 International Joint Conference
on Neural Networks (IJCNN 2017
Deep Convolutional Neural Networks Based on Semi-Discrete Frames
Deep convolutional neural networks have led to breakthrough results in
practical feature extraction applications. The mathematical analysis of these
networks was pioneered by Mallat, 2012. Specifically, Mallat considered
so-called scattering networks based on identical semi-discrete wavelet frames
in each network layer, and proved translation-invariance as well as deformation
stability of the resulting feature extractor. The purpose of this paper is to
develop Mallat's theory further by allowing for different and, most
importantly, general semi-discrete frames (such as, e.g., Gabor frames,
wavelets, curvelets, shearlets, ridgelets) in distinct network layers. This
allows to extract wider classes of features than point singularities resolved
by the wavelet transform. Our generalized feature extractor is proven to be
translation-invariant, and we develop deformation stability results for a
larger class of deformations than those considered by Mallat. For Mallat's
wavelet-based feature extractor, we get rid of a number of technical
conditions. The mathematical engine behind our results is continuous frame
theory, which allows us to completely detach the invariance and deformation
stability proofs from the particular algebraic structure of the underlying
frames.Comment: Proc. of IEEE International Symposium on Information Theory (ISIT),
Hong Kong, China, June 2015, to appea
- …