1,527 research outputs found
Phase Harmonic Correlations and Convolutional Neural Networks
A major issue in harmonic analysis is to capture the phase dependence of
frequency representations, which carries important signal properties. It seems
that convolutional neural networks have found a way. Over time-series and
images, convolutional networks often learn a first layer of filters which are
well localized in the frequency domain, with different phases. We show that a
rectifier then acts as a filter on the phase of the resulting coefficients. It
computes signal descriptors which are local in space, frequency and phase. The
non-linear phase filter becomes a multiplicative operator over phase harmonics
computed with a Fourier transform along the phase. We prove that it defines a
bi-Lipschitz and invertible representation. The correlations of phase harmonics
coefficients characterise coherent structures from their phase dependence
across frequencies. For wavelet filters, we show numerically that signals
having sparse wavelet coefficients can be recovered from few phase harmonic
correlations, which provide a compressive representationComment: 26 pages, 8 figure
Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks
In this paper we propose and investigate a novel nonlinear unit, called
unit, for deep neural networks. The proposed unit receives signals from
several projections of a subset of units in the layer below and computes a
normalized norm. We notice two interesting interpretations of the
unit. First, the proposed unit can be understood as a generalization of a
number of conventional pooling operators such as average, root-mean-square and
max pooling widely used in, for instance, convolutional neural networks (CNN),
HMAX models and neocognitrons. Furthermore, the unit is, to a certain
degree, similar to the recently proposed maxout unit (Goodfellow et al., 2013)
which achieved the state-of-the-art object recognition results on a number of
benchmark datasets. Secondly, we provide a geometrical interpretation of the
activation function based on which we argue that the unit is more
efficient at representing complex, nonlinear separating boundaries. Each
unit defines a superelliptic boundary, with its exact shape defined by the
order . We claim that this makes it possible to model arbitrarily shaped,
curved boundaries more efficiently by combining a few units of different
orders. This insight justifies the need for learning different orders for each
unit in the model. We empirically evaluate the proposed units on a number
of datasets and show that multilayer perceptrons (MLP) consisting of the
units achieve the state-of-the-art results on a number of benchmark datasets.
Furthermore, we evaluate the proposed unit on the recently proposed deep
recurrent neural networks (RNN).Comment: ECML/PKDD 201
- …