5,317 research outputs found
Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
Convolutional neural networks (CNN) have led to many state-of-the-art results
spanning through various fields. However, a clear and profound theoretical
understanding of the forward pass, the core algorithm of CNN, is still lacking.
In parallel, within the wide field of sparse approximation, Convolutional
Sparse Coding (CSC) has gained increasing attention in recent years. A
theoretical study of this model was recently conducted, establishing it as a
reliable and stable alternative to the commonly practiced patch-based
processing. Herein, we propose a novel multi-layer model, ML-CSC, in which
signals are assumed to emerge from a cascade of CSC layers. This is shown to be
tightly connected to CNN, so much so that the forward pass of the CNN is in
fact the thresholding pursuit serving the ML-CSC model. This connection brings
a fresh view to CNN, as we are able to attribute to this architecture
theoretical claims such as uniqueness of the representations throughout the
network, and their stable estimation, all guaranteed under simple local
sparsity conditions. Lastly, identifying the weaknesses in the above pursuit
scheme, we propose an alternative to the forward pass, which is connected to
deconvolutional, recurrent and residual networks, and has better theoretical
guarantees
Deep Component Analysis via Alternating Direction Neural Networks
Despite a lack of theoretical understanding, deep neural networks have
achieved unparalleled performance in a wide range of applications. On the other
hand, shallow representation learning with component analysis is associated
with rich intuition and theory, but smaller capacity often limits its
usefulness. To bridge this gap, we introduce Deep Component Analysis (DeepCA),
an expressive multilayer model formulation that enforces hierarchical structure
through constraints on latent variables in each layer. For inference, we
propose a differentiable optimization algorithm implemented using recurrent
Alternating Direction Neural Networks (ADNNs) that enable parameter learning
using standard backpropagation. By interpreting feed-forward networks as
single-iteration approximations of inference in our model, we provide both a
novel theoretical perspective for understanding them and a practical technique
for constraining predictions with prior knowledge. Experimentally, we
demonstrate performance improvements on a variety of tasks, including
single-image depth prediction with sparse output constraints
Adversarial Noise Attacks of Deep Learning Architectures -- Stability Analysis via Sparse Modeled Signals
Despite their impressive performance, deep convolutional neural networks
(CNNs) have been shown to be sensitive to small adversarial perturbations.
These nuisances, which one can barely notice, are powerful enough to fool
sophisticated and well performing classifiers, leading to ridiculous
misclassification results. In this paper we analyze the stability of
state-of-the-art deep-learning classification machines to adversarial
perturbations, where we assume that the signals belong to the (possibly
multi-layer) sparse representation model. We start with convolutional sparsity
and then proceed to its multi-layered version, which is tightly connected to
CNNs. Our analysis links between the stability of the classification to noise
and the underlying structure of the signal, quantified by the sparsity of its
representation under a fixed dictionary. In addition, we offer similar
stability theorems for two practical pursuit algorithms, which are posed as two
different deep-learning architectures - the layered Thresholding and the
layered Basis Pursuit. Our analysis establishes the better robustness of the
later to adversarial attacks. We corroborate these theoretical results by
numerical experiments on three datasets: MNIST, CIFAR-10 and CIFAR-100
Convolutional Neural Networks Analyzed via Inverse Problem Theory and Sparse Representations
Inverse problems in imaging such as denoising, deblurring, superresolution
(SR) have been addressed for many decades. In recent years, convolutional
neural networks (CNNs) have been widely used for many inverse problem areas.
Although their indisputable success, CNNs are not mathematically validated as
to how and what they learn. In this paper, we prove that during training, CNN
elements solve for inverse problems which are optimum solutions stored as CNN
neuron filters. We discuss the necessity of mutual coherence between CNN layer
elements in order for a network to converge to the optimum solution. We prove
that required mutual coherence can be provided by the usage of residual
learning and skip connections. We have set rules over training sets and depth
of networks for better convergence, i.e. performance.Comment: PostPrint IET Signal Processing Journa
Deep Convolutional Compressed Sensing for LiDAR Depth Completion
In this paper we consider the problem of estimating a dense depth map from a
set of sparse LiDAR points. We use techniques from compressed sensing and the
recently developed Alternating Direction Neural Networks (ADNNs) to create a
deep recurrent auto-encoder for this task. Our architecture internally performs
an algorithm for extracting multi-level convolutional sparse codes from the
input which are then used to make a prediction. Our results demonstrate that
with only two layers and 1800 parameters we are able to out perform all
previously published results, including deep networks with orders of magnitude
more parameters
Deep Convolutional Neural Network and Sparse Least Squares Migration
We recast the forward pass of a multilayered convolutional neural network
(CNN) as the solution to the problem of sparse least squares migration (LSM).
The CNN filters and feature maps are shown to be analogous, but not equivalent,
to the migration Green's functions and the quasi-reflectivity distribution,
respectively. This provides a physical interpretation of the filters and
feature maps in deep CNN in terms of the operators for seismic imaging.
Motivated by the connection between sparse LSM and CNN, we propose the neural
network version of sparse LSM. Unlike the standard LSM method that finds the
optimal reflectivity image, neural network LSM (NNLSM) finds both the optimal
quasi-reflectivity image and the quasi-migration Green's functions. These
quasi-migration-Green's functions are also denoted as the convolutional filters
in a CNN and are similar to migration Green's functions. The advantage of NNLSM
over standard LSM is that its computational cost is significantly less and it
can be used for denoising coherent and incoherent noise in migration images.
Its disadvantage is that the NNLSM quasi-reflectivity image is only an
approximation to the actual reflectivity distribution. However, the
quasi-reflectivity image can be used as a superresolution attribute image for
high-resolution delineation of geologic bodies.Comment: 25 pages, 13 figure
An ETF view of Dropout regularization
Dropout is a popular regularization technique in deep learning. Yet, the
reason for its success is still not fully understood. This paper provides a new
interpretation of Dropout from a frame theory perspective. By drawing a
connection to recent developments in analog channel coding, we suggest that for
a certain family of autoencoders with a linear encoder, optimizing the encoder
with dropout regularization leads to an equiangular tight frame (ETF). Since
this optimization is non-convex, we add another regularization that promotes
such structures by minimizing the cross-correlation between filters in the
network. We demonstrate its applicability in convolutional and fully connected
layers in both feed-forward and recurrent networks. All these results suggest
that there is indeed a relationship between dropout and ETF structure of the
regularized linear operations.Comment: Accepted to BMVC 202
Accelerating Convolutional Neural Networks via Activation Map Compression
The deep learning revolution brought us an extensive array of neural network
architectures that achieve state-of-the-art performance in a wide variety of
Computer Vision tasks including among others, classification, detection and
segmentation. In parallel, we have also been observing an unprecedented demand
in computational and memory requirements, rendering the efficient use of neural
networks in low-powered devices virtually unattainable. Towards this end, we
propose a three-stage compression and acceleration pipeline that sparsifies,
quantizes and entropy encodes activation maps of Convolutional Neural Networks.
Sparsification increases the representational power of activation maps leading
to both acceleration of inference and higher model accuracy. Inception-V3 and
MobileNet-V1 can be accelerated by as much as with an increase in
accuracy of and on the ImageNet and CIFAR-10 datasets
respectively. Quantizing and entropy coding the sparser activation maps lead to
higher compression over the baseline, reducing the memory cost of the network
execution. Inception-V3 and MobileNet-V1 activation maps, quantized to
bits, are compressed by as much as with an increase in accuracy of
and respectively
Image Super-Resolution Using Deep Convolutional Networks
We propose a deep learning method for single image super-resolution (SR). Our
method directly learns an end-to-end mapping between the low/high-resolution
images. The mapping is represented as a deep convolutional neural network (CNN)
that takes the low-resolution image as the input and outputs the
high-resolution one. We further show that traditional sparse-coding-based SR
methods can also be viewed as a deep convolutional network. But unlike
traditional methods that handle each component separately, our method jointly
optimizes all layers. Our deep CNN has a lightweight structure, yet
demonstrates state-of-the-art restoration quality, and achieves fast speed for
practical on-line usage. We explore different network structures and parameter
settings to achieve trade-offs between performance and speed. Moreover, we
extend our network to cope with three color channels simultaneously, and show
better overall reconstruction quality.Comment: 14 pages, 14 figures, journa
Deep Residual Auto-Encoders for Expectation Maximization-inspired Dictionary Learning
We introduce a neural-network architecture, termed the constrained recurrent
sparse auto-encoder (CRsAE), that solves convolutional dictionary learning
problems, thus establishing a link between dictionary learning and neural
networks. Specifically, we leverage the interpretation of the
alternating-minimization algorithm for dictionary learning as an approximate
Expectation-Maximization algorithm to develop auto-encoders that enable the
simultaneous training of the dictionary and regularization parameter (ReLU
bias). The forward pass of the encoder approximates the sufficient statistics
of the E-step as the solution to a sparse coding problem, using an iterative
proximal gradient algorithm called FISTA. The encoder can be interpreted either
as a recurrent neural network or as a deep residual network, with two-sided
ReLU non-linearities in both cases. The M-step is implemented via a two-stage
back-propagation. The first stage relies on a linear decoder applied to the
encoder and a norm-squared loss. It parallels the dictionary update step in
dictionary learning. The second stage updates the regularization parameter by
applying a loss function to the encoder that includes a prior on the parameter
motivated by Bayesian statistics. We demonstrate in an image-denoising task
that CRsAE learns Gabor-like filters, and that the EM-inspired approach for
learning biases is superior to the conventional approach. In an application to
recordings of electrical activity from the brain, we demonstrate that CRsAE
learns realistic spike templates and speeds up the process of identifying spike
times by 900x compared to algorithms based on convex optimization
- …