15 research outputs found
From Bayesian Sparsity to Gated Recurrent Nets
The iterations of many first-order algorithms, when applied to minimizing
common regularized regression functions, often resemble neural network layers
with pre-specified weights. This observation has prompted the development of
learning-based approaches that purport to replace these iterations with
enhanced surrogates forged as DNN models from available training data. For
example, important NP-hard sparse estimation problems have recently benefitted
from this genre of upgrade, with simple feedforward or recurrent networks
ousting proximal gradient-based iterations. Analogously, this paper
demonstrates that more powerful Bayesian algorithms for promoting sparsity,
which rely on complex multi-loop majorization-minimization techniques, mirror
the structure of more sophisticated long short-term memory (LSTM) networks, or
alternative gated feedback networks previously designed for sequence
prediction. As part of this development, we examine the parallels between
latent variable trajectories operating across multiple time-scales during
optimization, and the activations within deep network structures designed to
adaptively model such characteristic sequences. The resulting insights lead to
a novel sparse estimation system that, when granted training data, can estimate
optimal solutions efficiently in regimes where other algorithms fail, including
practical direction-of-arrival (DOA) and 3D geometry recovery problems. The
underlying principles we expose are also suggestive of a learning process for a
richer class of multi-loop algorithms in other domains
A Learning-Based Framework for Line-Spectra Super-resolution
We propose a learning-based approach for estimating the spectrum of a
multisinusoidal signal from a finite number of samples. A neural-network is
trained to approximate the spectra of such signals on simulated data. The
proposed methodology is very flexible: adapting to different signal and noise
models only requires modifying the training data accordingly. Numerical
experiments show that the approach performs competitively with classical
methods designed for additive Gaussian noise at a range of noise levels, and is
also effective in the presence of impulsive noise.Comment: Accepted at ICASSP 201
Fourier Phase Retrieval with Extended Support Estimation via Deep Neural Network
We consider the problem of sparse phase retrieval from Fourier transform
magnitudes to recover the -sparse signal vector and its support
. We exploit extended support estimate with size
larger than satisfying and obtained by
a trained deep neural network (DNN). To make the DNN learnable, it provides
as the union of equivalent solutions of by
utilizing modulo Fourier invariances. Set can be estimated with
short running time via the DNN, and support can be determined
from the DNN output rather than from the full index set by applying hard
thresholding to . Thus, the DNN-based extended support estimation
improves the reconstruction performance of the signal with a low complexity
burden dependent on . Numerical results verify that the proposed scheme has
a superior performance with lower complexity compared to local search-based
greedy sparse phase retrieval and a state-of-the-art variant of the Fienup
method
Data-driven Estimation of Sinusoid Frequencies
Frequency estimation is a fundamental problem in signal processing, with
applications in radar imaging, underwater acoustics, seismic imaging, and
spectroscopy. The goal is to estimate the frequency of each component in a
multisinusoidal signal from a finite number of noisy samples. A recent
machine-learning approach uses a neural network to output a learned
representation with local maxima at the position of the frequency estimates. In
this work, we propose a novel neural-network architecture that produces a
significantly more accurate representation, and combine it with an additional
neural-network module trained to detect the number of frequencies. This yields
a fast, fully-automatic method for frequency estimation that achieves
state-of-the-art results. In particular, it outperforms existing techniques by
a substantial margin at medium-to-high noise levels
Neural Inverse Rendering for General Reflectance Photometric Stereo
We present a novel convolutional neural network architecture for photometric
stereo (Woodham, 1980), a problem of recovering 3D object surface normals from
multiple images observed under varying illuminations. Despite its long history
in computer vision, the problem still shows fundamental challenges for surfaces
with unknown general reflectance properties (BRDFs). Leveraging deep neural
networks to learn complicated reflectance models is promising, but studies in
this direction are very limited due to difficulties in acquiring accurate
ground truth for training and also in designing networks invariant to
permutation of input images. In order to address these challenges, we propose a
physics based unsupervised learning framework where surface normals and BRDFs
are predicted by the network and fed into the rendering equation to synthesize
observed images. The network weights are optimized during testing by minimizing
reconstruction loss between observed and synthesized images. Thus, our learning
process does not require ground truth normals or even pre-training on external
images. Our method is shown to achieve the state-of-the-art performance on a
challenging real-world scene benchmark.Comment: To appear in International Conference on Machine Learning 2018 (ICML
2018). 10 pages + 20 pages (appendices
Learning a Compressed Sensing Measurement Matrix via Gradient Unrolling
Linear encoding of sparse vectors is widely popular, but is commonly
data-independent -- missing any possible extra (but a priori unknown) structure
beyond sparsity. In this paper we present a new method to learn linear encoders
that adapt to data, while still performing well with the widely used
decoder. The convex decoder prevents gradient propagation as needed in
standard gradient-based training. Our method is based on the insight that
unrolling the convex decoder into projected subgradient steps can address
this issue. Our method can be seen as a data-driven way to learn a compressed
sensing measurement matrix. We compare the empirical performance of 10
algorithms over 6 sparse datasets (3 synthetic and 3 real). Our experiments
show that there is indeed additional structure beyond sparsity in the real
datasets; our method is able to discover it and exploit it to create excellent
reconstructions with fewer measurements (by a factor of 1.1-3x) compared to the
previous state-of-the-art methods. We illustrate an application of our method
in learning label embeddings for extreme multi-label classification, and
empirically show that our method is able to match or outperform the precision
scores of SLEEC, which is one of the state-of-the-art embedding-based
approaches.Comment: 17 pages, 7 tables, 8 figures, published in ICML 2019; part of this
work was done while Shanshan was an intern at Google Research, New Yor
Sparse Bayesian Learning via Stepwise Regression
Sparse Bayesian Learning (SBL) is a powerful framework for attaining sparsity
in probabilistic models. Herein, we propose a coordinate ascent algorithm for
SBL termed Relevance Matching Pursuit (RMP) and show that, as its noise
variance parameter goes to zero, RMP exhibits a surprising connection to
Stepwise Regression. Further, we derive novel guarantees for Stepwise
Regression algorithms, which also shed light on RMP. Our guarantees for Forward
Regression improve on deterministic and probabilistic results for Orthogonal
Matching Pursuit with noise. Our analysis of Backward Regression on determined
systems culminates in a bound on the residual of the optimal solution to the
subset selection problem that, if satisfied, guarantees the optimality of the
result. To our knowledge, this bound is the first that can be computed in
polynomial time and depends chiefly on the smallest singular value of the
matrix. We report numerical experiments using a variety of feature selection
algorithms. Notably, RMP and its limiting variant are both efficient and
maintain strong performance with correlated features
Long short-term relevance learning
To incorporate prior knowledge as well as measurement uncertainties in the
traditional long short term memory (LSTM) neural networks, an efficient sparse
Bayesian training algorithm is introduced to the network architecture. The
proposed scheme automatically determines relevant neural connections and adapts
accordingly, in contrast to the classical LSTM solution. Due to its
flexibility, the new LSTM scheme is less prone to overfitting, and hence can
approximate time dependent solutions by use of a smaller data set. On a
structural nonlinear finite element application we show that the
self-regulating framework does not require prior knowledge of a suitable
network architecture and size, while ensuring satisfying accuracy at reasonable
computational cost
Compressed Sensing via Measurement-Conditional Generative Models
A pre-trained generator has been frequently adopted in compressed sensing
(CS) due to its ability to effectively estimate signals with the prior of NNs.
In order to further refine the NN-based prior, we propose a framework that
allows the generator to utilize additional information from a given measurement
for prior learning, thereby yielding more accurate prediction for signals. As
our framework has a simple form, it is easily applied to existing CS methods
using pre-trained generators. We demonstrate through extensive experiments that
our framework exhibits uniformly superior performances by large margin and can
reduce the reconstruction error up to an order of magnitude for some
applications. We also explain the experimental success in theory by showing
that our framework can slightly relax the stringent signal presence condition,
which is required to guarantee the success of signal recovery
Tree Search Network for Sparse Regression
We consider the classical sparse regression problem of recovering a sparse
signal given a measurement vector . We propose a tree
search algorithm driven by the deep neural network for sparse regression (TSN).
TSN improves the signal reconstruction performance of the deep neural network
designed for sparse regression by performing a tree search with pruning. It is
observed in both noiseless and noisy cases, TSN recovers synthetic and real
signals with lower complexity than a conventional tree search and is superior
to existing algorithms by a large margin for various types of the sensing
matrix , widely used in sparse regression