45,100 research outputs found
On sparse representations of linear operators and the approximation of matrix products
Thus far, sparse representations have been exploited largely in the context
of robustly estimating functions in a noisy environment from a few
measurements. In this context, the existence of a basis in which the signal
class under consideration is sparse is used to decrease the number of necessary
measurements while controlling the approximation error. In this paper, we
instead focus on applications in numerical analysis, by way of sparse
representations of linear operators with the objective of minimizing the number
of operations needed to perform basic operations (here, multiplication) on
these operators. We represent a linear operator by a sum of rank-one operators,
and show how a sparse representation that guarantees a low approximation error
for the product can be obtained from analyzing an induced quadratic form. This
construction in turn yields new algorithms for computing approximate matrix
products.Comment: 6 pages, 3 figures; presented at the 42nd Annual Conference on
Information Sciences and Systems (CISS 2008
An Efficient Parallel Solver for SDD Linear Systems
We present the first parallel algorithm for solving systems of linear
equations in symmetric, diagonally dominant (SDD) matrices that runs in
polylogarithmic time and nearly-linear work. The heart of our algorithm is a
construction of a sparse approximate inverse chain for the input matrix: a
sequence of sparse matrices whose product approximates its inverse. Whereas
other fast algorithms for solving systems of equations in SDD matrices exploit
low-stretch spanning trees, our algorithm only requires spectral graph
sparsifiers
Geometrical inverse preconditioning for symmetric positive definite matrices
We focus on inverse preconditioners based on minimizing , where is the preconditioned matrix
and is symmetric and positive definite. We present and analyze
gradient-type methods to minimize
on a suitable compact set. For that we use the geometrical properties of the
non-polyhedral
cone of symmetric and positive definite matrices, and also the special
properties of on the feasible set.
Preliminary and encouraging numerical results are also presented
in which dense and sparse approximations are included
Practical Gauss-Newton Optimisation for Deep Learning
We present an efficient block-diagonal ap- proximation to the Gauss-Newton
matrix for feedforward neural networks. Our result- ing algorithm is
competitive against state- of-the-art first order optimisation methods, with
sometimes significant improvement in optimisation performance. Unlike
first-order methods, for which hyperparameter tuning of the optimisation
parameters is often a labo- rious process, our approach can provide good
performance even when used with default set- tings. A side result of our work
is that for piecewise linear transfer functions, the net- work objective
function can have no differ- entiable local maxima, which may partially explain
why such transfer functions facilitate effective optimisation.Comment: ICML 201
Computationally efficient approximations of the joint spectral radius
The joint spectral radius of a set of matrices is a measure of the maximal
asymptotic growth rate that can be obtained by forming long products of
matrices taken from the set. This quantity appears in a number of application
contexts but is notoriously difficult to compute and to approximate. We
introduce in this paper a procedure for approximating the joint spectral radius
of a finite set of matrices with arbitrary high accuracy. Our approximation
procedure is polynomial in the size of the matrices once the number of matrices
and the desired accuracy are fixed
Covariance Estimation in High Dimensions via Kronecker Product Expansions
This paper presents a new method for estimating high dimensional covariance
matrices. The method, permuted rank-penalized least-squares (PRLS), is based on
a Kronecker product series expansion of the true covariance matrix. Assuming an
i.i.d. Gaussian random sample, we establish high dimensional rates of
convergence to the true covariance as both the number of samples and the number
of variables go to infinity. For covariance matrices of low separation rank,
our results establish that PRLS has significantly faster convergence than the
standard sample covariance matrix (SCM) estimator. The convergence rate
captures a fundamental tradeoff between estimation error and approximation
error, thus providing a scalable covariance estimation framework in terms of
separation rank, similar to low rank approximation of covariance matrices. The
MSE convergence rates generalize the high dimensional rates recently obtained
for the ML Flip-flop algorithm for Kronecker product covariance estimation. We
show that a class of block Toeplitz covariance matrices is approximatable by
low separation rank and give bounds on the minimal separation rank that
ensures a given level of bias. Simulations are presented to validate the
theoretical bounds. As a real world application, we illustrate the utility of
the proposed Kronecker covariance estimator for spatio-temporal linear least
squares prediction of multivariate wind speed measurements.Comment: 47 pages, accepted to IEEE Transactions on Signal Processin
- …