55 research outputs found
Scalable iterative methods for sampling from massive Gaussian random vectors
Sampling from Gaussian Markov random fields (GMRFs), that is multivariate
Gaussian ran- dom vectors that are parameterised by the inverse of their
covariance matrix, is a fundamental problem in computational statistics. In
this paper, we show how we can exploit arbitrarily accu- rate approximations to
a GMRF to speed up Krylov subspace sampling methods. We also show that these
methods can be used when computing the normalising constant of a large
multivariate Gaussian distribution, which is needed for both any
likelihood-based inference method. The method we derive is also applicable to
other structured Gaussian random vectors and, in particu- lar, we show that
when the precision matrix is a perturbation of a (block) circulant matrix, it
is still possible to derive O(n log n) sampling schemes.Comment: 17 Pages, 4 Figure
70 years of Krylov subspace methods: The journey continues
Using computed examples for the Conjugate Gradient method and GMRES, we
recall important building blocks in the understanding of Krylov subspace
methods over the last 70 years. Each example consists of a description of the
setup and the numerical observations, followed by an explanation of the
observed phenomena, where we keep technical details as small as possible. Our
goal is to show the mathematical beauty and hidden intricacies of the methods,
and to point out some persistent misunderstandings as well as important open
problems. We hope that this work initiates further investigations of Krylov
subspace methods, which are efficient computational tools and exciting
mathematical objects that are far from being fully understood.Comment: 38 page
Large-Scale Gaussian Processes via Alternating Projection
Gaussian process (GP) hyperparameter optimization requires repeatedly solving
linear systems with kernel matrices. To address the prohibitive
time complexity, recent work has employed fast iterative
numerical methods, like conjugate gradients (CG). However, as datasets increase
in magnitude, the corresponding kernel matrices become increasingly
ill-conditioned and still require space without
partitioning. Thus, while CG increases the size of datasets GPs can be trained
on, modern datasets reach scales beyond its applicability. In this work, we
propose an iterative method which only accesses subblocks of the kernel matrix,
effectively enabling \emph{mini-batching}. Our algorithm, based on alternating
projection, has per-iteration time and space complexity,
solving many of the practical challenges of scaling GPs to very large datasets.
Theoretically, we prove our method enjoys linear convergence and empirically we
demonstrate its robustness to ill-conditioning. On large-scale benchmark
datasets up to four million datapoints our approach accelerates training by a
factor of 2 to 27 compared to CG
Convex Optimization for Big Data
This article reviews recent advances in convex optimization algorithms for
Big Data, which aim to reduce the computational, storage, and communications
bottlenecks. We provide an overview of this emerging field, describe
contemporary approximation techniques like first-order methods and
randomization for scalability, and survey the important role of parallel and
distributed computation. The new Big Data algorithms are based on surprisingly
simple principles and attain staggering accelerations even on classical
problems.Comment: 23 pages, 4 figurs, 8 algorithm
- …