52,977 research outputs found
Algorithms for Kullback-Leibler Approximation of Probability Measures in Infinite Dimensions
In this paper we study algorithms to find a Gaussian approximation to a
target measure defined on a Hilbert space of functions; the target measure
itself is defined via its density with respect to a reference Gaussian measure.
We employ the Kullback-Leibler divergence as a distance and find the best
Gaussian approximation by minimizing this distance. It then follows that the
approximate Gaussian must be equivalent to the Gaussian reference measure,
defining a natural function space setting for the underlying calculus of
variations problem. We introduce a computational algorithm which is
well-adapted to the required minimization, seeking to find the mean as a
function, and parameterizing the covariance in two different ways: through low
rank perturbations of the reference covariance; and through Schr\"odinger
potential perturbations of the inverse reference covariance. Two applications
are shown: to a nonlinear inverse problem in elliptic PDEs, and to a
conditioned diffusion process. We also show how the Gaussian approximations we
obtain may be used to produce improved pCN-MCMC methods which are not only
well-adapted to the high-dimensional setting, but also behave well with respect
to small observational noise (resp. small temperatures) in the inverse problem
(resp. conditioned diffusion).Comment: 28 page
Distributional Property Testing in a Quantum World
A fundamental problem in statistics and learning theory is to test properties of distributions. We show that quantum computers can solve such problems with significant speed-ups. We also introduce a novel access model for quantum distributions, enabling the coherent preparation of quantum samples, and propose a general framework that can naturally handle both classical and quantum distributions in a unified manner. Our framework generalizes and improves previous quantum algorithms for testing closeness between unknown distributions, testing independence between two distributions, and estimating the Shannon / von Neumann entropy of distributions. For classical distributions our algorithms significantly improve the precision dependence of some earlier results. We also show that in our framework procedures for classical distributions can be directly lifted to the more general case of quantum distributions, and thus obtain the first speed-ups for testing properties of density operators that can be accessed coherently rather than only via sampling
Far-Field Compression for Fast Kernel Summation Methods in High Dimensions
We consider fast kernel summations in high dimensions: given a large set of
points in dimensions (with ) and a pair-potential function (the
{\em kernel} function), we compute a weighted sum of all pairwise kernel
interactions for each point in the set. Direct summation is equivalent to a
(dense) matrix-vector multiplication and scales quadratically with the number
of points. Fast kernel summation algorithms reduce this cost to log-linear or
linear complexity.
Treecodes and Fast Multipole Methods (FMMs) deliver tremendous speedups by
constructing approximate representations of interactions of points that are far
from each other. In algebraic terms, these representations correspond to
low-rank approximations of blocks of the overall interaction matrix. Existing
approaches require an excessive number of kernel evaluations with increasing
and number of points in the dataset.
To address this issue, we use a randomized algebraic approach in which we
first sample the rows of a block and then construct its approximate, low-rank
interpolative decomposition. We examine the feasibility of this approach
theoretically and experimentally. We provide a new theoretical result showing a
tighter bound on the reconstruction error from uniformly sampling rows than the
existing state-of-the-art. We demonstrate that our sampling approach is
competitive with existing (but prohibitively expensive) methods from the
literature. We also construct kernel matrices for the Laplacian, Gaussian, and
polynomial kernels -- all commonly used in physics and data analysis. We
explore the numerical properties of blocks of these matrices, and show that
they are amenable to our approach. Depending on the data set, our randomized
algorithm can successfully compute low rank approximations in high dimensions.
We report results for data sets with ambient dimensions from four to 1,000.Comment: 43 pages, 21 figure
Weighted Polynomial Approximations: Limits for Learning and Pseudorandomness
Polynomial approximations to boolean functions have led to many positive
results in computer science. In particular, polynomial approximations to the
sign function underly algorithms for agnostically learning halfspaces, as well
as pseudorandom generators for halfspaces. In this work, we investigate the
limits of these techniques by proving inapproximability results for the sign
function.
Firstly, the polynomial regression algorithm of Kalai et al. (SIAM J. Comput.
2008) shows that halfspaces can be learned with respect to log-concave
distributions on in the challenging agnostic learning model. The
power of this algorithm relies on the fact that under log-concave
distributions, halfspaces can be approximated arbitrarily well by low-degree
polynomials. We ask whether this technique can be extended beyond log-concave
distributions, and establish a negative result. We show that polynomials of any
degree cannot approximate the sign function to within arbitrarily low error for
a large class of non-log-concave distributions on the real line, including
those with densities proportional to .
Secondly, we investigate the derandomization of Chernoff-type concentration
inequalities. Chernoff-type tail bounds on sums of independent random variables
have pervasive applications in theoretical computer science. Schmidt et al.
(SIAM J. Discrete Math. 1995) showed that these inequalities can be established
for sums of random variables with only -wise independence,
for a tail probability of . We show that their results are tight up to
constant factors.
These results rely on techniques from weighted approximation theory, which
studies how well functions on the real line can be approximated by polynomials
under various distributions. We believe that these techniques will have further
applications in other areas of computer science.Comment: 22 page
Approximating Probability Densities by Iterated Laplace Approximations
The Laplace approximation is an old, but frequently used method to
approximate integrals for Bayesian calculations. In this paper we develop an
extension of the Laplace approximation, by applying it iteratively to the
residual, i.e., the difference between the current approximation and the true
function. The final approximation is thus a linear combination of multivariate
normal densities, where the coefficients are chosen to achieve a good fit to
the target distribution. We illustrate on real and artificial examples that the
proposed procedure is a computationally efficient alternative to current
approaches for approximation of multivariate probability densities. The
R-package iterLap implementing the methods described in this article is
available from the CRAN servers.Comment: to appear in Journal of Computational and Graphical Statistics,
http://pubs.amstat.org/loi/jcg
Limits on Sparse Data Acquisition: RIC Analysis of Finite Gaussian Matrices
One of the key issues in the acquisition of sparse data by means of
compressed sensing (CS) is the design of the measurement matrix. Gaussian
matrices have been proven to be information-theoretically optimal in terms of
minimizing the required number of measurements for sparse recovery. In this
paper we provide a new approach for the analysis of the restricted isometry
constant (RIC) of finite dimensional Gaussian measurement matrices. The
proposed method relies on the exact distributions of the extreme eigenvalues
for Wishart matrices. First, we derive the probability that the restricted
isometry property is satisfied for a given sufficient recovery condition on the
RIC, and propose a probabilistic framework to study both the symmetric and
asymmetric RICs. Then, we analyze the recovery of compressible signals in noise
through the statistical characterization of stability and robustness. The
presented framework determines limits on various sparse recovery algorithms for
finite size problems. In particular, it provides a tight lower bound on the
maximum sparsity order of the acquired data allowing signal recovery with a
given target probability. Also, we derive simple approximations for the RICs
based on the Tracy-Widom distribution.Comment: 11 pages, 6 figures, accepted for publication in IEEE transactions on
information theor
Approximate Graph Coloring by Semidefinite Programming
We consider the problem of coloring k-colorable graphs with the fewest
possible colors. We present a randomized polynomial time algorithm that colors
a 3-colorable graph on vertices with min O(Delta^{1/3} log^{1/2} Delta log
n), O(n^{1/4} log^{1/2} n) colors where Delta is the maximum degree of any
vertex. Besides giving the best known approximation ratio in terms of n, this
marks the first non-trivial approximation result as a function of the maximum
degree Delta. This result can be generalized to k-colorable graphs to obtain a
coloring using min O(Delta^{1-2/k} log^{1/2} Delta log n), O(n^{1-3/(k+1)}
log^{1/2} n) colors. Our results are inspired by the recent work of Goemans and
Williamson who used an algorithm for semidefinite optimization problems, which
generalize linear programs, to obtain improved approximations for the MAX CUT
and MAX 2-SAT problems. An intriguing outcome of our work is a duality
relationship established between the value of the optimum solution to our
semidefinite program and the Lovasz theta-function. We show lower bounds on the
gap between the optimum solution of our semidefinite program and the actual
chromatic number; by duality this also demonstrates interesting new facts about
the theta-function
- …