1,887 research outputs found
Tracking the gradients using the Hessian: A new look at variance reducing stochastic methods
Our goal is to improve variance reducing stochastic methods through better
control variates. We first propose a modification of SVRG which uses the
Hessian to track gradients over time, rather than to recondition, increasing
the correlation of the control variates and leading to faster theoretical
convergence close to the optimum. We then propose accurate and computationally
efficient approximations to the Hessian, both using a diagonal and a low-rank
matrix. Finally, we demonstrate the effectiveness of our method on a wide range
of problems.Comment: 17 pages, 2 figures, 1 tabl
A Comparison of Relaxations of Multiset Cannonical Correlation Analysis and Applications
Canonical correlation analysis is a statistical technique that is used to
find relations between two sets of variables. An important extension in pattern
analysis is to consider more than two sets of variables. This problem can be
expressed as a quadratically constrained quadratic program (QCQP), commonly
referred to Multi-set Canonical Correlation Analysis (MCCA). This is a
non-convex problem and so greedy algorithms converge to local optima without
any guarantees on global optimality. In this paper, we show that despite being
highly structured, finding the optimal solution is NP-Hard. This motivates our
relaxation of the QCQP to a semidefinite program (SDP). The SDP is convex, can
be solved reasonably efficiently and comes with both absolute and
output-sensitive approximation quality. In addition to theoretical guarantees,
we do an extensive comparison of the QCQP method and the SDP relaxation on a
variety of synthetic and real world data. Finally, we present two useful
extensions: we incorporate kernel methods and computing multiple sets of
canonical vectors
Local antithetic sampling with scrambled nets
We consider the problem of computing an approximation to the integral
. Monte Carlo (MC) sampling typically attains a root
mean squared error (RMSE) of from independent random function
evaluations. By contrast, quasi-Monte Carlo (QMC) sampling using carefully
equispaced evaluation points can attain the rate for
any and randomized QMC (RQMC) can attain the RMSE
, both under mild conditions on . Classical
variance reduction methods for MC can be adapted to QMC. Published results
combining QMC with importance sampling and with control variates have found
worthwhile improvements, but no change in the error rate. This paper extends
the classical variance reduction method of antithetic sampling and combines it
with RQMC. One such method is shown to bring a modest improvement in the RMSE
rate, attaining for any , for
smooth enough .Comment: Published in at http://dx.doi.org/10.1214/07-AOS548 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org
- …