5,444 research outputs found
Gradient Projection Iterative Sketch for Large Scale Constrained Least-Squares
We propose a randomized first order optimization algorithm Gradient
Projection Iterative Sketch (GPIS) and an accelerated variant for efficiently
solving large scale constrained Least Squares (LS). We provide theoretical
convergence analysis for both proposed algorithms and demonstrate our methods'
computational efficiency compared to classical accelerated gradient method, and
the state of the art variance-reduced stochastic gradient methods through
numerical experiments in various large synthetic/real data sets.Comment: Technical report. 15 page
Large Scale Constrained Linear Regression Revisited: Faster Algorithms via Preconditioning
In this paper, we revisit the large-scale constrained linear regression
problem and propose faster methods based on some recent developments in
sketching and optimization. Our algorithms combine (accelerated) mini-batch SGD
with a new method called two-step preconditioning to achieve an approximate
solution with a time complexity lower than that of the state-of-the-art
techniques for the low precision case. Our idea can also be extended to the
high precision case, which gives an alternative implementation to the Iterative
Hessian Sketch (IHS) method with significantly improved time complexity.
Experiments on benchmark and synthetic datasets suggest that our methods indeed
outperform existing ones considerably in both the low and high precision cases.Comment: Appear in AAAI-1
Randomized Riemannian Preconditioning for Orthogonality Constrained Problems
Optimization problems with (generalized) orthogonality constraints are
prevalent across science and engineering. For example, in computational science
they arise in the symmetric (generalized) eigenvalue problem, in nonlinear
eigenvalue problems, and in electronic structures computations, to name a few
problems. In statistics and machine learning, they arise, for example, in
canonical correlation analysis and in linear discriminant analysis. In this
article, we consider using randomized preconditioning in the context of
optimization problems with generalized orthogonality constraints. Our proposed
algorithms are based on Riemannian optimization on the generalized Stiefel
manifold equipped with a non-standard preconditioned geometry, which
necessitates development of the geometric components necessary for developing
algorithms based on this approach. Furthermore, we perform asymptotic
convergence analysis of the preconditioned algorithms which help to
characterize the quality of a given preconditioner using second-order
information. Finally, for the problems of canonical correlation analysis and
linear discriminant analysis, we develop randomized preconditioners along with
corresponding bounds on the relevant condition number
Optimization Methods for Inverse Problems
Optimization plays an important role in solving many inverse problems.
Indeed, the task of inversion often either involves or is fully cast as a
solution of an optimization problem. In this light, the mere non-linear,
non-convex, and large-scale nature of many of these inversions gives rise to
some very challenging optimization problems. The inverse problem community has
long been developing various techniques for solving such optimization tasks.
However, other, seemingly disjoint communities, such as that of machine
learning, have developed, almost in parallel, interesting alternative methods
which might have stayed under the radar of the inverse problem community. In
this survey, we aim to change that. In doing so, we first discuss current
state-of-the-art optimization methods widely used in inverse problems. We then
survey recent related advances in addressing similar challenges in problems
faced by the machine learning community, and discuss their potential advantages
for solving inverse problems. By highlighting the similarities among the
optimization challenges faced by the inverse problem and the machine learning
communities, we hope that this survey can serve as a bridge in bringing
together these two communities and encourage cross fertilization of ideas.Comment: 13 page
Optimization algorithms for the solution of the frictionless normal contact between rough surfaces
This paper revisits the fundamental equations for the solution of the
frictionless unilateral normal contact problem between a rough rigid surface
and a linear elastic half-plane using the boundary element method (BEM). After
recasting the resulting Linear Complementarity Problem (LCP) as a convex
quadratic program (QP) with nonnegative constraints, different optimization
algorithms are compared for its solution: (i) a Greedy method, based on
different solvers for the unconstrained linear system (Conjugate Gradient CG,
Gauss-Seidel, Cholesky factorization), (ii) a constrained CG algorithm, (iii)
the Alternating Direction Method of Multipliers (ADMM), and () the
Non-Negative Least Squares (NNLS) algorithm, possibly warm-started by
accelerated gradient projection steps or taking advantage of a loading history.
The latter method is two orders of magnitude faster than the Greedy CG method
and one order of magnitude faster than the constrained CG algorithm. Finally,
we propose another type of warm start based on a refined criterion for the
identification of the initial trial contact domain that can be used in
conjunction with all the previous optimization algorithms. This method, called
Cascade Multi-Resolution (CMR), takes advantage of physical considerations
regarding the scaling of the contact predictions by changing the surface
resolution. The method is very efficient and accurate when applied to real or
numerically generated rough surfaces, provided that their power spectral
density function is of power-law type, as in case of self-similar fractal
surfaces.Comment: 38 pages, 11 figure
CLEAR: Covariant LEAst-square Re-fitting with applications to image restoration
In this paper, we propose a new framework to remove parts of the systematic
errors affecting popular restoration algorithms, with a special focus for image
processing tasks. Generalizing ideas that emerged for regularization,
we develop an approach re-fitting the results of standard methods towards the
input data. Total variation regularizations and non-local means are special
cases of interest. We identify important covariant information that should be
preserved by the re-fitting method, and emphasize the importance of preserving
the Jacobian (w.r.t. the observed signal) of the original estimator. Then, we
provide an approach that has a "twicing" flavor and allows re-fitting the
restored signal by adding back a local affine transformation of the residual
term. We illustrate the benefits of our method on numerical simulations for
image restoration tasks
Computational Methods for Sparse Solution of Linear Inverse Problems
The goal of the sparse approximation problem is to approximate a target signal using a linear combination of a few elementary signals drawn from a fixed collection. This paper surveys the major practical algorithms for sparse approximation. Specific attention is paid to computational issues, to the circumstances in which individual methods tend to perform well, and to the theoretical guarantees available. Many fundamental questions in electrical engineering, statistics, and applied mathematics can be posed as sparse approximation problems, making these algorithms versatile and relevant to a plethora of applications
Convex optimization problem prototyping for image reconstruction in computed tomography with the Chambolle-Pock algorithm
The primal-dual optimization algorithm developed in Chambolle and Pock (CP),
2011 is applied to various convex optimization problems of interest in computed
tomography (CT) image reconstruction. This algorithm allows for rapid
prototyping of optimization problems for the purpose of designing iterative
image reconstruction algorithms for CT. The primal-dual algorithm is briefly
summarized in the article, and its potential for prototyping is demonstrated by
explicitly deriving CP algorithm instances for many optimization problems
relevant to CT. An example application modeling breast CT with low-intensity
X-ray illumination is presented.Comment: Resubmitted to Physics in Medicine and Biology. Text has been
modified according to referee comments, and typos in the equations have been
correcte
- …