3,919 research outputs found
Global convergence rate analysis of unconstrained optimization methods based on probabilistic models
We present global convergence rates for a line-search method which is based
on random first-order models and directions whose quality is ensured only with
certain probability. We show that in terms of the order of the accuracy, the
evaluation complexity of such a method is the same as its counterparts that use
deterministic accurate models; the use of probabilistic models only increases
the complexity by a constant, which depends on the probability of the models
being good. We particularize and improve these results in the convex and
strongly convex case.
We also analyze a probabilistic cubic regularization variant that allows
approximate probabilistic second-order models and show improved complexity
bounds compared to probabilistic first-order methods; again, as a function of
the accuracy, the probabilistic cubic regularization bounds are of the same
(optimal) order as for the deterministic case
Optimization Methods for Inverse Problems
Optimization plays an important role in solving many inverse problems.
Indeed, the task of inversion often either involves or is fully cast as a
solution of an optimization problem. In this light, the mere non-linear,
non-convex, and large-scale nature of many of these inversions gives rise to
some very challenging optimization problems. The inverse problem community has
long been developing various techniques for solving such optimization tasks.
However, other, seemingly disjoint communities, such as that of machine
learning, have developed, almost in parallel, interesting alternative methods
which might have stayed under the radar of the inverse problem community. In
this survey, we aim to change that. In doing so, we first discuss current
state-of-the-art optimization methods widely used in inverse problems. We then
survey recent related advances in addressing similar challenges in problems
faced by the machine learning community, and discuss their potential advantages
for solving inverse problems. By highlighting the similarities among the
optimization challenges faced by the inverse problem and the machine learning
communities, we hope that this survey can serve as a bridge in bringing
together these two communities and encourage cross fertilization of ideas.Comment: 13 page
Stochastic Trust Region Methods with Trust Region Radius Depending on Probabilistic Models
We present a stochastic trust-region model-based framework in which its
radius is related to the probabilistic models. Especially, we propose a
specific algorithm, termed STRME, in which the trust-region radius depends
linearly on the latest model gradient. The complexity of STRME method in
non-convex, convex and strongly convex settings has all been analyzed, which
matches the existing algorithms based on probabilistic properties. In addition,
several numerical experiments are carried out to reveal the benefits of the
proposed methods compared to the existing stochastic trust-region methods and
other relevant stochastic gradient methods
- …