904 research outputs found
Inexact restoration with subsampled trust-region methods for finite-sum minimization
Convex and nonconvex finite-sum minimization arises in many scientific
computing and machine learning applications. Recently, first-order and
second-order methods where objective functions, gradients and Hessians are
approximated by randomly sampling components of the sum have received great
attention. We propose a new trust-region method which employs suitable
approximations of the objective function, gradient and Hessian built via random
subsampling techniques. The choice of the sample size is deterministic and
ruled by the inexact restoration approach. We discuss local and global
properties for finding approximate first- and second-order optimal points and
function evaluation complexity results. Numerical experience shows that the new
procedure is more efficient, in terms of overall computational cost, than the
standard trust-region scheme with subsampled Hessians
Economic inexact restoration for derivative-free expensive function minimization and applications
The Inexact Restoration approach has proved to be an adequate tool for
handling the problem of minimizing an expensive function within an arbitrary
feasible set by using different degrees of precision in the objective function.
The Inexact Restoration framework allows one to obtain suitable convergence and
complexity results for an approach that rationally combines low- and
high-precision evaluations. In the present research, it is recognized that many
problems with expensive objective functions are nonsmooth and, sometimes, even
discontinuous. Having this in mind, the Inexact Restoration approach is
extended to the nonsmooth or discontinuous case. Although optimization phases
that rely on smoothness cannot be used in this case, basic convergence and
complexity results are recovered. A derivative-free optimization phase is
defined and the subproblems that arise at this phase are solved using a
regularization approach that take advantage of different notions of
stationarity. The new methodology is applied to the problem of reproducing a
controlled experiment that mimics the failure of a dam
Minimizing Nonsmooth Convex Functions with Variable Accuracy
We consider unconstrained optimization problems with nonsmooth and convex
objective function in the form of mathematical expectation. The proposed method
approximates the objective function with a sample average function by using
different sample size in each iteration. The sample size is chosen in an
adaptive manner based on the Inexact Restoration. The method uses line search
and assumes descent directions with respect to the current approximate
function. We prove the almost sure convergence under the standard assumptions.
The convergence rate is also considered and the worst-case complexity of
is proved. Numerical results for two types of
problems, machine learning hinge loss and stochastic linear complementarity
problems, show the efficiency of the proposed scheme
First order algorithms in variational image processing
Variational methods in imaging are nowadays developing towards a quite
universal and flexible tool, allowing for highly successful approaches on tasks
like denoising, deblurring, inpainting, segmentation, super-resolution,
disparity, and optical flow estimation. The overall structure of such
approaches is of the form ; where the functional is a data fidelity term also
depending on some input data and measuring the deviation of from such
and is a regularization functional. Moreover is a (often linear)
forward operator modeling the dependence of data on an underlying image, and
is a positive regularization parameter. While is often
smooth and (strictly) convex, the current practice almost exclusively uses
nonsmooth regularization functionals. The majority of successful techniques is
using nonsmooth and convex functionals like the total variation and
generalizations thereof or -norms of coefficients arising from scalar
products with some frame system. The efficient solution of such variational
problems in imaging demands for appropriate algorithms. Taking into account the
specific structure as a sum of two very different terms to be minimized,
splitting algorithms are a quite canonical choice. Consequently this field has
revived the interest in techniques like operator splittings or augmented
Lagrangians. Here we shall provide an overview of methods currently developed
and recent results as well as some computational studies providing a comparison
of different methods and also illustrating their success in applications.Comment: 60 pages, 33 figure
A stochastic first-order trust-region method with inexact restoration for finite-sum minimization
We propose a stochastic first-order trust-region method with inexact function
and gradient evaluations for solving finite-sum minimization problems. At each
iteration, the function and the gradient are approximated by sampling. The
sample size in gradient approximations is smaller than the sample size in
function approximations and the latter is determined using a deterministic rule
inspired by the inexact restoration method, which allows the decrease of the
sample size at some iterations. The trust-region step is then either accepted
or rejected using a suitable merit function, which combines the function
estimate with a measure of accuracy in the evaluation. We show that the
proposed method eventually reaches full precision in evaluating the objective
function and we provide a worst-case complexity result on the number of
iterations required to achieve full precision. We validate the proposed
algorithm on nonconvex binary classification problems showing good performance
in terms of cost and accuracy and the important feature that a burdensome
tuning of the parameters involved is not required
- …