SPECTRAL PROJECTED GRADIENT METHOD WITH INEXACT RESTORATION FOR MINIMIZATION WITH NONCONVEX CONSTRAINTS

Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)This work takes advantage of the spectral projected gradient direction within the inexact restoration framework to address nonlinear optimization problems with nonconvex constraints. The proposed strategy includes a convenient handling of the constraints, together with nonmonotonic features to speed up convergence. The numerical performance is assessed by experiments with hard-spheres problems, pointing out that the inexact restoration framework provides an adequate environment for the extension of the spectral projected gradient method for general nonlinearly constrained optimization.31316281652Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)CNPq [E-26/171.164/2003 - APQ1]FAPESP [01/04597-4, 06/53768-0

Economic inexact restoration for derivative-free expensive function minimization and applications

The Inexact Restoration approach has proved to be an adequate tool for handling the problem of minimizing an expensive function within an arbitrary feasible set by using different degrees of precision in the objective function. The Inexact Restoration framework allows one to obtain suitable convergence and complexity results for an approach that rationally combines low- and high-precision evaluations. In the present research, it is recognized that many problems with expensive objective functions are nonsmooth and, sometimes, even discontinuous. Having this in mind, the Inexact Restoration approach is extended to the nonsmooth or discontinuous case. Although optimization phases that rely on smoothness cannot be used in this case, basic convergence and complexity results are recovered. A derivative-free optimization phase is defined and the subproblems that arise at this phase are solved using a regularization approach that take advantage of different notions of stationarity. The new methodology is applied to the problem of reproducing a controlled experiment that mimics the failure of a dam

New convergence results for the scaled gradient projection method

The aim of this paper is to deepen the convergence analysis of the scaled gradient projection (SGP) method, proposed by Bonettini et al. in a recent paper for constrained smooth optimization. The main feature of SGP is the presence of a variable scaling matrix multiplying the gradient, which may change at each iteration. In the last few years, an extensive numerical experimentation showed that SGP equipped with a suitable choice of the scaling matrix is a very effective tool for solving large scale variational problems arising in image and signal processing. In spite of the very reliable numerical results observed, only a weak, though very general, convergence theorem is provided, establishing that any limit point of the sequence generated by SGP is stationary. Here, under the only assumption that the objective function is convex and that a solution exists, we prove that the sequence generated by SGP converges to a minimum point, if the scaling matrices sequence satisfies a simple and implementable condition. Moreover, assuming that the gradient of the objective function is Lipschitz continuous, we are also able to prove the O(1/k) convergence rate with respect to the objective function values. Finally, we present the results of a numerical experience on some relevant image restoration problems, showing that the proposed scaling matrix selection rule performs well also from the computational point of view

First order algorithms in variational image processing

Variational methods in imaging are nowadays developing towards a quite universal and flexible tool, allowing for highly successful approaches on tasks like denoising, deblurring, inpainting, segmentation, super-resolution, disparity, and optical flow estimation. The overall structure of such approaches is of the form ${\cal D}(Ku) + \alpha {\cal R} (u) \rightarrow \min_u$ ; where the functional ${\cal D}$ is a data fidelity term also depending on some input data $f$ and measuring the deviation of $Ku$ from such and ${\cal R}$ is a regularization functional. Moreover $K$ is a (often linear) forward operator modeling the dependence of data on an underlying image, and $\alpha$ is a positive regularization parameter. While ${\cal D}$ is often smooth and (strictly) convex, the current practice almost exclusively uses nonsmooth regularization functionals. The majority of successful techniques is using nonsmooth and convex functionals like the total variation and generalizations thereof or $\ell_1$-norms of coefficients arising from scalar products with some frame system. The efficient solution of such variational problems in imaging demands for appropriate algorithms. Taking into account the specific structure as a sum of two very different terms to be minimized, splitting algorithms are a quite canonical choice. Consequently this field has revived the interest in techniques like operator splittings or augmented Lagrangians. Here we shall provide an overview of methods currently developed and recent results as well as some computational studies providing a comparison of different methods and also illustrating their success in applications.Comment: 60 pages, 33 figure

Variable metric inexact line-search based methods for nonsmooth optimization

We develop a new proximal-gradient method for minimizing the sum of a differentiable, possibly nonconvex, function plus a convex, possibly non differentiable, function. The key features of the proposed method are the definition of a suitable descent direction, based on the proximal operator associated to the convex part of the objective function, and an Armijo-like rule to determine the step size along this direction ensuring the sufficient decrease of the objective function. In this frame, we especially address the possibility of adopting a metric which may change at each iteration and an inexact computation of the proximal point defining the descent direction. For the more general nonconvex case, we prove that all limit points of the iterates sequence are stationary, while for convex objective functions we prove the convergence of the whole sequence to a minimizer, under the assumption that a minimizer exists. In the latter case, assuming also that the gradient of the smooth part of the objective function is Lipschitz, we also give a convergence rate estimate, showing the O(1/k) complexity with respect to the function values. We also discuss verifiable sufficient conditions for the inexact proximal point and we present the results of a numerical experience on a convex total variation based image restoration problem, showing that the proposed approach is competitive with another state-of-the-art method

Modifikacije metoda NJutnovog tipa za rešavanje semi-glatkih problema stohastičke optimizacije

 In numerous optimization problems originating from real-world and scientific applications, we often face nonsmoothness. A large number of problems belong to this class, from models of natural phenomena that exhibit sudden changes, shape optimization, to hinge loss functions in machine learning and deep neural networks. In practice, solving a on smooth convex problem tends to be more challenging, usually more difficult and costly than a smooth one. The aim of this thesis is the formulation and theoretical analysis of Newton-type algorithms for solving nonsmooth convex stochastic optimization problems. The optimization problems with the objective function given in the form of a mathematical expectation without differentiability assumption of the function are considered. The Sample Average Approximation (SAA) is used to estimate the objective function. As the accuracy of the SAA objective functions and its derivatives is naturally proportional to the computational costs – higher precision implies larger costs in general, it is important to design an efficient balance between accuracy and costs. Therefore, the main focus of this thesis is the development of adaptive sample size control algorithms in a nonsmooth environment, with particular attention given to the control of the accuracy and selection of search directions. Several options are investigated for the search direction, while the accuracy control involves cheaper objective function approximations (with looser accuracy) during the initial stages of the process to save computational effort. This approach aims to conserve computational resources, reserving the deployment of high-accuracy objective function approximations for the final stages of the optimization process. A detailed description of the proposed methods is presented in Chapter 5 and 6. Also, the theoretical properties of the numerical procedures are analyzed, i.e., their convergence is proved, and the complexity of the developed methods is studied. In addition to the theoretical framework, the successful practical implementation of the given algorithms is presented. It is shown that the proposed methods are more efficient in practical application compared to the existing methods from the literature. Chapter 1 of this thesis serves as a foundation for the subsequent chapters by providing the necessary background information. Chapter 2 covers the fundamentals of nonlinear optimization, with a particular emphasis on line search techniques. In Chapter 3, the focus shifts to the nonsmooth framework. This chapter serves the purpose of reviewing the existing knowledge and established results in the field. The remaining sections of the thesis, starting from Chapter 4, where the framework for the subject of this thesis (the minimization of the expected value function) is introduced, onwards, represent the original contribution made by the author.У бројним проблемима оптимизације који потичу из стварних и научних примена, често се суочавамо са недиференцијабилношћу. У ову класу спада велики број проблема, од модела природних феномена који показују нагле промене, оптимизације облика, до функције циља у машинском учењу и дубоким неуронским мрежама. У пракси, решавање семи-глатких конвексних проблема обично је изазовније и захтева веће рачунске трошкове у односу на глатке проблеме. Циљ ове тезе је формулација и теоријска анализа метода Њутновог типа за решавање семи-глатких конвексних стохастичких проблема оптимизације. Разматрани су проблеми оптимизације са функцијом циља датом у облику математичког очекивања без претпоставке о диференцијабилности функције. Како је врло тешко, па некад чак и немогуће одредити аналитички облик математичког очекивања, функција циља се апроксимира узорачким очекивањем. Имајући у виду да је тачност апроксимације функције циља и њених извода пропорционална рачунским трошковима – већа прецизност подразумева веће трошкове у општем случају, важно је дизајнирати ефикасан баланс између тачности и трошкова. Стога, главни фокус ове тезе је развојалгоритама базираних на одређивању оптималне динамике увећања узорка у семи-глатком окружењу, са посебном пажњом на контроли тачности и одабиру праваца претраге. По питању одабира правца, размотрено је неколико опција, док контрола тачности укључује јефтиније апроксимације функције циља (са мањом прецизношћу) током почетних фаза процеса да би се уштедели рачунски напори. Овај приступ има за циљ очување рачунских ресурса, резервишући примену апроксимација функције циља високе тачности за завршне фазе процеса оптимизације. Детаљан опис предложених метода представљен је у поглављима 5 и 6, где су анализиране и теоријске особине нумеричких поступака, тј. доказана је њихова конвергенција и приказана сложеност развијених метода. Поред теоријског оквира, потврђена је успешна практична имплементација датих алгоритама. Показано је да су предложене методе ефикасније у практичној примени у односу на постојеће методе из литературе. Поглавље 1 ове тезе служи као основа за праћење наредних поглавља пружајући преглед основних појмова. Поглавље 2 се односи на нелинеарну оптимизацију, при чему је посебан акценат стављен на технике линијског претраживања. У поглављу 3 фокус се помера на семи-глатке проблеме оптимизације и методе за њихово решавање и служи као преглед постојећих резултата из ове области. Преостали делови тезе, почевши од поглавља 4, где се уводи проблем изучавања ове тезе (минимизација функције дате у облику очекиване вредности), па надаље, представљају оригинални допринос аутора.U brojnim problemima optimizacije koji potiču iz stvarnih i naučnih primena, često se suočavamo sa nediferencijabilnošću. U ovu klasu spada veliki broj problema, od modela prirodnih fenomena koji pokazuju nagle promene, optimizacije oblika, do funkcije cilja u mašinskom učenju i dubokim neuronskim mrežama. U praksi, rešavanje semi-glatkih konveksnih problema obično je izazovnije i zahteva veće računske troškove u odnosu na glatke probleme. Cilj ove teze je formulacija i teorijska analiza metoda NJutnovog tipa za rešavanje semi-glatkih konveksnih stohastičkih problema optimizacije. Razmatrani su problemi optimizacije sa funkcijom cilja datom u obliku matematičkog očekivanja bez pretpostavke o diferencijabilnosti funkcije. Kako je vrlo teško, pa nekad čak i nemoguće odrediti analitički oblik matematičkog očekivanja, funkcija cilja se aproksimira uzoračkim očekivanjem. Imajući u vidu da je tačnost aproksimacije funkcije cilja i njenih izvoda proporcionalna računskim troškovima – veća preciznost podrazumeva veće troškove u opštem slučaju, važno je dizajnirati efikasan balans između tačnosti i troškova. Stoga, glavni fokus ove teze je razvojalgoritama baziranih na određivanju optimalne dinamike uvećanja uzorka u semi-glatkom okruženju, sa posebnom pažnjom na kontroli tačnosti i odabiru pravaca pretrage. Po pitanju odabira pravca, razmotreno je nekoliko opcija, dok kontrola tačnosti uključuje jeftinije aproksimacije funkcije cilja (sa manjom preciznošću) tokom početnih faza procesa da bi se uštedeli računski napori. Ovaj pristup ima za cilj očuvanje računskih resursa, rezervišući primenu aproksimacija funkcije cilja visoke tačnosti za završne faze procesa optimizacije. Detaljan opis predloženih metoda predstavljen je u poglavljima 5 i 6, gde su analizirane i teorijske osobine numeričkih postupaka, tj. dokazana je njihova konvergencija i prikazana složenost razvijenih metoda. Pored teorijskog okvira, potvrđena je uspešna praktična implementacija datih algoritama. Pokazano je da su predložene metode efikasnije u praktičnoj primeni u odnosu na postojeće metode iz literature. Poglavlje 1 ove teze služi kao osnova za praćenje narednih poglavlja pružajući pregled osnovnih pojmova. Poglavlje 2 se odnosi na nelinearnu optimizaciju, pri čemu je poseban akcenat stavljen na tehnike linijskog pretraživanja. U poglavlju 3 fokus se pomera na semi-glatke probleme optimizacije i metode za njihovo rešavanje i služi kao pregled postojećih rezultata iz ove oblasti. Preostali delovi teze, počevši od poglavlja 4, gde se uvodi problem izučavanja ove teze (minimizacija funkcije date u obliku očekivane vrednosti), pa nadalje, predstavljaju originalni doprinos autora