18 research outputs found

    Transversality, regularity and error bounds in variational analysis and optimisation

    Get PDF
    Transversality properties of collections of sets, regularity properties of set-valued mappings, and error bounds of extended-real-valued functions lie at the core of variational analysis because of their importance for stability analysis, constraint qualifications, qualification conditions in coderivative and subdifferential calculus, and convergence analysis of numerical algorithms. The thesis is devoted to investigation of several research questions related to the aforementioned properties. We develop a general framework for quantitative analysis of nonlinear transversality properties by establishing primal and dual characterizations of the properties in both convex and nonconvex settings. The Hšolder case is given special attention. Quantitative relations between transversality properties and the corresponding regularity properties of set-valued mappings as well as nonlinear extensions of the new transversality properties of a set-valued mapping to a set in the range space are also discussed. We study a new property so called semitransversality of collections of set-valued mappings on metric (in particular, normed) spaces. The property is a generalization of the semitransversality of collections of sets and the negation of the corresponding stationarity, a weaker property than the extremality of collections of set-valued mappings. Primal and dual characterizations of the property as well as quantitative relations between the property and semiregularity of set-valued mappings are formulated. As a consequence, we establish dual necessary and sufficient conditions for stationarity of collections of set-valued mappings as well as optimality conditions for efficient solutions with respect to variable ordering structures in multiobjective optimization. We examine a comprehensive (i.e. not assuming the mapping to have any particular structure) view on the regularity theory of set-valued mappings and clarify the relationships between the existing primal and dual quantitative sufficient and necessary conditions including their hierarchy. The typical sequence of regularity assertions, often hidden in the proofs, and the roles of the assumptions involved in the assertions, in particular, on the underlying space: general metric, normed, Banach or Asplund are exposed. As a consequence, we formulate primal and dual conditions for the stability properties of solution mappings to inclusions. We propose a unifying general framework of quantitative primal and dual sufficient and necessary error bound conditions covering linear and nonlinear, local and global settings. The function is not assumed to possess any particular structure apart from the standard assumptions of lower semicontinuity in the case of sufficient conditions and (in some cases) convexity in the case of necessary conditions. We expose the roles of the assumptions involved in the error bound assertions, in particular, on the underlying space: general metric, normed, Banach or Asplund. As a consequence, the error bound theory is applied to characterize subregularity of set-valued mappings, and calmness of the solution mapping in convex semi-infinite optimization problems.Doctor of Philosoph

    International Conference on Continuous Optimization (ICCOPT) 2019 Conference Book

    Get PDF
    The Sixth International Conference on Continuous Optimization took place on the campus of the Technical University of Berlin, August 3-8, 2019. The ICCOPT is a flagship conference of the Mathematical Optimization Society (MOS), organized every three years. ICCOPT 2019 was hosted by the Weierstrass Institute for Applied Analysis and Stochastics (WIAS) Berlin. It included a Summer School and a Conference with a series of plenary and semi-plenary talks, organized and contributed sessions, and poster sessions. This book comprises the full conference program. It contains, in particular, the scientific program in survey style as well as with all details, and information on the social program, the venue, special meetings, and more

    Randomized Algorithms for Nonconvex Nonsmooth Optimization

    Get PDF
    Nonsmooth optimization problems arise in a variety of applications including robust control, robust optimization, eigenvalue optimization, compressed sensing, and decomposition methods for large-scale or complex optimization problems. When convexity is present, such problems are relatively easier to solve. Optimization methods for convex nonsmooth optimization have been studied for decades. For example, bundle methods are a leading technique for convex nonsmooth minimization. However, these and other methods that have been developed for solving convex problems are either inapplicable or can be inefficient when applied to solve nonconvex problems. The motivation of the work in this thesis is to design robust and efficient algorithms for solving nonsmooth optimization problems, particularly when nonconvexity is present.First, we propose an adaptive gradient sampling (AGS) algorithm, which is based on a recently developed technique known as the gradient sampling (GS) algorithm. Our AGS algorithm improves the computational efficiency of GS in critical ways. Then, we propose a BFGS gradient sampling (BFGS-GS) algorithm, which is a hybrid between a standard Broyden-Fletcher-Goldfarb-Shanno (BFGS) and the GS method. Our BFGS-GS algorithm is more efficient than our previously proposed AGS algorithm and also competitive with (and in some ways outperforms) other contemporary solvers for nonsmooth nonconvex optimization. Finally, we propose a few additional extensions of the GS framework---one in which we merge GS ideas with those from bundle methods, one in which we incorporate smoothing techniques in order to minimize potentially non-Lipschitz objective functions, and one in which we tailor GS methods for solving regularization problems. We describe all the proposed algorithms in detail. In addition, for all the algorithm variants, we prove global convergence guarantees under suitable assumptions. Moreover, we perform numerical experiments to illustrate the efficiency of our algorithms. The test problems considered in our experiments include academic test problems as well as practical problems that arise in applications of nonsmooth optimization

    Proceedings of the second "international Traveling Workshop on Interactions between Sparse models and Technology" (iTWIST'14)

    Get PDF
    The implicit objective of the biennial "international - Traveling Workshop on Interactions between Sparse models and Technology" (iTWIST) is to foster collaboration between international scientific teams by disseminating ideas through both specific oral/poster presentations and free discussions. For its second edition, the iTWIST workshop took place in the medieval and picturesque town of Namur in Belgium, from Wednesday August 27th till Friday August 29th, 2014. The workshop was conveniently located in "The Arsenal" building within walking distance of both hotels and town center. iTWIST'14 has gathered about 70 international participants and has featured 9 invited talks, 10 oral presentations, and 14 posters on the following themes, all related to the theory, application and generalization of the "sparsity paradigm": Sparsity-driven data sensing and processing; Union of low dimensional subspaces; Beyond linear and convex inverse problem; Matrix/manifold/graph sensing/processing; Blind inverse problems and dictionary learning; Sparsity and computational neuroscience; Information theory, geometry and randomness; Complexity/accuracy tradeoffs in numerical methods; Sparsity? What's next?; Sparse machine learning and inference.Comment: 69 pages, 24 extended abstracts, iTWIST'14 website: http://sites.google.com/site/itwist1

    Inertial and Second-order Optimization Algorithms for Training Neural Networks

    Get PDF
    Neural network models became highly popular during the last decade due to their efficiency in various applications. These are very large parametric models whose parameters must be set for each specific task. This crucial process of choosing the parameters, known as training, is done using large datasets. Due to the large amount of data and the size of the neural networks, the training phase is very expensive in terms of computational time and resources. From a mathematical point of view, training a neural network means solving a large-scale optimization problem. More specifically it involves the minimization of a sum of functions. The large-scale nature of the optimization problem highly restrains the types of algorithms available to minimize this sum of functions. In this context, standard algorithms almost exclusively rely on inexact gradients through the backpropagation method and mini-batch sub-sampling. As a result, firstorder methods such as stochastic gradient descent (SGD) remain the most used ones to train neural networks. Additionally, the function to minimize is non-convex and possibly nondifferentiable, resulting in limited convergence guarantees for these methods. In this thesis, we focus on building new algorithms exploiting second-order information only by means of noisy firstorder automatic differentiation. Starting from a dynamical system (an ordinary differential equation), we build INNA, an inertial and Newtonian algorithm. By analyzing together the dynamical system and INNA, we prove the convergence of the algorithm to the critical points of the function to minimize. Then, we show that the limit is actually a local minimum with overwhelming probability. Finally, we introduce Step-Tuned SGD that automatically adjusts the step-sizes of SGD. It does so by cleverly modifying the mini-batch sub-sampling allowing for an efficient discretization of second-order information. We prove the almost sure convergence of Step-Tuned SGD to critical points and provide rates of convergence. All the theoretical results are backed by promising numerical experiments on deep learning problems

    Acceleration Methods

    Full text link
    This monograph covers some recent advances in a range of acceleration techniques frequently used in convex optimization. We first use quadratic optimization problems to introduce two key families of methods, namely momentum and nested optimization schemes. They coincide in the quadratic case to form the Chebyshev method. We discuss momentum methods in detail, starting with the seminal work of Nesterov and structure convergence proofs using a few master templates, such as that for optimized gradient methods, which provide the key benefit of showing how momentum methods optimize convergence guarantees. We further cover proximal acceleration, at the heart of the Catalyst and Accelerated Hybrid Proximal Extragradient frameworks, using similar algorithmic patterns. Common acceleration techniques rely directly on the knowledge of some of the regularity parameters in the problem at hand. We conclude by discussing restart schemes, a set of simple techniques for reaching nearly optimal convergence rates while adapting to unobserved regularity parameters.Comment: Published in Foundation and Trends in Optimization (see https://www.nowpublishers.com/article/Details/OPT-036
    corecore