Playing with Duality: An Overview of Recent Primal-Dual Approaches for Solving Large-Scale Optimization Problems

Optimization methods are at the core of many problems in signal/image processing, computer vision, and machine learning. For a long time, it has been recognized that looking at the dual of an optimization problem may drastically simplify its solution. Deriving efficient strategies which jointly brings into play the primal and the dual problems is however a more recent idea which has generated many important new contributions in the last years. These novel developments are grounded on recent advances in convex analysis, discrete optimization, parallel processing, and non-smooth optimization with emphasis on sparsity issues. In this paper, we aim at presenting the principles of primal-dual approaches, while giving an overview of numerical methods which have been proposed in different contexts. We show the benefits which can be drawn from primal-dual algorithms both for solving large-scale convex optimization problems and discrete ones, and we provide various application examples to illustrate their usefulness

Bi-Objective Nonnegative Matrix Factorization: Linear Versus Kernel-Based Models

Nonnegative matrix factorization (NMF) is a powerful class of feature extraction techniques that has been successfully applied in many fields, namely in signal and image processing. Current NMF techniques have been limited to a single-objective problem in either its linear or nonlinear kernel-based formulation. In this paper, we propose to revisit the NMF as a multi-objective problem, in particular a bi-objective one, where the objective functions defined in both input and feature spaces are taken into account. By taking the advantage of the sum-weighted method from the literature of multi-objective optimization, the proposed bi-objective NMF determines a set of nondominated, Pareto optimal, solutions instead of a single optimal decomposition. Moreover, the corresponding Pareto front is studied and approximated. Experimental results on unmixing real hyperspectral images confirm the efficiency of the proposed bi-objective NMF compared with the state-of-the-art methods

Computing the set of Epsilon-efficient solutions in multiobjective space mission design

In this work, we consider multiobjective space mission design problems. We will start from the need, from a practical point of view, to consider in addition to the (Pareto) optimal solutions also nearly optimal ones. In fact, extending the set of solutions for a given mission to those nearly optimal significantly increases the number of options for the decision maker and gives a measure of the size of the launch windows corresponding to each optimal solution, i.e., a measure of its robustness. Whereas the possible loss of such approximate solutions compared to optimal—and possibly even ‘better’—ones is dispensable. For this, we will examine several typical problems in space trajectory design—a biimpulsive transfer from the Earth to the asteroid Apophis and two low-thrust multigravity assist transfers—and demonstrate the possible benefit of the novel approach. Further, we will present a multiobjective evolutionary algorithm which is designed for this purpose

Low Complexity Regularization of Linear Inverse Problems

Inverse problems and regularization theory is a central theme in contemporary signal processing, where the goal is to reconstruct an unknown signal from partial indirect, and possibly noisy, measurements of it. A now standard method for recovering the unknown signal is to solve a convex optimization problem that enforces some prior knowledge about its structure. This has proved efficient in many problems routinely encountered in imaging sciences, statistics and machine learning. This chapter delivers a review of recent advances in the field where the regularization prior promotes solutions conforming to some notion of simplicity/low-complexity. These priors encompass as popular examples sparsity and group sparsity (to capture the compressibility of natural signals and images), total variation and analysis sparsity (to promote piecewise regularity), and low-rank (as natural extension of sparsity to matrix-valued data). Our aim is to provide a unified treatment of all these regularizations under a single umbrella, namely the theory of partial smoothness. This framework is very general and accommodates all low-complexity regularizers just mentioned, as well as many others. Partial smoothness turns out to be the canonical way to encode low-dimensional models that can be linear spaces or more general smooth manifolds. This review is intended to serve as a one stop shop toward the understanding of the theoretical properties of the so-regularized solutions. It covers a large spectrum including: (i) recovery guarantees and stability to noise, both in terms of $\ell^2$-stability and model (manifold) identification; (ii) sensitivity analysis to perturbations of the parameters involved (in particular the observations), with applications to unbiased risk estimation ; (iii) convergence properties of the forward-backward proximal splitting scheme, that is particularly well suited to solve the corresponding large-scale regularized optimization problem

Branch-and-lift algorithm for deterministic global optimization in nonlinear optimal control

This paper presents a branch-and-lift algorithm for solving optimal control problems with smooth nonlinear dynamics and potentially nonconvex objective and constraint functionals to guaranteed global optimality. This algorithm features a direct sequential method and builds upon a generic, spatial branch-and-bound algorithm. A new operation, called lifting, is introduced, which refines the control parameterization via a Gram-Schmidt orthogonalization process, while simultaneously eliminating control subregions that are either infeasible or that provably cannot contain any global optima. Conditions are given under which the image of the control parameterization error in the state space contracts exponentially as the parameterization order is increased, thereby making the lifting operation efficient. A computational technique based on ellipsoidal calculus is also developed that satisfies these conditions. The practical applicability of branch-and-lift is illustrated in a numerical example. © 2013 Springer Science+Business Media New York

Reduced basis method for source mask optimization

Image modeling and simulation are critical to extending the limits of leading edge lithography technologies used for IC making. Simultaneous source mask optimization (SMO) has become an important objective in the field of computational lithography. SMO is considered essential to extending immersion lithography beyond the 45nm node. However, SMO is computationally extremely challenging and time-consuming. The key challenges are due to run time vs. accuracy tradeoffs of the imaging models used for the computational lithography. We present a new technique to be incorporated in the SMO flow. This new approach is based on the reduced basis method (RBM) applied to the simulation of light transmission through the lithography masks. It provides a rigorous approximation to the exact lithographical problem, based on fully vectorial Maxwell's equations. Using the reduced basis method, the optimization process is divided into an offline and an online steps. In the offline step, a RBM model with variable geometrical parameters is built self-adaptively and using a Finite Element (FEM) based solver. In the online step, the RBM model can be solved very fast for arbitrary illumination and geometrical parameters, such as dimensions of OPC features, line widths, etc. This approach dramatically reduces computational costs of the optimization procedure while providing accuracy superior to the approaches involving simplified mask models. RBM furthermore provides rigorous error estimators, which assure the quality and reliability of the reduced basis solutions. We apply the reduced basis method to a 3D SMO example. We quantify performance, computational costs and accuracy of our method.Comment: BACUS Photomask Technology 201