Constrained Deep Networks: Lagrangian Optimization via Log-Barrier Extensions

This study investigates the optimization aspects of imposing hard inequality constraints on the outputs of CNNs. In the context of deep networks, constraints are commonly handled with penalties for their simplicity, and despite their well-known limitations. Lagrangian-dual optimization has been largely avoided, except for a few recent works, mainly due to the computational complexity and stability/convergence issues caused by alternating explicit dual updates/projections and stochastic optimization. Several studies showed that, surprisingly for deep CNNs, the theoretical and practical advantages of Lagrangian optimization over penalties do not materialize in practice. We propose log-barrier extensions, which approximate Lagrangian optimization of constrained-CNN problems with a sequence of unconstrained losses. Unlike standard interior-point and log-barrier methods, our formulation does not need an initial feasible solution. Furthermore, we provide a new technical result, which shows that the proposed extensions yield an upper bound on the duality gap. This generalizes the duality-gap result of standard log-barriers, yielding sub-optimality certificates for feasible solutions. While sub-optimality is not guaranteed for non-convex problems, our result shows that log-barrier extensions are a principled way to approximate Lagrangian optimization for constrained CNNs via implicit dual variables. We report comprehensive weakly supervised segmentation experiments, with various constraints, showing that our formulation outperforms substantially the existing constrained-CNN methods, both in terms of accuracy, constraint satisfaction and training stability, more so when dealing with a large number of constraints

Multiplier-continuation algorthms for constrained optimization

Several path following algorithms based on the combination of three smooth penalty functions, the quadratic penalty for equality constraints and the quadratic loss and log barrier for inequality constraints, their modern counterparts, augmented Lagrangian or multiplier methods, sequential quadratic programming, and predictor-corrector continuation are described. In the first phase of this methodology, one minimizes the unconstrained or linearly constrained penalty function or augmented Lagrangian. A homotopy path generated from the functions is then followed to optimality using efficient predictor-corrector continuation methods. The continuation steps are asymptotic to those taken by sequential quadratic programming which can be used in the final steps. Numerical test results show the method to be efficient, robust, and a competitive alternative to sequential quadratic programming

Constrained nonlinear optimal control: a converse HJB approach

Extending the concept of solving the Hamilton-Jacobi-Bellman (HJB) optimization equation backwards [2], the so called converse constrained optimal control problem is introduced, and used to create various classes of nonlinear systems for which the optimal controller subject to constraints is known. In this way a systematic method for the testing, validation and comparison of different control techniques with the optimal is established. Because it naturally and explicitly handles constraints, particularly control input saturation, model predictive control (MPC) is a potentially powerful approach for nonlinear control design. However, nonconvexity of the nonlinear programs (NLP) involved in the MPC optimization makes the solution problematic. In order to explore properties of MPC-based constrained control schemes, and to point out the potential issues in implementing MPC, challenging benchmark examples are generated and analyzed. Properties of MPC-based constrained techniques are then evaluated and implementation issues are explored by applying both nonlinear MPC and MPC with feedback linearization

Portfolio selection problems in practice: a comparison between linear and quadratic optimization models

Several portfolio selection models take into account practical limitations on the number of assets to include and on their weights in the portfolio. We present here a study of the Limited Asset Markowitz (LAM), of the Limited Asset Mean Absolute Deviation (LAMAD) and of the Limited Asset Conditional Value-at-Risk (LACVaR) models, where the assets are limited with the introduction of quantity and cardinality constraints. We propose a completely new approach for solving the LAM model, based on reformulation as a Standard Quadratic Program and on some recent theoretical results. With this approach we obtain optimal solutions both for some well-known financial data sets used by several other authors, and for some unsolved large size portfolio problems. We also test our method on five new data sets involving real-world capital market indices from major stock markets. Our computational experience shows that, rather unexpectedly, it is easier to solve the quadratic LAM model with our algorithm, than to solve the linear LACVaR and LAMAD models with CPLEX, one of the best commercial codes for mixed integer linear programming (MILP) problems. Finally, on the new data sets we have also compared, using out-of-sample analysis, the performance of the portfolios obtained by the Limited Asset models with the performance provided by the unconstrained models and with that of the official capital market indices

Final-State Constrained Optimal Control via a Projection Operator Approach

In this paper we develop a numerical method to solve nonlinear optimal control problems with final-state constraints. Specifically, we extend the PRojection Operator based Netwon's method for Trajectory Optimization (PRONTO), which was proposed by Hauser for unconstrained optimal control problems. While in the standard method final-state constraints can be only approximately handled by means of a terminal penalty, in this work we propose a methodology to meet the constraints exactly. Moreover, our method guarantees recursive feasibility of the final-state constraint. This is an appealing property especially in realtime applications in which one would like to be able to stop the computation even if the desired tolerance has not been reached, but still satisfy the constraints. Following the same conceptual idea of PRONTO, the proposed strategy is based on two main steps which (differently from the standard scheme) preserve the feasibility of the final-state constraints: (i) solve a quadratic approximation of the nonlinear problem to find a descent direction, and (ii) get a (feasible) trajectory by means of a feedback law (which turns out to be a nonlinear projection operator). To find the (feasible) descent direction we take advantage of final-state constrained Linear Quadratic optimal control methods, while the second step is performed by suitably designing a constrained version of the trajectory tracking projection operator. The effectiveness of the proposed strategy is tested on the optimal state transfer of an inverted pendulum

Optimal exact designs of experiments via Mixed Integer Nonlinear Programming

Optimal exact designs are problematic to find and study because there is no unified theory for determining them and studyingtheir properties. Each has its own challenges and when a method exists to confirm the design optimality, it is invariablyapplicable to the particular problem only.We propose a systematic approach to construct optimal exact designs by incorporatingthe Cholesky decomposition of the Fisher Information Matrix in a Mixed Integer Nonlinear Programming formulation. Asexamples, we apply the methodology to find D- and A-optimal exact designs for linear and nonlinear models using global orlocal optimizers. Our examples include design problems with constraints on the locations or the number of replicates at theoptimal design points