An Exponential Lower Bound on the Complexity of Regularization Paths

For a variety of regularized optimization problems in machine learning, algorithms computing the entire solution path have been developed recently. Most of these methods are quadratic programs that are parameterized by a single parameter, as for example the Support Vector Machine (SVM). Solution path algorithms do not only compute the solution for one particular value of the regularization parameter but the entire path of solutions, making the selection of an optimal parameter much easier. It has been assumed that these piecewise linear solution paths have only linear complexity, i.e. linearly many bends. We prove that for the support vector machine this complexity can be exponential in the number of training points in the worst case. More strongly, we construct a single instance of n input points in d dimensions for an SVM such that at least \Theta(2^{n/2}) = \Theta(2^d) many distinct subsets of support vectors occur as the regularization parameter changes.Comment: Journal version, 28 Pages, 5 Figure

Inverse Optimization with Noisy Data

Inverse optimization refers to the inference of unknown parameters of an optimization problem based on knowledge of its optimal solutions. This paper considers inverse optimization in the setting where measurements of the optimal solutions of a convex optimization problem are corrupted by noise. We first provide a formulation for inverse optimization and prove it to be NP-hard. In contrast to existing methods, we show that the parameter estimates produced by our formulation are statistically consistent. Our approach involves combining a new duality-based reformulation for bilevel programs with a regularization scheme that smooths discontinuities in the formulation. Using epi-convergence theory, we show the regularization parameter can be adjusted to approximate the original inverse optimization problem to arbitrary accuracy, which we use to prove our consistency results. Next, we propose two solution algorithms based on our duality-based formulation. The first is an enumeration algorithm that is applicable to settings where the dimensionality of the parameter space is modest, and the second is a semiparametric approach that combines nonparametric statistics with a modified version of our formulation. These numerical algorithms are shown to maintain the statistical consistency of the underlying formulation. Lastly, using both synthetic and real data, we demonstrate that our approach performs competitively when compared with existing heuristics

Strong Stationarity Conditions for Optimal Control of Hybrid Systems

We present necessary and sufficient optimality conditions for finite time optimal control problems for a class of hybrid systems described by linear complementarity models. Although these optimal control problems are difficult in general due to the presence of complementarity constraints, we provide a set of structural assumptions ensuring that the tangent cone of the constraints possesses geometric regularity properties. These imply that the classical Karush-Kuhn-Tucker conditions of nonlinear programming theory are both necessary and sufficient for local optimality, which is not the case for general mathematical programs with complementarity constraints. We also present sufficient conditions for global optimality. We proceed to show that the dynamics of every continuous piecewise affine system can be written as the optimizer of a mathematical program which results in a linear complementarity model satisfying our structural assumptions. Hence, our stationarity results apply to a large class of hybrid systems with piecewise affine dynamics. We present simulation results showing the substantial benefits possible from using a nonlinear programming approach to the optimal control problem with complementarity constraints instead of a more traditional mixed-integer formulation.Comment: 30 pages, 4 figure