On generalized semi-infinite optimization and bilevel optimization

The paper studies the connections and differences between bilevel problems (BL) and generalized semi-infinite problems (GSIP). Under natural assumptions (GSIP) can be seen as a special case of a (BL). We consider the so-called reduction approach for (BL) and (GSIP) leading to optimality conditions and Newton-type methods for solving the problems. We show by a structural analysis that for (GSIP)-problems the regularity assumptions for the reduction approach can be expected to hold generically at a solution but for general (BL)-problems not. The genericity behavior of (BL) and (GSIP) is in particular studied for linear problems

Semi-infinite optimization: Structure and stability of the feasible set

The problem of the minimization of a functionf: ℝn→ℝ under finitely many equality constraints and perhaps infinitely many inequality constraints gives rise to a structural analysis of the feasible setM[H, G]={x∈ℝn¦H(x)=0,G(x, y)≥0,y∈Y} with compactY⊂ℝr. An extension of the well-known Mangasarian-Fromovitz constraint qualification (EMFCQ) is introduced. The main result for compactM[H, G] is the equivalence of the topological stability of the feasible setM[H, G] and the validity of EMFCQ. As a byproduct, we obtain under EMFCQ that the feasible set admits local linearizations and also thatM[H, G] depends continuously on the pair (H, G). Moreover, EMFCQ is shown to be satisfied generically

The Loss Rank Principle for Model Selection

We introduce a new principle for model selection in regression and classification. Many regression models are controlled by some smoothness or flexibility or complexity parameter c, e.g. the number of neighbors to be averaged over in k nearest neighbor (kNN) regression or the polynomial degree in regression with polynomials. Let f_D^c be the (best) regressor of complexity c on data D. A more flexible regressor can fit more data D' well than a more rigid one. If something (here small loss) is easy to achieve it's typically worth less. We define the loss rank of f_D^c as the number of other (fictitious) data D' that are fitted better by f_D'^c than D is fitted by f_D^c. We suggest selecting the model complexity c that has minimal loss rank (LoRP). Unlike most penalized maximum likelihood variants (AIC,BIC,MDL), LoRP only depends on the regression function and loss function. It works without a stochastic noise model, and is directly applicable to any non-parametric regressor, like kNN. In this paper we formalize, discuss, and motivate LoRP, study it for specific regression problems, in particular linear ones, and compare it to other model selection schemes.Comment: 16 page

A Generalized Framework for Virtual Substitution

We generalize the framework of virtual substitution for real quantifier elimination to arbitrary but bounded degrees. We make explicit the representation of test points in elimination sets using roots of parametric univariate polynomials described by Thom codes. Our approach follows an early suggestion by Weispfenning, which has never been carried out explicitly. Inspired by virtual substitution for linear formulas, we show how to systematically construct elimination sets containing only test points representing lower bounds

Mathematical programs with complementarity constraints: convergence properties of a smoothing method

In this paper, optimization problems $P$ with complementarity constraints are considered. Characterizations for local minimizers $\bar{x}$ of $P$ of Orders 1 and 2 are presented. We analyze a parametric smoothing approach for solving these programs in which $P$ is replaced by a perturbed problem $P_{\tau}$ depending on a (small) parameter $\tau$. We are interested in the convergence behavior of the feasible set $\cal{F}_{\tau}$ and the convergence of the solutions $\bar{x}_{\tau}$ of $P_{\tau}$ for $\tau\to 0.$ In particular, it is shown that, under generic assumptions, the solutions $\bar{x}_{\tau}$ are unique and converge to a solution $\bar{x}$ of $P$ with a rate $\cal{O}(\sqrt{\tau})$. Moreover, the convergence for the Hausdorff distance $d(\cal{F}_{\tau}$, $\cal{F})$ between the feasible sets of $P_{\tau}$ and $P$ is of order $\cal{O}(\sqrt{\tau})$

Demystifying Deep Learning: A Geometric Approach to Iterative Projections

Parametric approaches to Learning, such as deep learning (DL), are highly popular in nonlinear regression, in spite of their extremely difficult training with their increasing complexity (e.g. number of layers in DL). In this paper, we present an alternative semi-parametric framework which foregoes the ordinarily required feedback, by introducing the novel idea of geometric regularization. We show that certain deep learning techniques such as residual network (ResNet) architecture are closely related to our approach. Hence, our technique can be used to analyze these types of deep learning. Moreover, we present preliminary results which confirm that our approach can be easily trained to obtain complex structures.Comment: To be appeared in the ICASSP 2018 proceeding