Best Subset Selection via a Modern Optimization Lens

In the last twenty-five years (1990-2014), algorithmic advances in integer optimization combined with hardware improvements have resulted in an astonishing 200 billion factor speedup in solving Mixed Integer Optimization (MIO) problems. We present a MIO approach for solving the classical best subset selection problem of choosing $k$ out of $p$ features in linear regression given $n$ observations. We develop a discrete extension of modern first order continuous optimization methods to find high quality feasible solutions that we use as warm starts to a MIO solver that finds provably optimal solutions. The resulting algorithm (a) provides a solution with a guarantee on its suboptimality even if we terminate the algorithm early, (b) can accommodate side constraints on the coefficients of the linear regression and (c) extends to finding best subset solutions for the least absolute deviation loss function. Using a wide variety of synthetic and real datasets, we demonstrate that our approach solves problems with $n$ in the 1000s and $p$ in the 100s in minutes to provable optimality, and finds near optimal solutions for $n$ in the 100s and $p$ in the 1000s in minutes. We also establish via numerical experiments that the MIO approach performs better than {\texttt {Lasso}} and other popularly used sparse learning procedures, in terms of achieving sparse solutions with good predictive power.Comment: This is a revised version (May, 2015) of the first submission in June 201

Robust methods for inferring sparse network structures

This is the post-print version of the final paper published in Computational Statistics & Data Analysis. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2013 Elsevier B.V.Networks appear in many fields, from finance to medicine, engineering, biology and social science. They often comprise of a very large number of entities, the nodes, and the interest lies in inferring the interactions between these entities, the edges, from relatively limited data. If the underlying network of interactions is sparse, two main statistical approaches are used to retrieve such a structure: covariance modeling approaches with a penalty constraint that encourages sparsity of the network, and nodewise regression approaches with sparse regression methods applied at each node. In the presence of outliers or departures from normality, robust approaches have been developed which relax the assumption of normality. Robust covariance modeling approaches are reviewed and compared with novel nodewise approaches where robust methods are used at each node. For low-dimensional problems, classical deviance tests are also included and compared with penalized likelihood approaches. Overall, copula approaches are found to perform best: they are comparable to the other methods under an assumption of normality or mild departures from this, but they are superior to the other methods when the assumption of normality is strongly violated

Least quantile regression via modern optimization

We address the Least Quantile of Squares (LQS) (and in particular the Least Median of Squares) regression problem using modern optimization methods. We propose a Mixed Integer Optimization (MIO) formulation of the LQS problem which allows us to find a provably global optimal solution for the LQS problem. Our MIO framework has the appealing characteristic that if we terminate the algorithm early, we obtain a solution with a guarantee on its sub-optimality. We also propose continuous optimization methods based on first-order subdifferential methods, sequential linear optimization and hybrid combinations of them to obtain near optimal solutions to the LQS problem. The MIO algorithm is found to benefit significantly from high quality solutions delivered by our continuous optimization based methods. We further show that the MIO approach leads to (a) an optimal solution for any dataset, where the data-points $(y_i,\mathbf{x}_i)$'s are not necessarily in general position, (b) a simple proof of the breakdown point of the LQS objective value that holds for any dataset and (c) an extension to situations where there are polyhedral constraints on the regression coefficient vector. We report computational results with both synthetic and real-world datasets showing that the MIO algorithm with warm starts from the continuous optimization methods solve small ($n=100$) and medium ($n=500$) size problems to provable optimality in under two hours, and outperform all publicly available methods for large-scale ($n={}$10,000) LQS problems.Comment: Published in at http://dx.doi.org/10.1214/14-AOS1223 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

Convex and non-convex regularization methods for spatial point processes intensity estimation

This paper deals with feature selection procedures for spatial point processes intensity estimation. We consider regularized versions of estimating equations based on Campbell theorem derived from two classical functions: Poisson likelihood and logistic regression likelihood. We provide general conditions on the spatial point processes and on penalty functions which ensure consistency, sparsity and asymptotic normality. We discuss the numerical implementation and assess finite sample properties in a simulation study. Finally, an application to tropical forestry datasets illustrates the use of the proposed methods

Explaining Africa’s public consumption procyclicality : revisiting old evidence

This paper compiles a novel dataset of time-varying measures of government consumption cyclicality for a panel of 46 African economies between 1960 and 2014. Government consumption has, generally, been highly procyclical over time in this group of countries. However, sample averages hide serious heterogeneity across countries with the majority of them showing procyclical behavior despite some positive signs of graduation from the “procyclicality trap” in a few cases. By means of weighted least squares regressions, we find that more developed African economies tend to have a smaller degree of government consumption procyclicality. Countries with higher social fragmentation and those are more reliant on foreign aid inflows tend to have a more procyclical government consumption policy. Better governance promotes counter- cyclical fiscal policy whileincreased democracy dampens it. Finally, some fiscal rules are important in curbing the procyclical behavior of government consumption.info:eu-repo/semantics/publishedVersio

Model of Robust Regression with Parametric and Nonparametric Methods

In the present work, we evaluate the performance of the classical parametric estimation method "ordinary least squares" with the classical nonparametric estimation methods, some robust estimation methods and two suggested methods for conditions in which varying degrees and directions of outliers are presented in the observed data. The study addresses the problem via computer simulation methods. In order to cover the effects of various situations of outliers on the simple linear regression model, samples were classified into four cases (no outliers, outliers in the X-direction, outliers in the Y-direction and outliers in the XY-direction) and the percentages of outliers are varied between 10%, 20% and 30%. The performances of estimators are evaluated in respect to their mean squares error and relative mean squares error. Keywords: Simple Linear Regression model; Ordinary Least Squares Method; Nonparametric Regression; Robust Regression; Least Absolute Deviations Regression; M-Estimation Regression; Trimmed Least Squares Regression

Using Stability to Select a Shrinkage Method

Shrinkage methods are estimation techniques based on optimizing expressions to find which variables to include in an analysis, typically a linear regression. The general form of these expressions is the sum of an empirical risk plus a complexity penalty based on the number of parameters. Many shrinkage methods are known to satisfy an ‘oracle’ property meaning that asymptotically they select the correct variables and estimate their coefficients efficiently. In Section 1.2, we show oracle properties in two general settings. The first uses a log likelihood in place of the empirical risk and allows a general class of penalties. The second uses a general class of empirical risks and a general class of penalties obtaining limiting behavior for a large class of smooth likelihoods. The second contribution of this thesis is to realize that shrinkage techniques with oracle properties are asymptotically the same, but differ in their finite sample properties. To address this, in Section 2.1, we propose selection of a shrinkage method based on a stability criterion. Part of our analysis in Section 2.2 is a computational comparison of several specific shrinkage methods. In future work, we hope to optimize a stability criterion directly to derive a data driven shrinkage method using techniques from genetic algorithms. We describe this in Section 2.3 as future work. Adviser: Bertrand Clark