Logistic Regression: Tight Bounds for Stochastic and Online Optimization

The logistic loss function is often advocated in machine learning and statistics as a smooth and strictly convex surrogate for the 0-1 loss. In this paper we investigate the question of whether these smoothness and convexity properties make the logistic loss preferable to other widely considered options such as the hinge loss. We show that in contrast to known asymptotic bounds, as long as the number of prediction/optimization iterations is sub exponential, the logistic loss provides no improvement over a generic non-smooth loss function such as the hinge loss. In particular we show that the convergence rate of stochastic logistic optimization is bounded from below by a polynomial in the diameter of the decision set and the number of prediction iterations, and provide a matching tight upper bound. This resolves the COLT open problem of McMahan and Streeter (2012)

Beyond Convexity: Stochastic Quasi-Convex Optimization

Stochastic convex optimization is a basic and well studied primitive in machine learning. It is well known that convex and Lipschitz functions can be minimized efficiently using Stochastic Gradient Descent (SGD). The Normalized Gradient Descent (NGD) algorithm, is an adaptation of Gradient Descent, which updates according to the direction of the gradients, rather than the gradients themselves. In this paper we analyze a stochastic version of NGD and prove its convergence to a global minimum for a wider class of functions: we require the functions to be quasi-convex and locally-Lipschitz. Quasi-convexity broadens the con- cept of unimodality to multidimensions and allows for certain types of saddle points, which are a known hurdle for first-order optimization methods such as gradient descent. Locally-Lipschitz functions are only required to be Lipschitz in a small region around the optimum. This assumption circumvents gradient explosion, which is another known hurdle for gradient descent variants. Interestingly, unlike the vanilla SGD algorithm, the stochastic normalized gradient descent algorithm provably requires a minimal minibatch size

On Graduated Optimization for Stochastic Non-Convex Problems

The graduated optimization approach, also known as the continuation method, is a popular heuristic to solving non-convex problems that has received renewed interest over the last decade. Despite its popularity, very little is known in terms of theoretical convergence analysis. In this paper we describe a new first-order algorithm based on graduated optimiza- tion and analyze its performance. We characterize a parameterized family of non- convex functions for which this algorithm provably converges to a global optimum. In particular, we prove that the algorithm converges to an {\epsilon}-approximate solution within O(1/\epsilon^2) gradient-based steps. We extend our algorithm and analysis to the setting of stochastic non-convex optimization with noisy gradient feedback, attaining the same convergence rate. Additionally, we discuss the setting of zero-order optimization, and devise a a variant of our algorithm which converges at rate of O(d^2/\epsilon^4).Comment: 17 page

Quantification of hemodynamic changes induced by virtual placement of multiple stents across a wide -necked Basilar trunk aneurysm

OBJECTIVE: The porous intravascular stents that are currently available may not cause complete aneurysm thrombosis and may therefore fail to provide durable protection against aneurysm rupture when used as a sole treatment modality. The goal of this study was to quantify the effects of porous stents on aneurysm hemodynamics using computational fluid dynamics. METHODS: The geometry of a wide-necked saccular basilar trunk aneurysm was reconstructed from a patient’s computed tomographic angiography images. Three commercial stents (Neuroform2; Boston Scientific/Target, San Leandro, CA; Wingspan; Boston Scientific, Fremont, CA; and Vision; Guidant Corp., Santa Clara, CA) were modeled. Various combinations of one to three stents were virtually conformed to fit into the vessel lumen and placed across the aneurysm orifice. An unstented aneurysm served as a control. Computational fluid dynamics analysis was performed to calculate the hemodynamic parameters considered important in aneurysm pathogenesis and thrombosis for each of the models. RESULTS: The complex flow pattern observed in the unstented aneurysm was suppressed by stenting. Stent placement lowered the wall shear stress in the aneurysm, and this effect was increased by additional stent deployment. Turnover time was moderately increased after single- and double-stent placement and markedly increased after three stents were placed. The influence of stent design on hemodynamic parameters was more significant in double-stented models than in other models. CONCLUSION: Aneurysm hemodynamic parameters were significantly modified by placement of multiple stents. Because the associated modifications may be helpful as well as harmful in terms of rupture risk, use of this technique requires careful consideration