GIANT: Globally Improved Approximate Newton Method for Distributed Optimization

For distributed computing environment, we consider the empirical risk minimization problem and propose a distributed and communication-efficient Newton-type optimization method. At every iteration, each worker locally finds an Approximate NewTon (ANT) direction, which is sent to the main driver. The main driver, then, averages all the ANT directions received from workers to form a {\it Globally Improved ANT} (GIANT) direction. GIANT is highly communication efficient and naturally exploits the trade-offs between local computations and global communications in that more local computations result in fewer overall rounds of communications. Theoretically, we show that GIANT enjoys an improved convergence rate as compared with first-order methods and existing distributed Newton-type methods. Further, and in sharp contrast with many existing distributed Newton-type methods, as well as popular first-order methods, a highly advantageous practical feature of GIANT is that it only involves one tuning parameter. We conduct large-scale experiments on a computer cluster and, empirically, demonstrate the superior performance of GIANT.Comment: Fixed some typos. Improved writin

Conic Optimization Theory: Convexification Techniques and Numerical Algorithms

Optimization is at the core of control theory and appears in several areas of this field, such as optimal control, distributed control, system identification, robust control, state estimation, model predictive control and dynamic programming. The recent advances in various topics of modern optimization have also been revamping the area of machine learning. Motivated by the crucial role of optimization theory in the design, analysis, control and operation of real-world systems, this tutorial paper offers a detailed overview of some major advances in this area, namely conic optimization and its emerging applications. First, we discuss the importance of conic optimization in different areas. Then, we explain seminal results on the design of hierarchies of convex relaxations for a wide range of nonconvex problems. Finally, we study different numerical algorithms for large-scale conic optimization problems.Comment: 18 page

A semismooth newton method for the nearest Euclidean distance matrix problem

The Nearest Euclidean distance matrix problem (NEDM) is a fundamentalcomputational problem in applications such asmultidimensional scaling and molecularconformation from nuclear magnetic resonance data in computational chemistry.Especially in the latter application, the problem is often large scale with the number ofatoms ranging from a few hundreds to a few thousands.In this paper, we introduce asemismooth Newton method that solves the dual problem of (NEDM). We prove that themethod is quadratically convergent.We then present an application of the Newton method to NEDM with $H$-weights.We demonstrate the superior performance of the Newton method over existing methodsincluding the latest quadratic semi-definite programming solver.This research also opens a new avenue towards efficient solution methods for the molecularembedding problem