Newton-type methods under generalized self-concordance and inexact oracles

Many modern applications in machine learning, image/signal processing, and statistics require to solve large-scale convex optimization problems. These problems share some common challenges such as high-dimensionality, nonsmoothness, and complex objectives and constraints. Due to these challenges, the theoretical assumptions for existing numerical methods are not satisfied. In numerical methods, it is also impractical to do exact computations in many cases (e.g. noisy computation, storage or time limitation). Therefore, new approaches as well as inexact computations to design new algorithms should be considered. In this thesis, we develop fundamental theories and numerical methods, especially second-order methods, to solve some classes of convex optimization problems, where first-order methods are inefficient or do not have a theoretical guarantee. We aim at exploiting the underlying smoothness structures of the problem to design novel Newton-type methods. More specifically, we generalize a powerful concept called \mbox{self-concordance} introduced by Nesterov and Nemirovski to a broader class of convex functions. We develop several basic properties of this concept and prove key estimates for function values and its derivatives. Then, we apply our theory to design different Newton-type methods such as damped-step Newton methods, full-step Newton methods, and proximal Newton methods. Our new theory allows us to establish both global and local convergence guarantees of these methods without imposing unverifiable conditions as in classical Newton-type methods. Numerical experiments show that our approach has several advantages compared to existing works. In the second part of this thesis, we introduce new global and local inexact oracle settings, and apply them to develop inexact proximal Newton-type schemes for optimizing general composite convex problems equipped with such inexact oracles. These schemes allow us to measure errors theoretically and systematically and still lead to desired convergence results. Moreover, they can be applied to solve a wider class of applications arising in statistics and machine learning.Doctor of Philosoph

Inexact proximal DC Newton-type method for nonconvex composite functions

We consider a class of difference-of-convex (DC) optimization problems where the objective function is the sum of a smooth function and a possible nonsmooth DC function. The application of proximal DC algorithms to address this problem class is well-known. In this paper, we combine a proximal DC algorithm with an inexact proximal Newton-type method to propose an inexact proximal DC Newton-type method. We demonstrate global convergence properties of the proposed method. In addition, we give a memoryless quasi-Newton matrix for scaled proximal mappings and consider a two-dimensional system of semi-smooth equations that arise in calculating scaled proximal mappings. To efficiently obtain the scaled proximal mappings, we adopt a semi-smooth Newton method to inexactly solve the system. Finally, we present some numerical experiments to investigate the efficiency of the proposed method, showing that the proposed method outperforms existing methods

A Riemannian Proximal Newton Method

In recent years, the proximal gradient method and its variants have been generalized to Riemannian manifolds for solving optimization problems with an additively separable structure, i.e., $f + h$, where $f$ is continuously differentiable, and $h$ may be nonsmooth but convex with computationally reasonable proximal mapping. In this paper, we generalize the proximal Newton method to embedded submanifolds for solving the type of problem with $h(x) = \mu \|x\|_1$. The generalization relies on the Weingarten and semismooth analysis. It is shown that the Riemannian proximal Newton method has a local superlinear convergence rate under certain reasonable assumptions. Moreover, a hybrid version is given by concatenating a Riemannian proximal gradient method and the Riemannian proximal Newton method. It is shown that if the objective function satisfies the Riemannian KL property and the switch parameter is chosen appropriately, then the hybrid method converges globally and also has a local superlinear convergence rate. Numerical experiments on random and synthetic data are used to demonstrate the performance of the proposed methods