8,215 research outputs found

    Newton-type methods under generalized self-concordance and inexact oracles

    Get PDF
    Many modern applications in machine learning, image/signal processing, and statistics require to solve large-scale convex optimization problems. These problems share some common challenges such as high-dimensionality, nonsmoothness, and complex objectives and constraints. Due to these challenges, the theoretical assumptions for existing numerical methods are not satisfied. In numerical methods, it is also impractical to do exact computations in many cases (e.g. noisy computation, storage or time limitation). Therefore, new approaches as well as inexact computations to design new algorithms should be considered. In this thesis, we develop fundamental theories and numerical methods, especially second-order methods, to solve some classes of convex optimization problems, where first-order methods are inefficient or do not have a theoretical guarantee. We aim at exploiting the underlying smoothness structures of the problem to design novel Newton-type methods. More specifically, we generalize a powerful concept called \mbox{self-concordance} introduced by Nesterov and Nemirovski to a broader class of convex functions. We develop several basic properties of this concept and prove key estimates for function values and its derivatives. Then, we apply our theory to design different Newton-type methods such as damped-step Newton methods, full-step Newton methods, and proximal Newton methods. Our new theory allows us to establish both global and local convergence guarantees of these methods without imposing unverifiable conditions as in classical Newton-type methods. Numerical experiments show that our approach has several advantages compared to existing works. In the second part of this thesis, we introduce new global and local inexact oracle settings, and apply them to develop inexact proximal Newton-type schemes for optimizing general composite convex problems equipped with such inexact oracles. These schemes allow us to measure errors theoretically and systematically and still lead to desired convergence results. Moreover, they can be applied to solve a wider class of applications arising in statistics and machine learning.Doctor of Philosoph

    Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information

    Full text link
    We consider variants of trust-region and cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under mild conditions on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve ϵ \epsilon -approximate second-order optimality which have shown to be tight. Our Hessian approximation conditions constitute a major relaxation over the existing ones in the literature. Consequently, we are able to show that such mild conditions allow for the construction of the approximate Hessian through various random sampling methods. In this light, we consider the canonical problem of finite-sum minimization, provide appropriate uniform and non-uniform sub-sampling strategies to construct such Hessian approximations, and obtain optimal iteration complexity for the corresponding sub-sampled trust-region and cubic regularization methods.Comment: 32 page

    Inexact proximal DC Newton-type method for nonconvex composite functions

    Full text link
    We consider a class of difference-of-convex (DC) optimization problems where the objective function is the sum of a smooth function and a possible nonsmooth DC function. The application of proximal DC algorithms to address this problem class is well-known. In this paper, we combine a proximal DC algorithm with an inexact proximal Newton-type method to propose an inexact proximal DC Newton-type method. We demonstrate global convergence properties of the proposed method. In addition, we give a memoryless quasi-Newton matrix for scaled proximal mappings and consider a two-dimensional system of semi-smooth equations that arise in calculating scaled proximal mappings. To efficiently obtain the scaled proximal mappings, we adopt a semi-smooth Newton method to inexactly solve the system. Finally, we present some numerical experiments to investigate the efficiency of the proposed method, showing that the proposed method outperforms existing methods

    Inexact Newton-Type Optimization with Iterated Sensitivities

    Get PDF
    This paper presents and analyzes an Inexact Newton-type optimization method based on Iterated Sensitivities (INIS). A particular class of Nonlinear Programming (NLP) problems is considered, where a subset of the variables is defined by nonlinear equality constraints. The proposed algorithm considers any problem-specific approximation for the Jacobian of these constraints. Unlike other inexact Newton methods, the INIS-type optimization algorithm is shown to preserve the local convergence properties and the asymptotic contraction rate of the Newton-type scheme for the feasibility problem, yielded by the same Jacobian approximation. The INIS approach results in a computational cost which can be made close to that of the standard inexact Newton implementation. In addition, an adjoint-free (AF-INIS) variant of the approach is presented which, under certain conditions, becomes considerably easier to implement than the adjoint based scheme. The applicability of these results is motivated, specifically for dynamic optimization problems. In addition, the numerical performance of a specific open-source implementation is illustrated

    On the local convergence of inexact Newton-type methods under residual control-type conditions

    Get PDF
    AbstractA local convergence analysis of inexact Newton-type methods using a new type of residual control was recently presented by C. Li and W. Shen. Here, we introduce the center-Hölder condition on the operator involved, and use it in combination with the Hölder condition to provide a new local convergence analysis with the following advantages: larger radius of convergence, and tighter error bounds on the distances involved. These results are obtained under the same hypotheses and computational cost. Numerical examples further validating the theoretical results are also provided in this study
    • …
    corecore