Linear Coupling: An Ultimate Unification of Gradient and Mirror Descent

First-order methods play a central role in large-scale machine learning. Even though many variations exist, each suited to a particular problem, almost all such methods fundamentally rely on two types of algorithmic steps: gradient descent, which yields primal progress, and mirror descent, which yields dual progress. We observe that the performances of gradient and mirror descent are complementary, so that faster algorithms can be designed by LINEARLY COUPLING the two. We show how to reconstruct Nesterov's accelerated gradient methods using linear coupling, which gives a cleaner interpretation than Nesterov's original proofs. We also discuss the power of linear coupling by extending it to many other settings that Nesterov's methods cannot apply to.Comment: A new section added; polished writin

Primal-dual variable neighborhood search for the simple plant-location problem

Copyright @ 2007 INFORMSThe variable neighborhood search metaheuristic is applied to the primal simple plant-location problem and to a reduced dual obtained by exploiting the complementary slackness conditions. This leads to (i) heuristic resolution of (metric) instances with uniform fixed costs, up to n = 15,000 users, and m = n potential locations for facilities with an error not exceeding 0.04%; (ii) exact solution of such instances with up to m = n = 7,000; and (iii) exact solutions of instances with variable fixed costs and up to m = n = 15, 000.This work is supported by NSERC Grant 105574-02; NSERC Grant OGP205041; and partly by the Serbian Ministry of Science, Project 1583

Jeffreys-prior penalty, finiteness and shrinkage in binomial-response generalized linear models

Penalization of the likelihood by Jeffreys' invariant prior, or by a positive power thereof, is shown to produce finite-valued maximum penalized likelihood estimates in a broad class of binomial generalized linear models. The class of models includes logistic regression, where the Jeffreys-prior penalty is known additionally to reduce the asymptotic bias of the maximum likelihood estimator; and also models with other commonly used link functions such as probit and log-log. Shrinkage towards equiprobability across observations, relative to the maximum likelihood estimator, is established theoretically and is studied through illustrative examples. Some implications of finiteness and shrinkage for inference are discussed, particularly when inference is based on Wald-type procedures. A widely applicable procedure is developed for computation of maximum penalized likelihood estimates, by using repeated maximum likelihood fits with iteratively adjusted binomial responses and totals. These theoretical results and methods underpin the increasingly widespread use of reduced-bias and similarly penalized binomial regression models in many applied fields

Thermodynamic fluctuation relation for temperature and energy

The present work extends the well-known thermodynamic relation $C=\beta ^{2}< \delta {E^{2}}>$ for the canonical ensemble. We start from the general situation of the thermodynamic equilibrium between a large but finite system of interest and a generalized thermostat, which we define in the course of the paper. The resulting identity $=1+< \delta {E^{2}}> \partial ^{2}S(E) /\partial {E^{2}}$ can account for thermodynamic states with a negative heat capacity $C<0$; at the same time, it represents a thermodynamic fluctuation relation that imposes some restrictions on the determination of the microcanonical caloric curve $\beta (E) =\partial S(E) /\partial E$. Finally, we comment briefly on the implications of the present result for the development of new Monte Carlo methods and an apparent analogy with quantum mechanics.Comment: Version accepted for publication in J. Phys. A: Math and The

Lattice structures for optimal design and robust implementation of two-channel perfect-reconstruction QMF banks

A lattice structure and an algorithm are presented for the design of two-channel QMF (quadrature mirror filter) banks, satisfying a sufficient condition for perfect reconstruction. The structure inherently has the perfect-reconstruction property, while the algorithm ensures a good stopband attenuation for each of the analysis filters. Implementations of such lattice structures are robust in the sense that the perfect-reconstruction property is preserved in spite of coefficient quantization. The lattice structure has the hierarchical property that a higher order perfect-reconstruction QMF bank can be obtained from a lower order perfect-reconstruction QMF bank, simply by adding more lattice sections. Several numerical examples are provided in the form of design tables