Riemannian Conjugate Gradient Methods: General Framework and Specific Algorithms with Convergence Analyses

Conjugate gradient methods are important first-order optimization algorithms both in Euclidean spaces and on Riemannian manifolds. However, while various types of conjugate gradient methods have been studied in Euclidean spaces, there are relatively fewer studies for those on Riemannian manifolds (i.e., Riemannian conjugate gradient methods). This paper proposes a novel general framework that unifies existing Riemannian conjugate gradient methods such as the ones that utilize a vector transport or inverse retraction. The proposed framework also develops other methods that have not been covered in previous studies. Furthermore, conditions for the convergence of a class of algorithms in the proposed framework are clarified. Moreover, the global convergence properties of several specific types of algorithms are extensively analyzed. The analysis provides the theoretical results for some algorithms in a more general setting than the existing studies and new developments for other algorithms. Numerical experiments are performed to confirm the validity of the theoretical results. The experimental results are used to compare the performances of several specific algorithms in the proposed framework

Luonnollinen gradientti variaatio-Bayes-oppimisessa

Todennäköisyysmalleilla on hyvin tärkeä asema koneoppimisessa, ja näiden mallien tehokas oppiminen on tärkeä ongelma. Valitettavasti näiden mallien matemaattinen käsittely suoraan on usein mahdotonta, ja mallien oppimisessa joudutaankin turvautumaan erilaisiin approksimaatioihin. Eräs tällainen approksimaatio on variaatiol3ayes-menetelmä, jossa todellista posteriorijakaumaa approksimoidaan toisella jakaumalla ja näiden kahden jakauman välistä eroa pyritään minimoimaan. Variaatio-Bayes-oppimisessa voidaan käyttää monia eri optimointialgoritmeja. Tässä työssä keskitytään gradienttipohjaisiin algoritmeihin. Näillä algoritmeilla on kuitenkin tyypillisesti yksi heikkous. Yleensä nämä menetelmät olettavat, että avaruus, jossa funktiota optimoidaan, on geometrialtaan euklidinen. Tilastollisissa malleissa tämä ei usein pidä paikkaansa, vaan avaruus on todellisuudessa Riemannin monisto. Luonnolliseen gradienttiin pohjautuvat optimointialgoritmit ottavat tämän geometrisen ominaisuuden huomioon ja ovat usein huomattavasti nopeampia kuin perinteiset optimointialgoritmit. Eräs tehokas ja suhteellisen yksinkertainen menetelmä saadaan yleistämällä konjugaattigradienttialgoritmi Riemannin monistoille. Näin saatua menetelmää kutsutaan Riemannin konjugaattigradientiksi. Tässä työssä esitellään tehokas Riemannin konjugaattigradienttialgoritmi variaatio-Bayes-menetelmää käyttävien tilastollisten mallien oppimiseen. Esimerkkiongelmana käytetään epälineaarisia tila-avaruusmalleja, joita käytetään sekä keinotekoisten että todellisten data-aineistojen oppimiseen. Näistä kokeista saadut tulokset osoittavat että esitelty algoritmi on huomattavasti tehokkaampi kuin muut vertailussa käytetyt perinteisemmät algoritmit.Probabilistic models play a very important role in machine learning, and the efficient learning of such models is a very important problem. Unfortunately, the exact statistical treatment of probabilistic models is often impossible and therefore various approximations have to be used. One such approximation is given by variational Bayesian (VB) learning which uses another distribution to approximate the true posterior distribution and tries to minimise the misfit between the two distributions. Many different optimisation algorithms can be used for variational Bayesian learning. This thesis concentrates on gradient based optimisation algorithms. Most of these algorithms suffer from one significant shortcoming, however. Typically these methods assume that the geometry of the problem space is flat, whereas in reality the space is a curved Riemannian manifold. Natural-gradient-based optimisation algorithms take this property into account, and can often result in significant speedups compared to traditional optimisation methods. One particularly powerful and relatively simple algorithm can be derived by extending conjugate gradient to Riemannian manifolds. The resulting algorithm is known as Riemannian conjugate gradient. This thesis presents an efficient Riemannian conjugate gradient algorithm for learning probabilistic models where variational approximation is used. Nonlinear state-space models are used as a case study, and results from experiments with both synthetic and real-world data sets are presented. The results demonstrate that the proposed algorithm provides significant performance gains over the other compared methods

Nonlinear eigenvalue problems

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Mathematics, 1998.Includes bibliographical references (p. 211-217).by Ross Adams Lippert.Ph.D

Deformable Medical Image Registration: A Survey

Deformable image registration is a fundamental task in medical image processing. Among its most important applications, one may cite: i) multi-modality fusion, where information acquired by different imaging devices or protocols is fused to facilitate diagnosis and treatment planning; ii) longitudinal studies, where temporal structural or anatomical changes are investigated; and iii) population modeling and statistical atlases used to study normal anatomical variability. In this technical report, we attempt to give an overview of deformable registration methods, putting emphasis on the most recent advances in the domain. Additional emphasis has been given to techniques applied to medical images. In order to study image registration methods in depth, their main components are identified and studied independently. The most recent techniques are presented in a systematic fashion. The contribution of this technical report is to provide an extensive account of registration techniques in a systematic manner.Le recalage déformable d'images est une des tâches les plus fondamentales dans l'imagerie médicale. Parmi ses applications les plus importantes, on compte: i) la fusion d' information provenant des différents types de modalités a n de faciliter le diagnostic et la planification du traitement; ii) les études longitudinales, oú des changements structurels ou anatomiques sont étudiées en fonction du temps; et iii) la modélisation de la variabilité anatomique normale d'une population et les atlas statistiques. Dans ce rapport de recherche, nous essayons de donner un aperçu des différentes méthodes du recalage déformables, en mettant l'accent sur les avancées les plus récentes du domaine. Nous avons particulièrement insisté sur les techniques appliquées aux images médicales. A n d'étudier les méthodes du recalage d'images, leurs composants principales sont d'abord identifiés puis étudiées de manière indépendante, les techniques les plus récentes étant classifiées en suivant un schéma logique déterminé. La contribution de ce rapport de recherche est de fournir un compte rendu détaillé des techniques de recalage d'une manière systématique

Stochastic processes on graphs with cycles : geometric and variational approaches

Thesis (Ph.D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2002.Includes bibliographical references (leaves 259-271).Stochastic processes defined on graphs arise in a tremendous variety of fields, including statistical physics, signal processing, computer vision, artificial intelligence, and information theory. The formalism of graphical models provides a useful language with which to formulate fundamental problems common to all of these fields, including estimation, model fitting, and sampling. For graphs without cycles, known as trees, all of these problems are relatively well-understood, and can be solved efficiently with algorithms whose complexity scales in a tractable manner with problem size. In contrast, these same problems present considerable challenges in general graphs with cycles. The focus of this thesis is the development and analysis of methods, both exact and approximate, for problems on graphs with cycles. Our contributions are in developing and analyzing techniques for estimation, as well as methods for computing upper and lower bounds on quantities of interest (e.g., marginal probabilities; partition functions). In order to do so, we make use of exponential representations of distributions, as well as insight from the associated information geometry and Legendre duality. Our results demonstrate the power of exponential representations for graphical models, as well as the utility of studying collections of modified problems defined on trees embedded within the original graph with cycles. The specific contributions of this thesis include the following. We develop a method for performing exact estimation of Gaussian processes on graphs with cycles by solving a sequence of modified problems on embedded spanning trees.(cont.) We introduce the tree-based reparameterization framework for approximate estimation of discrete processes. This perspective leads to a number of theoretical results on belief propagation and related algorithms, including characterizations of their fixed points and the associated approximation error. Next we extend the notion of reparameterization to a much broader class of methods for approximate inference, including Kikuchi methods, and present results on their fixed points and accuracy. Finally, we develop and analyze a novel class of upper bounds on the log partition function based on convex combinations of distributions in the exponential domain. In the special case of combining tree-structured distributions, the associated dual function gives an interesting perspective on the Bethe free energy.by Martn J. Wainwright.Ph.D

Generalized averaged Gaussian quadrature and applications

A simple numerical method for constructing the optimal generalized averaged Gaussian quadrature formulas will be presented. These formulas exist in many cases in which real positive GaussKronrod formulas do not exist, and can be used as an adequate alternative in order to estimate the error of a Gaussian rule. We also investigate the conditions under which the optimal averaged Gaussian quadrature formulas and their truncated variants are internal

MS FT-2-2 7 Orthogonal polynomials and quadrature: Theory, computation, and applications

Quadrature rules find many applications in science and engineering. Their analysis is a classical area of applied mathematics and continues to attract considerable attention. This seminar brings together speakers with expertise in a large variety of quadrature rules. It is the aim of the seminar to provide an overview of recent developments in the analysis of quadrature rules. The computation of error estimates and novel applications also are described

Theoretical Concepts of Quantum Mechanics

Quantum theory as a scientific revolution profoundly influenced human thought about the universe and governed forces of nature. Perhaps the historical development of quantum mechanics mimics the history of human scientific struggles from their beginning. This book, which brought together an international community of invited authors, represents a rich account of foundation, scientific history of quantum mechanics, relativistic quantum mechanics and field theory, and different methods to solve the Schrodinger equation. We wish for this collected volume to become an important reference for students and researchers