A partial information decomposition for multivariate Gaussian systems based on information geometry

There is much interest in the topic of partial information decomposition, both in developing new algorithms and in developing applications. An algorithm, based on standard results from information geometry, was recently proposed by Niu and Quinn (2019). They considered the case of three scalar random variables from an exponential family, including both discrete distributions and a trivariate Gaussian distribution. The purpose of this article is to extend their work to the general case of multivariate Gaussian systems having vector inputs and a vector output. By making use of standard results from information geometry, explicit expressions are derived for the components of the partial information decomposition for this system. These expressions depend on a real-valued parameter which is determined by performing a simple constrained convex optimisation. Furthermore, it is proved that the theoretical properties of non-negativity, self-redundancy, symmetry and monotonicity, which were proposed by Williams and Beer (2010), are valid for the decomposition Iig derived herein. Application of these results to real and simulated data show that the Iig algorithm does produce the results expected when clear expectations are available, although in some scenarios, it can overestimate the level of the synergy and shared information components of the decomposition, and correspondingly underestimate the levels of unique information. Comparisons of the Iig and Idep (Kay and Ince, 2018) methods show that they can both produce very similar results, but interesting differences are provided. The same may be said about comparisons between the Iig and Immi (Barrett, 2015) methods

An Information Geometric Approach to Increase Representational Power in Unsupervised Learning

Machine learning models increase their representational power by increasing the number of parameters in the model. The number of parameters in the model can be increased by introducing hidden nodes, higher-order interaction effects or by introducing new features into the model. In this thesis we study different approaches to increase the representational power in unsupervised machine learning models. We investigate the use of incidence algebra and information geometry to develop novel machine learning models to include higher-order interactions effects into the model. Incidence algebra provides a natural formulation for combinatorics by expressing it as a generative function and information geometry provides many theoretical guarantees in the model by projecting the problem onto a dually flat Riemannian structure for optimization. Combining the two techniques together formulates the information geometric formulation of the binary log-linear model. We first use the information geometric formulation of the binary log-linear model to formulate the higher-order Boltzmann machine (HBM) to compare the different behaviours when using hidden nodes and higher-order feature interactions to increase the representational power of the model. We then apply the concepts learnt from this study to include higher-order interaction terms in Blind Source Separation (BSS) and to create an efficient approach to estimate higher order functions in Poisson process. Lastly, we explore the possibility to use Bayesian non-parametrics to automatically reduce the number of higher-order interactions effects included in the model