3 research outputs found
On the Distribution of the Information Density of Gaussian Random Vectors: Explicit Formulas and Tight Approximations
Based on the canonical correlation analysis we derive series representations
of the probability density function (PDF) and the cumulative distribution
function (CDF) of the information density of arbitrary Gaussian random vectors
as well as a general formula to calculate the central moments. Using the
general results we give closed-form expressions of the PDF and CDF and explicit
formulas of the central moments for important special cases. Furthermore, we
derive recurrence formulas and tight approximations of the general series
representations, which allow very efficient numerical calculations with an
arbitrarily high accuracy as demonstrated with an implementation in Python
publicly available on GitLab. Finally, we discuss the (in)validity of Gaussian
approximations of the information density.Comment: This extended version of the manuscript replaces the previous
versions and is submitted to the journal "Problems of Information
Transmission". An implementation in Python allowing efficient numerical
calculations related to the main results of the paper is publicly available
on GitLab: https://gitlab.com/infth/information-densit
Canonical Correlation and the Calculation of Information Measures for Infinite-Dimensional Distributions: Kanonische Korrelationen und die Berechnung von InformationsmaĂźen fĂĽr unendlichdimensionale Verteilungen
This thesis investigates the extension of the well-known canonical correlation analysis for random elements on abstract real measurable Hilbert spaces. One focus is on the application of this extension to the calculation of information-theoretical quantities on finite time intervals. Analytical approaches for the calculation of the mutual information and the information density between Gaussian distributed random elements on arbitrary real measurable Hilbert spaces are derived.
With respect to mutual information, the results obtained are comparable to [4] and [1] (Baker, 1970, 1978). They can also be seen as a generalization of earlier findings in [20] (Gelfand and Yaglom, 1958). In addition, some of the derived equations for calculating the information density, its characteristic function and its n-th central moments extend results from [45] and [44] (Pinsker, 1963, 1964).
Furthermore, explicit examples for the calculation of the mutual information, the characteristic function of the information density as well as the n-th central moments of the information density for the important special case of an additive Gaussian channel with Gaussian distributed input signal with rational spectral density are elaborated, on the one hand for white Gaussian noise and on the other hand for Gaussian noise with rational spectral density. These results extend the corresponding concrete examples for the calculation of the mutual information from [20] (Gelfand and Yaglom, 1958) as well as [28] and [29] (Huang and Johnson, 1963, 1962).:Kurzfassung
Abstract
Notations
Abbreviations
1 Introduction
1.1 Software Used
2 Mathematical Background
2.1 Basic Notions of Measure and Probability Theory
2.1.1 Characteristic Functions
2.2 Stochastic Processes
2.2.1 The Consistency Theorem of Daniell and Kolmogorov
2.2.2 Second Order Random Processes
2.3 Some Properties of Fourier Transforms
2.4 Some Basic Inequalities
2.5 Some Fundamentals in Functional Analysis
2.5.1 Hilbert Spaces
2.5.2 Linear Operators on Hilbert Spaces
2.5.3 The Fréchet-Riesz Representation Theorem
2.5.4 Adjoint and Compact Operators
2.5.5 The Spectral Theorem for Compact Operators
3 Mutual Information and Information Density
3.1 Mutual Information
3.2 Information Density
4 Probability Measures on Hilbert Spaces
4.1 Measurable Hilbert Spaces
4.2 The Characteristic Functional
4.3 Mean Value and Covariance Operator
4.4 Gaussian Probability Measures on Hilbert Spaces
4.5 The Product of Two Measurable Hilbert Spaces
4.5.1 The Product Measure
4.5.2 Cross-Covariance Operator
5 Canonical Correlation Analysis on Hilbert Spaces
5.1 The Hellinger Distance and the Theorem of Kakutani
5.2 Canonical Correlation Analysis on Hilbert Spaces
5.3 The Theorem of Hájek and Feldman
6 Mutual Information and Information Density Between Gaussian Measures
6.1 A General Formula for Mutual Information and Information Density for Gaussian Random Elements
6.2 Hadamard’s Factorization Theorem
6.3 Closed Form Expressions for Mutual Information and Related Quantities
6.4 The Discrete-Time Case
6.5 The Continuous-Time Case
6.6 Approximation Error
7 Additive Gaussian Channels
7.1 Abstract Channel Model and General Definitions
7.2 Explicit Expressions for Mutual Information and Related Quantities
7.2.1 Gaussian Random Elements as Input to an Additive Gaussian Channel
8 Continuous-Time Gaussian Channels
8.1 White Gaussian Channels
8.1.1 Two Simple Examples
8.1.2 Gaussian Input with Rational Spectral Density
8.1.3 A Method of Youla, Kadota and Slepian
8.2 Noise and Input Signal with Rational Spectral Density
8.2.1 Again a Method by Slepian and Kadota
BibliographyDiese Arbeit untersucht die Erweiterung der bekannten kanonischen Korrelationsanalyse (canonical correlation analysis) für Zufallselemente auf abstrakten reellen messbaren Hilberträumen. Ein Schwerpunkt liegt dabei auf der Anwendung dieser Erweiterung zur Berechnung informationstheoretischer Größen auf endlichen Zeitintervallen. Analytische Ansätze für die Berechnung der Transinformation und der Informationsdichte zwischen gaußverteilten Zufallselementen auf beliebigen reelen messbaren Hilberträumen werden hergeleitet.
Bezüglich der Transinformation sind die gewonnenen Resultate vergleichbar zu [4] und [1] (Baker, 1970, 1978). Sie können auch als Verallgemeinerung früherer Erkenntnisse aus [20] (Gelfand und Yaglom, 1958) aufgefasst werden. Zusätzlich erweitern einige der hergeleiteten Formeln zur Berechnung der Informationsdichte, ihrer charakteristischen Funktion und ihrer n-ten zentralen Momente Ergebnisse aus [45] und [44] (Pinsker, 1963, 1964).
Weiterhin werden explizite Beispiele fĂĽr die Berechnung der Transinformation, der charakteristischen Funktion der Informationsdichte sowie der n-ten zentralen Momente der Informationsdichte fĂĽr den wichtigen Spezialfall eines additiven GauĂźkanals mit gauĂźverteiltem Eingangssignal mit rationaler Spektraldichte erarbeitet, einerseits fĂĽr gauĂźsches weiĂźes Rauschen und andererseits fĂĽr gauĂźsches Rauschen mit einer rationalen Spektraldichte. Diese Ergebnisse erweitern die entsprechenden konkreten Beispiele zur Berechnung der Transinformation aus [20] (Gelfand und Yaglom, 1958) sowie [28] und [29] (Huang und Johnson, 1963, 1962).:Kurzfassung
Abstract
Notations
Abbreviations
1 Introduction
1.1 Software Used
2 Mathematical Background
2.1 Basic Notions of Measure and Probability Theory
2.1.1 Characteristic Functions
2.2 Stochastic Processes
2.2.1 The Consistency Theorem of Daniell and Kolmogorov
2.2.2 Second Order Random Processes
2.3 Some Properties of Fourier Transforms
2.4 Some Basic Inequalities
2.5 Some Fundamentals in Functional Analysis
2.5.1 Hilbert Spaces
2.5.2 Linear Operators on Hilbert Spaces
2.5.3 The Fréchet-Riesz Representation Theorem
2.5.4 Adjoint and Compact Operators
2.5.5 The Spectral Theorem for Compact Operators
3 Mutual Information and Information Density
3.1 Mutual Information
3.2 Information Density
4 Probability Measures on Hilbert Spaces
4.1 Measurable Hilbert Spaces
4.2 The Characteristic Functional
4.3 Mean Value and Covariance Operator
4.4 Gaussian Probability Measures on Hilbert Spaces
4.5 The Product of Two Measurable Hilbert Spaces
4.5.1 The Product Measure
4.5.2 Cross-Covariance Operator
5 Canonical Correlation Analysis on Hilbert Spaces
5.1 The Hellinger Distance and the Theorem of Kakutani
5.2 Canonical Correlation Analysis on Hilbert Spaces
5.3 The Theorem of Hájek and Feldman
6 Mutual Information and Information Density Between Gaussian Measures
6.1 A General Formula for Mutual Information and Information Density for Gaussian Random Elements
6.2 Hadamard’s Factorization Theorem
6.3 Closed Form Expressions for Mutual Information and Related Quantities
6.4 The Discrete-Time Case
6.5 The Continuous-Time Case
6.6 Approximation Error
7 Additive Gaussian Channels
7.1 Abstract Channel Model and General Definitions
7.2 Explicit Expressions for Mutual Information and Related Quantities
7.2.1 Gaussian Random Elements as Input to an Additive Gaussian Channel
8 Continuous-Time Gaussian Channels
8.1 White Gaussian Channels
8.1.1 Two Simple Examples
8.1.2 Gaussian Input with Rational Spectral Density
8.1.3 A Method of Youla, Kadota and Slepian
8.2 Noise and Input Signal with Rational Spectral Density
8.2.1 Again a Method by Slepian and Kadota
Bibliograph
On the Distribution of the Information Density of Gaussian Random Vectors: Explicit Formulas and Tight Approximations
Based on the canonical correlation analysis, we derive series representations of the probability density function (PDF) and the cumulative distribution function (CDF) of the information density of arbitrary Gaussian random vectors as well as a general formula to calculate the central moments. Using the general results, we give closed-form expressions of the PDF and CDF and explicit formulas of the central moments for important special cases. Furthermore, we derive recurrence formulas and tight approximations of the general series representations, which allow efficient numerical calculations with an arbitrarily high accuracy as demonstrated with an implementation in Python publicly available on GitLab. Finally, we discuss the (in)validity of Gaussian approximations of the information density