9,508 research outputs found
Relative Information Loss in the PCA
In this work we analyze principle component analysis (PCA) as a deterministic
input-output system. We show that the relative information loss induced by
reducing the dimensionality of the data after performing the PCA is the same as
in dimensionality reduction without PCA. Finally, we analyze the case where the
PCA uses the sample covariance matrix to compute the rotation. If the rotation
matrix is not available at the output, we show that an infinite amount of
information is lost. The relative information loss is shown to decrease with
increasing sample size.Comment: 9 pages, 4 figure; extended version of a paper accepted for
publicatio
Information Loss and Anti-Aliasing Filters in Multirate Systems
This work investigates the information loss in a decimation system, i.e., in
a downsampler preceded by an anti-aliasing filter. It is shown that, without a
specific signal model in mind, the anti-aliasing filter cannot reduce
information loss, while, e.g., for a simple signal-plus-noise model it can. For
the Gaussian case, the optimal anti-aliasing filter is shown to coincide with
the one obtained from energetic considerations. For a non-Gaussian signal
corrupted by Gaussian noise, the Gaussian assumption yields an upper bound on
the information loss, justifying filter design principles based on second-order
statistics from an information-theoretic point-of-view.Comment: 12 pages; a shorter version of this paper was published at the 2014
International Zurich Seminar on Communication
Information-Preserving Markov Aggregation
We present a sufficient condition for a non-injective function of a Markov
chain to be a second-order Markov chain with the same entropy rate as the
original chain. This permits an information-preserving state space reduction by
merging states or, equivalently, lossless compression of a Markov source on a
sample-by-sample basis. The cardinality of the reduced state space is bounded
from below by the node degrees of the transition graph associated with the
original Markov chain.
We also present an algorithm listing all possible information-preserving
state space reductions, for a given transition graph. We illustrate our results
by applying the algorithm to a bi-gram letter model of an English text.Comment: 7 pages, 3 figures, 2 table
- …