354 research outputs found
Determining Principal Component Cardinality through the Principle of Minimum Description Length
PCA (Principal Component Analysis) and its variants areubiquitous techniques
for matrix dimension reduction and reduced-dimensionlatent-factor extraction.
One significant challenge in using PCA, is thechoice of the number of principal
components. The information-theoreticMDL (Minimum Description Length) principle
gives objective compression-based criteria for model selection, but it is
difficult to analytically applyits modern definition - NML (Normalized Maximum
Likelihood) - to theproblem of PCA. This work shows a general reduction of NML
prob-lems to lower-dimension problems. Applying this reduction, it boundsthe
NML of PCA, by terms of the NML of linear regression, which areknown.Comment: LOD 201
Minimum Energy Information Fusion in Sensor Networks
In this paper we consider how to organize the sharing of information in a
distributed network of sensors and data processors so as to provide
explanations for sensor readings with minimal expenditure of energy. We point
out that the Minimum Description Length principle provides an approach to
information fusion that is more naturally suited to energy minimization than
traditional Bayesian approaches. In addition we show that for networks
consisting of a large number of identical sensors Kohonen self-organization
provides an exact solution to the problem of combining the sensor outputs into
minimal description length explanations.Comment: postscript, 8 pages. Paper 65 in Proceedings of The 2nd International
Conference on Information Fusio
Structure or Noise?
We show how rate-distortion theory provides a mechanism for automated theory
building by naturally distinguishing between regularity and randomness. We
start from the simple principle that model variables should, as much as
possible, render the future and past conditionally independent. From this, we
construct an objective function for model making whose extrema embody the
trade-off between a model's structural complexity and its predictive power. The
solutions correspond to a hierarchy of models that, at each level of
complexity, achieve optimal predictive power at minimal cost. In the limit of
maximal prediction the resulting optimal model identifies a process's intrinsic
organization by extracting the underlying causal states. In this limit, the
model's complexity is given by the statistical complexity, which is known to be
minimal for achieving maximum prediction. Examples show how theory building can
profit from analyzing a process's causal compressibility, which is reflected in
the optimal models' rate-distortion curve--the process's characteristic for
optimally balancing structure and noise at different levels of representation.Comment: 6 pages, 2 figures;
http://cse.ucdavis.edu/~cmg/compmech/pubs/son.htm
- …