35,928 research outputs found
Mixing and non-mixing local minima of the entropy contrast for blind source separation
In this paper, both non-mixing and mixing local minima of the entropy are
analyzed from the viewpoint of blind source separation (BSS); they correspond
respectively to acceptable and spurious solutions of the BSS problem. The
contribution of this work is twofold. First, a Taylor development is used to
show that the \textit{exact} output entropy cost function has a non-mixing
minimum when this output is proportional to \textit{any} of the non-Gaussian
sources, and not only when the output is proportional to the lowest entropic
source. Second, in order to prove that mixing entropy minima exist when the
source densities are strongly multimodal, an entropy approximator is proposed.
The latter has the major advantage that an error bound can be provided. Even if
this approximator (and the associated bound) is used here in the BSS context,
it can be applied for estimating the entropy of any random variable with
multimodal density.Comment: 11 pages, 6 figures, To appear in IEEE Transactions on Information
Theor
The Hitchhiker's Guide to Nonlinear Filtering
Nonlinear filtering is the problem of online estimation of a dynamic hidden
variable from incoming data and has vast applications in different fields,
ranging from engineering, machine learning, economic science and natural
sciences. We start our review of the theory on nonlinear filtering from the
simplest `filtering' task we can think of, namely static Bayesian inference.
From there we continue our journey through discrete-time models, which is
usually encountered in machine learning, and generalize to and further
emphasize continuous-time filtering theory. The idea of changing the
probability measure connects and elucidates several aspects of the theory, such
as the parallels between the discrete- and continuous-time problems and between
different observation models. Furthermore, it gives insight into the
construction of particle filtering algorithms. This tutorial is targeted at
scientists and engineers and should serve as an introduction to the main ideas
of nonlinear filtering, and as a segway to more advanced and specialized
literature.Comment: 64 page
Design and Analysis of Nonbinary LDPC Codes for Arbitrary Discrete-Memoryless Channels
We present an analysis, under iterative decoding, of coset LDPC codes over
GF(q), designed for use over arbitrary discrete-memoryless channels
(particularly nonbinary and asymmetric channels). We use a random-coset
analysis to produce an effect that is similar to output-symmetry with binary
channels. We show that the random selection of the nonzero elements of the
GF(q) parity-check matrix induces a permutation-invariance property on the
densities of the decoder messages, which simplifies their analysis and
approximation. We generalize several properties, including symmetry and
stability from the analysis of binary LDPC codes. We show that under a Gaussian
approximation, the entire q-1 dimensional distribution of the vector messages
is described by a single scalar parameter (like the distributions of binary
LDPC messages). We apply this property to develop EXIT charts for our codes. We
use appropriately designed signal constellations to obtain substantial shaping
gains. Simulation results indicate that our codes outperform multilevel codes
at short block lengths. We also present simulation results for the AWGN
channel, including results within 0.56 dB of the unconstrained Shannon limit
(i.e. not restricted to any signal constellation) at a spectral efficiency of 6
bits/s/Hz.Comment: To appear, IEEE Transactions on Information Theory, (submitted
October 2004, revised and accepted for publication, November 2005). The
material in this paper was presented in part at the 41st Allerton Conference
on Communications, Control and Computing, October 2003 and at the 2005 IEEE
International Symposium on Information Theor
Discretizing Distributions with Exact Moments: Error Estimate and Convergence Analysis
The maximum entropy principle is a powerful tool for solving underdetermined
inverse problems. This paper considers the problem of discretizing a continuous
distribution, which arises in various applied fields. We obtain the
approximating distribution by minimizing the Kullback-Leibler information
(relative entropy) of the unknown discrete distribution relative to an initial
discretization based on a quadrature formula subject to some moment
constraints. We study the theoretical error bound and the convergence of this
approximation method as the number of discrete points increases. We prove that
(i) the theoretical error bound of the approximate expectation of any bounded
continuous function has at most the same order as the quadrature formula we
start with, and (ii) the approximate discrete distribution weakly converges to
the given continuous distribution. Moreover, we present some numerical examples
that show the advantage of the method and apply to numerically solving an
optimal portfolio problem.Comment: 20 pages, 14 figure
Probabilistic Inference from Arbitrary Uncertainty using Mixtures of Factorized Generalized Gaussians
This paper presents a general and efficient framework for probabilistic
inference and learning from arbitrary uncertain information. It exploits the
calculation properties of finite mixture models, conjugate families and
factorization. Both the joint probability density of the variables and the
likelihood function of the (objective or subjective) observation are
approximated by a special mixture model, in such a way that any desired
conditional distribution can be directly obtained without numerical
integration. We have developed an extended version of the expectation
maximization (EM) algorithm to estimate the parameters of mixture models from
uncertain training examples (indirect observations). As a consequence, any
piece of exact or uncertain information about both input and output values is
consistently handled in the inference and learning stages. This ability,
extremely useful in certain situations, is not found in most alternative
methods. The proposed framework is formally justified from standard
probabilistic principles and illustrative examples are provided in the fields
of nonparametric pattern classification, nonlinear regression and pattern
completion. Finally, experiments on a real application and comparative results
over standard databases provide empirical evidence of the utility of the method
in a wide range of applications
- …