32,039 research outputs found
An Extended Result on the Optimal Estimation under Minimum Error Entropy Criterion
The minimum error entropy (MEE) criterion has been successfully used in
fields such as parameter estimation, system identification and the supervised
machine learning. There is in general no explicit expression for the optimal
MEE estimate unless some constraints on the conditional distribution are
imposed. A recent paper has proved that if the conditional density is
conditionally symmetric and unimodal (CSUM), then the optimal MEE estimate
(with Shannon entropy) equals the conditional median. In this study, we extend
this result to the generalized MEE estimation where the optimality criterion is
the Renyi entropy or equivalently, the \alpha-order information potential (IP).Comment: 15 pages, no figures, submitted to Entrop
An extended orthogonal forward regression algorithm for system identification using entropy
In this paper, a fast identification algorithm for nonlinear dynamic stochastic system identification is presented. The algorithm extends the classical Orthogonal Forward Regression (OFR) algorithm so that instead of using the Error Reduction Ratio (ERR) for term selection, a new optimality criterion —Shannon’s Entropy Power Reduction Ratio(EPRR) is introduced to deal with both Gaussian and non-Gaussian signals. It is shown that the new algorithm is both fast and reliable and examples are provided to illustrate the effectiveness of the new approach
Finite-Block-Length Analysis in Classical and Quantum Information Theory
Coding technology is used in several information processing tasks. In
particular, when noise during transmission disturbs communications, coding
technology is employed to protect the information. However, there are two types
of coding technology: coding in classical information theory and coding in
quantum information theory. Although the physical media used to transmit
information ultimately obey quantum mechanics, we need to choose the type of
coding depending on the kind of information device, classical or quantum, that
is being used. In both branches of information theory, there are many elegant
theoretical results under the ideal assumption that an infinitely large system
is available. In a realistic situation, we need to account for finite size
effects. The present paper reviews finite size effects in classical and quantum
information theory with respect to various topics, including applied aspects
Minimum Rates of Approximate Sufficient Statistics
Given a sufficient statistic for a parametric family of distributions, one
can estimate the parameter without access to the data. However, the memory or
code size for storing the sufficient statistic may nonetheless still be
prohibitive. Indeed, for independent samples drawn from a -nomial
distribution with degrees of freedom, the length of the code scales as
. In many applications, we may not have a useful notion of
sufficient statistics (e.g., when the parametric family is not an exponential
family) and we also may not need to reconstruct the generating distribution
exactly. By adopting a Shannon-theoretic approach in which we allow a small
error in estimating the generating distribution, we construct various {\em
approximate sufficient statistics} and show that the code length can be reduced
to . We consider errors measured according to the
relative entropy and variational distance criteria. For the code constructions,
we leverage Rissanen's minimum description length principle, which yields a
non-vanishing error measured according to the relative entropy. For the
converse parts, we use Clarke and Barron's formula for the relative entropy of
a parametrized distribution and the corresponding mixture distribution.
However, this method only yields a weak converse for the variational distance.
We develop new techniques to achieve vanishing errors and we also prove strong
converses. The latter means that even if the code is allowed to have a
non-vanishing error, its length must still be at least .Comment: To appear in the IEEE Transactions on Information Theor
- …