We consider the cumulative residual entropy (CRE) a recently introduced measure of entropy. While in previous works distributions with positive support are considered, we generalize the definition of CRE to the case of distributions with general support. We show that several interesting properties of the earlier CRE remain valid and supply further properties and insight to problems such as maximum CRE power moment problems. In addition, we show that this generalized CRE can be used as an alternative to differential entropy to derive information-based optimization criteria for system identification purpose

Jean Marc Boucher

Noomane Drissi

Thierry Chonavel

CiteSeerX

Hindawi Publishing Corporation
Research Letters in Signal Processing
Volume 2008, Article ID 790607, 5 pages
doi:10.1155/2008/790607
Research Letter
Generalized Cumulative Residual Entropy for
Distributions with Unrestricted Supports
Noomane Drissi, Thierry Chonavel, and Jean Marc Boucher
Lab-STICC (CNRS FRE 3167), Institut Telecom, Telecom Bretagne, Technopole Brest Iroise, CS 83818, 29238 Brest Cédex, France
Correspondence should be addressed to Noomane Drissi, noomane.drissi@telecom-bretagne.eu
Received 6 April 2008; Accepted 19 June 2008
Recommended by Andreas Jakobsson
We consider the cumulative residual entropy (CRE) a recently introduced measure of entropy. While in previous works
distributions with positive support are considered, we generalize the definition of CRE to the case of distributions with general
support. We show that several interesting properties of the earlier CRE remain valid and supply further properties and insight to
problems such as maximum CRE power moment problems. In addition, we show that this generalized CRE can be used as an
alternative to differential entropy to derive information-based optimization criteria for system identification purpose.
Copyright © 2008 Noomane Drissi et al. This is an open access article distributed under the Creative Commons Attribution
License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly
cited.
1. INTRODUCTION
The concept of entropy is important for studies in many
areas of engineering such as thermodynamics, mechanics,
or digital communications. An early definition of a measure
of the entropy is the Shannon entropy [1, 2]. In Shannon’s
approach, discrete values and absolutely continuous distri-
butions are treated in a somewhat different way through
entropy and differential entropy, respectively. Consider-
ing the complementary cumulative distribution function
(CCDF) instead of the probability density function in the
definition of differential entropy leads to a new entropy
measure named cumulative residual entropy (CRE) [3, 4]. In
[3, 4], CRE is defined as
E(X) = −
∫
Rn+
P
(|X| > u) logP(|X| > u)du, (1)
where n is the dimension of the random vector X . Clearly,
this formula is valid both for a discrete or an absolutely con-
tinuous random variable (RV), or with both a discrete and an
absolutely continuous part, because it resorts to the CCDF
of |X|. In addition, unlike Shannon differential entropy it is
always positive, while preserving many interesting properties
of Shannon entropy. The concept of CRE has found nice
interpretations and applications in the fields of reliability (see
[5] where the concept of dynamic CRE is introduced) and
images alignment [3].
Shannon entropy can be seen as a particular case
of exponential entropy, when entropy order tends to 1.
Thus, following the work in [4], a modified version of
the exponential entropy, where PDF is replaced by CCDF,
has been introduced in [6], leading to new entropy-type
measures, called survival entropies.
However, both Rao et al.’s CRE and its exponential
entropy generalization by Zografos and Nadarajah lead to
entropy-type definitions that assume either positive valued
RVs or apply to |X| otherwise. Although the positive case is
of great interest for many applications, CRE and exponential
entropies entail difficulties when working with RVs with
supports that are not restricted to positive values.
In this paper, we show that for an RV X , (1) remains
a valid expression when P(|X| > u) is replaced by P(X >
u) and integration is performed over Rn, without further
hypothesis than in [4]. In addition, some desirable properties
are enabled by this CRE definition extension. We also
complete the power moment constrained maximum CRE
distributions problem that was adressed in [7], for classes of
distributions that have lower-unbounded supports. Finally,
we illustrate the potential superiority of the proposed
generalized CRE (GCRE) against differential entropy in
mutual information-based estimation problems.
2 Research Letters in Signal Processing
The paper is organized as follows. Section 2 introduces
the GCRE definition. Some properties of GCRE are discussed
in Section 3. In Section 4, we introduce cumulative entropy
rate and mutual information rate. Section 5 deals with
maximum GCRE distributions. With a view to illustrate
the potentiality of GCRE, in Section 6, we show on a
simple example a possible benefit of GCRE for systems
identification.
2. GENERALIZED CUMULATIVE RESIDUAL
ENTROPY (GCRE)
We will denote by FcX(x) the complementary cumulative
distribution function (survival function) of a multivariate
RV X = [X1, . . . ,Xn]T of dimension n: FcX(X) = P(X > x) =
P(Xi > xi, i = 1, . . . ,n). We denote by HC(X) the GCRE of X
that we define by
HC(X) = −
∫
Rn
FcX(u) logF
c
X(u)du. (2)
Clearly, like the CRE, the GCRE is a positive and concave
function of FcX . In addition, existence of GCRE can be
established without further assumption upon distribution
than those assumed for the CRE in [4].
Theorem 1. HC(X) <∞ if for some p > n, E[|X|p] <∞.
Proof. First let us remark that from the proof of the existence
of CRE in [4], it is sufficient to prove the result when X is a
scalar RV, that is n = 1, and for p > 1. Then, letting p−1 <
α < 1, we use the following inequality:
−FcX(x) logFcX(x) ≤
e−1
1− α
[
FcX(x)
]α
1I]0,∞[(x)
+
(
1− FcX(x)
)
1I]−∞,0](x),
(3)
where 1IA(x) = 1 if x ∈ A and 1IA(x) = 0 otherwise.
The existence of
∫
R(e
−1/(1− α))[FcX(x)]α1I[0,∞[(x)dx can be
proven just as in [4]. Now, letting u = −x, we have
(
1− FcX(x)
)
1I]−∞,0](x)
= FX(x)1I]−∞,0](x)
≤ 1I[−1,0[(t) + FX(x)1I]−∞,−1[(x)
≤ 1I]0,1](u) + FX(−u)1I]1,∞](u)
≤ 1I]0,1](u) + F−X(u)1I]1,∞](u)
≤ 1I]0,1](u) + Fc|−X|(u)1I]1,∞](u)
≤ 1I]0,1](u) + u−pE
[|X|p]1I]1,∞](u).
(4)
Thus,∫
R
(
1− FcX(x)
)
1I]−∞,0](x)dx
≤
∫
R
(
1I]0,1](u) + u−pE
[|X|p]1I]1,∞](u))du
≤ 1 +
∫∞
1
u−pE
[|X|p]du
<∞.
(5)
Finally, putting all pieces together one finally proves conver-
gence of right-hand side of (2).
3. A FEW PROPERTIES OF GCRE
Let us now exhibit a few more interesting properties of the
GCRE. First, it is easy to check that like Shannon entropy the
GCRE remains constant with respect to variable translation:
∀a ∈ Rn, HC(X + a) = HC(X). (6)
In the same way, it is clear that
∀a ∈ R+, HC(aX) = aHC(X). (7)
When a < 0, we do not have such a nice property. However,
let us consider the important particular case where the
distribution of X has a symmetry of the form
∃μ, ∀x, FcX(μ− x) = 1− FcX(μ + x). (8)
In this case, we get the following result.
Theorem 2. For an RV X that satisfies symmetry property (8),
one has
∀a ∈ R, HC(aX) = |a|HC(X). (9)
Proof. Since it is clear that for all a ∈ R+,HC(aX) = aHC(X),
we just have to check that HC(−X) = HC(X), which can be
established as follows:
−HC(−X) =
∫
R
Fc−X(x) logF
c
−X(x)dx
=
∫
R
Fc−X(x − μ) logFc−X(x − μ)dx
=
∫
R
FX(−x + μ) logFX(−x + μ)dx
=
∫
R
(
1− FX(x + μ)
)
log
(
1− FX(x + μ)
)
dx
=
∫
R
FcX(x + μ) logF
c
X(x + μ)dx
=
∫
R
FcX(x) logF
c
X(x)dx
= −HC(X).
(10)
When the entries of vectorX are independent, it has been
shown in [4] that if the Xi are nonnegative, then
HC(X) =
∑
i
(
Π j /=iE
[
Xj
])
HC
(
Xi
)
. (11)
However, this formula does not extend to RVs with dis-
tributions carried by Rn because FC(X) can be integrated
over Rn+ in general but never over R
n. However, if the Xis
Noomane Drissi et al. 3
are independent and have lower bounded supports with
respective lower bounds m1, . . . ,mn,
HC(X) =
∫
Πi]mi,∞]
−FcX(x) logFcX(x)dx
=
∑
i
(
Π j /=i
(
E
[
Xj
]−mj))HC(Xi),
(12)
because
∫∞
mi
FcXi(u)du =
[
uFcXi(u)
]∞
mi
+
∫∞
mi
uPXi(du)
= −mi + E
[
Xi
]
.
(13)
Conditional GCRE definition is a direct extension of the
definition of conditional CRE: the conditional GCRE of X
knowing that Y is equal to y is defined by
HC(X|Y = y)
=
∫
Rn
−P(X > x|Y = y) logP(X > x|Y = y)dx.
(14)
We recall an important result from [4] that states that
conditioning reduces the entropy.
Theorem 3. For any X and Y ,
HC(X|Y) ≤ HC(X) (15)
equality holds if and only if X is independent of Y .
As a consequence, if X → Y → Z is a Markov chain, we
have the data processing inequality for GCRE:
HC(Z|X ,Y) ≤ HC(Z|X). (16)
4. ENTROPY AND MUTUAL INFORMATION RATES
4.1. Entropy rate
The GCRE of a stochastic process {Xi} is defined by
HC(X) = lim
n→∞HC
(
Xn
∣∣Xn−1,Xn−2, . . . ,X1) (17)
when the limit exists.
Theorem 4. For stationary processes, the limit exists.
Proof. Consider
HC
(
Xn+1
∣∣Xn, . . . ,X1) ≤ HC(Xn+1∣∣Xn, . . . ,X2)
≤ HC
(
Xn
∣∣Xn−1, . . . ,X1). (18)
The first line follows from the fact that conditioning reduces
entropy and the second follows from the stationarity (see [2]
for the equivalent proof in the case of Shannon entropy).
4.2. Mutual information
Let X and Y be two RVs. We define the cumulative mutual
information between X and Y as follows:
IC(X ;Y) = HC(X)−HC(X|Y). (19)
Theorem 5. IC is nonnegative and it vanishes if and only if X
and Y are independent.
Proof. It is clear that IC is nonnegative because of Theorem 3.
For a random vector X = (X1,X2, . . . ,Xn) of size n,
mutual information is defined by
IC(X) =
n∑
i=1
HC
(
Xi
)−HC(Xn∣∣Xn−1, . . . ,X1). (20)
In the case of stochastic processes {Xi}, we have HC(X) =
limn→∞HC(Xn|Xn−1, . . . ,X1) and the limit exists for station-
ary processes. Then the mutual information rate for {Xi} is
defined as
IC(X) = lim
T→∞
1
T
T∑
t=1
HC
(
Xt
)−HC(X), (21)
where HC(Xt) is the marginal GCRE of the process X .
5. MAXIMUM GCRE DISTRIBUTIONS
In this section, we only consider the case of one-dimensional
RVs (n = 1). Maximum entropy principle is useful in many
scientific areas and most important distributions can be
derived from it [8]. The maximum CRE distribution has
been studied in [7]. For an RV X with a symmetric CCDF
in the sense of (8), we are looking for the maximum GCRE
distribution, that is, the CCDF that solves the following
moment problem:
max
Fc
HC
(
Fc
)
∫
R
ri(x)p(x)dx = ci, i = 1, . . . ,m,
(22)
where p(x) = −(d/dx)Fc(x), (ri)i=1,m, and (ci)i=1,m are fixed
C1 real valued functions and real coefficients, respectively.
The solution of this problem is supplied by the following
result.
Theorem 6. When the symmetry property (8) holds, the
solution of problem (22), when it can be reached, is of the form
Fc(x) = 1
1 + exp
[∑m
1 λir
′
i (x − μ)
] (x ≥ μ). (23)
4 Research Letters in Signal Processing
Proof.
−HC(X) =
∫ μ
−∞
Fc(x) log
(
Fc(x)
)
dx
+
∫∞
μ
Fc(x) log
(
Fc(x)
)
dx
=
∫
R−
Fc(x + μ) log
(
Fc(x + μ)
)
dx
+
∫
R+
Fc(x + μ) log
(
Fc(x + μ)
)
dx
=
∫
R−
(
1− Fc(μ− x)) log (1− Fc(μ− x))dx
+
∫
R+
Fc(x + μ) log
(
Fc(x + μ)
)
dx
=
∫
R+
[(
1− Fc(μ + x)) log (1− Fc(μ + x))
+ Fc(μ + x) log
(
Fc(μ + x)
)]
dx.
(24)
Let us define f̃ by
f̃
(
Fc(x),−p)
= −(1− Fc(μ + x)) log (1− Fc(μ + x))
−Fc(μ + x) log (Fc(μ + x)) +
m∑
1
λi
(− p(x))ri(x).
(25)
Then, since (Fc(x))′ = −p(x), the Euler-Lagrange equation
[9] states that the solution Fc of problem (22) is a solution of
equation
d
dx
f̃−p(x) = f̃Fc(x), (26)
where f̃u is the partial derivative of f̃ with respect to
component u. From (25), we get
d
dx
f̃−p(x) =
m∑
1
λir
′
i (x),
f̃Fc(x) = log
(
1− Fc(μ + x))− logFc(μ + x).
(27)
Then,
log
1− Fc(μ + x)
Fc(μ + x)
=
m∑
1
λir
′
i (x),
Fc(x) = 1
1 + exp
(∑m
1 λir
′
i (x − μ)
) ,
(28)
for x ∈ [μ,∞[.
5.1. Example
We set the constraints E[X] = μ and E[X2] = σ2. Then the
maximum GCRE symmetric solution for the CCDF of X is
given by
FcX(x) =
1
1 + exp
(
λ1 + 2λ2x
) , (29)
for x > 0, which is the CCDF of a logistic distribution. The
moment constraints lead to λ1 + 2λ2x = (σ
√
3/π)(x−μ). The
corresponding PDF is defined on R by
pX(x) = σ
√
3
π
exp
(− (σ√3/π)(x − μ))(
1 + exp
(− (σ√3/π)(x − μ)))2 . (30)
5.2. Positive random variables
It has been shown in [7] that the maximum CRE (i.e., the
maximum GCRE under additional nonnegative constraint)
distribution has CCDF in the form
FC(x) = exp
(
−
m∑
i=1
λir
′
i (x)
)
, (31)
for x ∈ [0,∞[. In [7], this result is derived from the log-
sum inequality, but of course it can also be derived from the
Euler-Lagrange equation along the same lines as in the proof
of Theorem 6.
With a positive support constraint and under first and
second moment constraints, it comes that the optimum
CCDF is of the form Fc(x) = exp(−λ1 − 2λ2x), for x > 0.
Thus the solution, if it exists, is an exponential distribution.
In fact, the first and second power moment constraints must
be such that E[X2] = 2(E[X])2, otherwise the problem has
no exact solution.
6. SIMULATION RESULTS
With a view to emphasize the potential practical interest of
GCRE, we consider a simple system identification problem.
Here, we consider an MA(1) process, denoted by Y =
(Yn)n∈Z, generated by a white noise X = (Xn)n∈Z and
corrupted by a white noise N :
Yn = Xn − aXn−1 +Nn. (32)
The model input X and output Y are observed and the
system model (MA(1)) is assumed to be known. We want
to estimate the coefficient a without prior knowledge upon
the distributions of X and Y . Thus, we resort to mutual
information (MI) to estimate a as the coefficient α such
that RVs Ŷ αn = Xn − αXn−1 and Yn show the highest
dependence. Shannon MI between Ŷ αn and Yn is given by
fS(α) = IS(Ŷ αn ,Yn) = HS(Ŷ αn ) − HS(Ŷn|Yn), where HS is
Shannon differential entropy. Similarly, for GCRE, MI will
be defined as fC(α) = IC(Ŷ αn ,Y) = HC(Ŷ αn )−HC(Ŷ αn |Yn). We
compare estimation performance for a by maximizing both
fS(α) and fC(α). Since true values of fS(α) and fC(α) are not
available, they are estimated from empirical distributions of
(Yn, Ŷ αn ).
For simulations, we have chosen X Gaussian and N
with a Laplace distribution: pN (x) = (λ/2) exp(−λ|x|). We
consider an experiment with a = 0.5 and noise variance
equal to 0.2. Estimation is carried out from observation
of (Xn,Yn)n=1,400. Here, optimization of MIs is realized
on a fixed regular grid of 200 points over interval [0,1].
Estimation performance is calculated from 200 successive
Noomane Drissi et al. 5
1.1
1
0.9
0.8
IS
0 0.2 0.4 0.6 0.8 1
α
(a)
1.1
1
0.9
0.8
IC
0 0.2 0.4 0.6 0.8 1
α
(b)
Figure 1: Ten estimates of (a) fS(α) and (b) fC(α). Dotted line:
mean estimate averaged from 200 realizations.
experiments. Estimation of a from Shannon MI leads to
bias and standard deviation that are equal to 0.032 and
0.18, respectively, while they are equal to 0.004 and 0.06,
respectively, for GCRE MI.
More important, we see on Figure 1(a) that Shannon
MI estimates are much more irregular than GCRE MI
(Figure 1(b)) estimates because of smoothing brought by
density integration in the calculation of CCDF. This dif-
ference is important since the use of an iterative local
optimization technique would have failed in general to find
Shannon’s estimated MI global optimum, because of its
many local maxima.
Of course, this drawback can be partly solved by kernel
smoothing of the empirical distribution of (Yn, Ŷ αn ), for
instance by using the method proposed in [10]. However,
we have checked that, for the above example, very strong
smoothing is necessary and then bias and variance perfor-
mance remain worse than with GCRE MI estimator.
7. CONCLUSION
We have shown that the concept of cumulative residual
entropy (CRE) introduced in [3, 4] can be extended to dis-
tributions with general supports. Generalized CRE (GCRE)
shares many nice features of CRE. We also pointed out
specific properties of GCRE such as its maximum, moment
constrained, and distribution and we have illustrated practi-
cal interest of GCRE by showing how it can be used in system
identification procedures.
REFERENCES
[1] C. E. Shannon, “A mathematical theory of communication,”
Bell System Technical Journal, vol. 27, pp. 379–423, 1948.
[2] J. M. Cover and J. H. Thomas, Elements of Information Theory,
Wiley Interscience, New York, NY, USA, 2006.
[3] F. Wang, B. C. Vemuri, M. Rao, and Y. Chen, “A new & robust
information theoretic measure and its application to image
alignment,” in Proceedings of the 18th International Conference
on Information Processing in Medical Imaging (IPMI ’03),
vol. 2732 of Lecture Notes in Computer Science, pp. 388–400,
Springer, Ambleside, UK, July 2003.
[4] M. Rao, Y. Chen, B. C. Vemuri, and F. Wang, “Cumulative
residual entropy: a new measure of information,” IEEE
Transactions on Information Theory, vol. 50, no. 6, pp. 1220–
1228, 2004.
[5] M. Asadi and Y. Zohrevand, “On the dynamic cumulative
residual entropy,” Journal of Statistical Planning and Inference,
vol. 137, no. 6, pp. 1931–1941, 2007.
[6] K. Zografos and S. Nadarajah, “Survival exponential
entropies,” IEEE Transactions on Information Theory, vol. 51,
no. 3, pp. 1239–1246, 2005.
[7] M. Rao, “More on a new concept of entropy and information,”
Journal of Theoretical Probability, vol. 18, no. 4, pp. 967–981,
2005.
[8] J. N. Kapur, Maximum Entropy Methods in Science and
Engineering, John Wiley & Sons, New York, NY, USA, 1989.
[9] J. L. Troutman, Variational Calculus and Optimal Control,
Optimization with Elementary Convexity, Springer, New York,
NY, USA, 2nd edition, 1996.
[10] D. T. Pham, “Fast algorithm for estimating mutual informa-
tion, entropies and score functions,” in Proceedings of the 4th
International Symposium on Independent Component Analysis
and Blind Signal Separation (ICA ’03), pp. 17–22, Nara, Japan,
April 2003.
International Journal of
Aerospace
Engineering
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2010
Robotics
Journal of
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
 Active and Passive  
Electronic Components
Control Science
and Engineering
Journal of
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
 International Journal of
 Rotating
Machinery
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Hindawi Publishing Corporation 
http://www.hindawi.com
 Journal ofEngineering
Volume 2014
Submit your manuscripts at
http://www.hindawi.com
VLSI Design
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Shock and Vibration
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Civil Engineering
Advances in
Acoustics and Vibration
Advances in
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Electrical and Computer 
Engineering
Journal of
Advances in
OptoElectronics
Hindawi Publishing Corporation 
http://www.hindawi.com
Volume 2014
The Scientific 
World Journal
Hindawi Publishing Corporation 
http://www.hindawi.com Volume 2014
Sensors
Journal of
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Modelling & 
Simulation 
in Engineering
Hindawi Publishing Corporation 
http://www.hindawi.com Volume 2014
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Chemical Engineering
International Journal of  Antennas and
Propagation
International Journal of
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Navigation and 
 Observation
International Journal of
Hindawi Publishing Corporation
http://www.hindawi.com Volume 2014
Distributed
Sensor Networks
International Journal of


Research Letter Generalized Cumulative Residual Entropy for Distributions with Unrestricted Supports

http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=E12B3BE42B973A1CC96C1FC6DF9073E0?doi=10.1.1.1038.8959&rep=rep1&type=pdf

Research Letter Generalized Cumulative Residual Entropy for Distributions with Unrestricted Supports

Abstract

Similar works

Full text

Available Versions

CiteSeerX