The principle of maximizing mutual information is applied to learning overcomplete and recurrent representations. The underlying model consists of a network of input units driving a larger number of output units with recurrent interactions. In the limit of zero noise, the network is deterministic and the mutual information can be related to the entropy of the output units. Maximizing this entropy with respect to both the feedforward connections as well as the recurrent interactions results in simple learning rules for both sets of parameters. The conventional independent components (ICA) learning algorithm can be recovered as a special case where there is an equal number of output units and no recurrent connections. The application of these new learning rules is illustrated on a simple two-dimensional input example

Lee, Daniel D

Shriki, Oren

Sompolinsky, Haim

English

ScholarlyCommons@Penn

University of PennsylvaniaScholarlyCommonsDepartmental Papers (ESE) Department of Electrical & Systems EngineeringNovember 2000An Information Maximization Approach toOvercomplete and Recurrent RepresentationsOren ShrikiHebrew UniversityHaim SompolinskyHebrew UniversityDaniel D. LeeUniversity of Pennsylvania, ddlee@seas.upenn.eduFollow this and additional works at: http://repository.upenn.edu/ese_papersReprinted from 12th Conference on Neural Information Processing Systems (NIPS 2000), November 2000, pages 87-93.NOTE: At the time of publication, author Daniel D. Lee was affiliated with Bell Laboratories. Currently (March 2005), he is a faculty member in theDepartment of Electrical and Systems Engineering at the University of Pennsylvania.This paper is posted at ScholarlyCommons. http://repository.upenn.edu/ese_papers/85For more information, please contact repository@pobox.upenn.edu.Recommended CitationOren Shriki, Haim Sompolinsky, and Daniel D. Lee, "An Information Maximization Approach to Overcomplete and RecurrentRepresentations", . November 2000.An Information Maximization Approach to Overcomplete and RecurrentRepresentationsAbstractThe principle of maximizing mutual information is applied to learning overcomplete and recurrentrepresentations. The underlying model consists of a network of input units driving a larger number of outputunits with recurrent interactions. In the limit of zero noise, the network is deterministic and the mutualinformation can be related to the entropy of the output units. Maximizing this entropy with respect to boththe feedforward connections as well as the recurrent interactions results in simple learning rules for both setsof parameters. The conventional independent components (ICA) learning algorithm can be recovered as aspecial case where there is an equal number of output units and no recurrent connections. The application ofthese new learning rules is illustrated on a simple two-dimensional input example.CommentsReprinted from 12th Conference on Neural Information Processing Systems (NIPS 2000), November 2000, pages87-93.NOTE: At the time of publication, author Daniel D. Lee was affiliated with Bell Laboratories. Currently(March 2005), he is a faculty member in the Department of Electrical and Systems Engineering at theUniversity of Pennsylvania.This conference paper is available at ScholarlyCommons: http://repository.upenn.edu/ese_papers/85An Information Maximization Approach toOvercomplete and Recurrent RepresentationsOren Shriki and Haim SompolinskyRacah Institute of Physics andCenter for Neural ComputationHebrew UniversityJerusalem, 91904, IsraelDaniel D. LeeBell LaboratoriesLucent TechnologiesMurray Hill, NJ 07974AbstractThe principle of maximizing mutual information is applied to learningovercomplete and recurrent representations. The underlying model con-sists of a network of input units driving a larger number of output unitswith recurrent interactions. In the limit of zero noise, the network is de-terministic and the mutual information can be related to the entropy ofthe output units. Maximizing this entropy with respect to both the feed-forward connections as well as the recurrent interactions results in simplelearning rules for both sets of parameters. The conventional independentcomponents (ICA) learning algorithm can be recovered as a special casewhere there is an equal number of output units and no recurrent con-nections. The application of these new learning rules is illustrated on asimple two-dimensional input example.IntroductionMany unsupervised learning algorithms such as principal component analysis, vector quan-tization, self-organizing feature maps, and others use the principle of minimizing recon-struction error to learn appropriate features from multivariate data [1, 2]. Independentcomponents analysis (ICA) can similarly be understood as maximizing the likelihood ofthe data under a non-Gaussian generative model, and thus is related to minimizing a re-construction cost [3, 4, 5]. On the other hand, the same ICA algorithm can also be derivedwithout regard to a particular generative model by maximizing the mutual information be-tween the data and a nonlinearly transformed version of the data [6]. This principle ofinformation maximization has also been previously applied to explain optimal propertiesfor single units, linear networks, and symplectic transformations [7, 8, 9].In these proceedings, we show how the principle of maximizing mutual information canbe generalized to overcomplete as well as recurrent representations. In the limit of zeronoise, we derive gradient descent learning rules for both the feedforward and recurrentweights. Finally, we show the application of these learning rules to some simple illustrativeexamples.M output variablesN input variablesx1 xNs1 s2 s3 sM11W MNW12K 3MKFeedforwardWeightsRecurrentWeightsFigure 1: Network diagram of an overcomplete, recurrent representation. x are input datawhich influence the output signals s through feedforward connections W . The signals salso interact with each other through the recurrent interactions K.Information MaximizationThe “Infomax” formulation of ICA considers the problem of maximizing the mutual in-formation between N -dimensional data observations fxg which are input to a networkresulting in N -dimensional output signals fsg [6]. Here, we consider the general problemwhere the signals s areM -dimensional with M  N . Thus, the representation is overcom-plete because there are more signal components than data components. We also considerthe situation where a signal component sican influence another component sjthrough arecurrent interaction Kji. As a network, this is diagrammed in Fig. 1 with the feedfor-ward connections described by the M N matrix W and the recurrent connections by theM M matrix K. The network response s is a deterministic function of the input x:si= g0@NXj=1Wijxj+MXk=1Kiksk1A (1)where g is some nonlinear squashing function. In this case, the mutual information betweenthe inputs x and outputs s is functionally only dependent on the entropy of the outputs:I(s;x) = H(s) H(sjx)  H(s): (2)The distribution of s is a N -dimensional manifold embedded in a M -dimensional vectorspace and nominally has a negatively divergent entropy. However, as shown in Appendix1, the probability density of s can be related to the input distribution via the relation:P (s) /P (x)pdet(T)(3)where the susceptibility (or Jacobian) matrix  is defined as:ij=@si@xj: (4)This result can be understood in terms of the singular value decomposition (SVD) of thematrix . The transformation performed by  can be decomposed into a series of threetransformations: an orthogonal transformation that rotates the axes, a diagonal transfor-mation that scales each axis, followed by another orthogonal transformation. A volumeelement in the input space is mapped onto a volume element in the output space, and itsvolume change is described by the diagonal scaling operation. This scale change is givenby the product of the square roots of the eigenvalues of T. Thus, the relationship be-tween the probability distribution in the input and output spaces includes the proportionalityfactor,pdet(T), as formally derived in Appendix 1.We now get the following expression for the entropy of the outputs:H(s)   ZdxP (x) log P (x)pdet(T)!=12log det(T)+H(x); (5)where the brackets indicate averaging over the input distribution.Learning rulesFrom Eq. (5), we see that minimizing the following cost function:E =  12Tr hlog(T)i; (6)is equivalent to maximizing the mutual information. We first note that the susceptibility satisfies the following recursion relation:ij= g0i Wij+XkKikkj!= (GW +GK)ij; (7)where Gij= Æijg0iand g0i g0PjWijxj+PkKiksk.Solving for  in Eq. (7) yields the result: = (G 1 K) 1W = W; (8)where  1  G 1  K. ijcan be interpreted as the sensitivity in the recurrent networkof the ith unit’s output to changes in the total input of the jth unit.We next derive the learning rules for the network parameters using gradient descent, asshown in detail in Appendix 2. The resulting expression for the learning rule for the feed-forward weights is:W =  @E@W=  T+TxT (9)where  is the learning rate, the matrix   is defined as  = (T) 1T (10)and the vector  is given byi= ( )iig00i(g0i)3: (11)Multiplying the gradient in Eq. (9) by the matrix (WW T ) yields an expression analogousto the “natural” gradient learning rule [10]:W = W I +TxT: (12)Similarly, the learning rule for the recurrent interactions isK =  @E@K= ( )T+TsT: (13)In the case when there are equal numbers of input and output units, M = N , and thereare no recurrent interactions, K = 0, most of the previous expressions simplify. Thesusceptibility matrix  is diagonal,  = G, and   = W 1. Substituting back into Eq. (9)for the learning rule for W results in the update rule:W = (WT) 1+zxT; (14)where zi= g00i=g0i. Thus, the well-known Infomax ICA learning rule is recovered as aspecial case of Eq. (9) [6].(a) (b) (c)Figure 2: Results of fitting 3 filters to a 2-dimensional hexagon distribution with 10000sample points.ExamplesWe now apply the preceding learning algorithms to a simple two-dimensional (N = 2)input example. Each input point is generated by a linear combination of three (two-dimensional) unit vectors with angles of 0Æ, 120Æ and 240Æ. The coefficients are takenfrom a uniform distribution on the unit interval. The resulting distribution has the shapeof a unit hexagon, which is slightly more dense close to the origin than at the boundaries.Samples of the input distribution are shown in Fig. 2. The second order cross correlationsvanish, so that all the structure in the data is described only by higher order correlations.We fix the sigmoidal nonlinearity to be g(x) = tanh(x).Feedforward weightsA set of M = 3 overcomplete filters for W are learned by applying the update rule inEq. (9) to random normalized initial conditions while keeping the recurrent interactionsfixed at K = 0. The length of the rows of W were constrained to be identical so that thefilters are projections along certain directions in the two-dimensional space. The algorithmconverged after about 20 iterations. Examples of the resulting learned filters are shownby plotting the rows of W as vectors in Fig. 2. As shown in the figure, there are severaldifferent local minimum solutions. If the lengths of the rows of W are left unconstrained,slight deviations from these solutions occur, but relative orientation differences of 60Æ or120Æ between the various filters are preserved.Recurrent interactionsTo investigate the effect of recurrent interactions on the representation, we fixed the feed-forward weights in W to point in the directions shown in Fig. 2(a), and learned the optimalrecurrent interactions K using Eq. (13). Depending upon the length of the rows of Wwhich scaled the input patterns, different optimal values are seen for the recurrent connec-tions. This is shown in Fig. 3 by plotting the value of the cost function against the strengthof the uniform recurrent interaction. For small scaled inputs, the optimal recurrent strengthis negative which effectively amplifies the output signals since the 3 signals are negativelycorrelated. With large scaled inputs, the optimal recurrent strength is positive which tend todecrease the outputs. Thus, in this example, optimizing the recurrent connections performsgain control on the inputs.-1.5 -1 -0.5 0 0.5 1 1.5-1-0.500.511.522.53kCost|W|=1 |W|=5Figure 3: Effect of adding recurrent interactions to the representation. The cost functionis plotted as a function of the recurrent interaction strength, for two different input scalingparameters.DiscussionThe learned feedforward weights are similar to the results of another ICA model that canlearn overcomplete representations [11]. Our algorithm, however, does not need to performapproximate inference on a generative model. Instead, it directly maximizes the mutual in-formation between the outputs and inputs of a nonlinear network. Our method also hasthe advantage of being able to learn recurrent connections that can enhance the representa-tional power of the network. We also note that this approach can be easily generalized toundercomplete representations by simply changing the order of the matrix product in thecost function. However, more work still needs to be done in order to understand technicalissues regarding speed of convergence and local minima in larger applications. Possibleextensions of this work would be to optimize the nonlinearity that is used, or to adaptivelychange the number of output units to best match the input distribution.We acknowledge the financial support of Bell Laboratories, Lucent Technologies, and theUS-Israel Binational Science Foundation.Appendix 1: Relationship between input and output distributionsIn general, the relation between the input and output distributions is given byP (s) =ZdxP (x)P (sjx): (15)Since we use a deterministic mapping, the conditional distribution of the response giventhe input is given by P (sjx) = Æ(s   g(Wx + Ks)). By adding independent Gaussiannoise to the responses of the output units and considering the limit where the variance ofthe noise goes to zero, we can write this term asP (sjx) = lim!01(22)N=2e 122ks g(Wx+Ks)k2: (16)The output space can be partitioned into those points which belong to the image of theinput space, and those which are not. For points outside the image of the input space,P (s) = 0. Consider a point s inside the image. This means that there exists x0such thats = g(Wx0+Ks). For small , we can expand g(Wx +Ks)   s ' Æx, where  isdefined in Eq. (4), and Æx = x  x0. We then getP (sjx) = lim!01(22)N=2e 12ÆxTT2Æx=1pdet (T)2664lim!0e 12ÆxTT2Æx(2)N=2rdet2(T) 13775: (17)The expression in the square brackets is a delta function in x around x0. Using Eq. (15) wefinally getP (s) =P (x)pdet(T)(s) (18)where the characteristic function (s) is 1 if s belongs to the image of the input spaceand is zero otherwise. Note that for the case when  is a square matrix (M = N ), thisexpression reduces to the relation P (s) = P (x)=j det()j.Appendix 2: Derivation of the learning rulesTo derive the appropriate learning rules, we need to calculate the derivatives of E withrespect to some set of parameters . In general, these derivatives are obtained from theexpression:@E@=  12Tr(T) 1@(T)@=  Tr(T) 1T@@: (19)Feedforward weightsIn order to derive the learning rule for the weights W , we first calculate@ab@Wlm=Xcac@Wcb@Wlm+@ac@WlmWcb= alÆbm+Xc@ac@WlmWcb: (20)From the definition of , we see that:@ac@Wlm=  Xijai@G 1ij@Wlmjc(21)and@G 1ij@Wlm=  Æij(g0i)2@g0i@Wlm=  Æijg00i(g0i)3@si@Wlm; (22)where g00i g00PjWijxj+PkKiksk.The derivatives of s also satisfy a recursion relation similar to Eq. (7):@si@Wlm= g0i0@Æilxm+XjKij@sj@Wlm1A; (23)which has the solution:@si@Wlm= ilxm: (24)Putting all these results together in Eq. (19) and taking the trace, we get the gradient descentrule in Eq. (9).Recurrent interactionsTo derive the learning rules for the recurrent weights K, we first calculate the derivativesof abwith respect to Klm:@ab@Klm=Xc@ac@KlmWcb=  Xc;i;jai@ 1ij@KlmjcWcb: (25)From the definition of , we obtain:@ 1ij@Klm=  Æij(g0i)2@g0i@Klm  ÆilÆjm: (26)The derivatives of g0 are obtained from the following relations:@g0i@Klm=g00ig0i@si@Klm(27)and@si@Klm= ilsm: (28)which results from a recursion relation similar to Eq. (23). Finally, after combining theseresults and calculating the trace, we get the gradient descent learning rule in Eq. (13).References[1] Jolliffe, IT (1986). Principal Component Analysis. New York: Springer-Verlag.[2] Haykin, S (1999). Neural networks: a comprehensive foundation. 2nd ed., Prentice-Hall, UpperSaddle River, NJ.[3] Jutten, C & Herault, J (1991). Blind separation of sources, part I: An adaptive algorithm basedon neuromimetic architecture. Signal Processing 24, 1–10.[4] Hinton, G & Ghahramani, Z (1997). Generative models for discovering sparse distributed rep-resentations. Philosophical Transactions Royal Society B 352, 1177–1190.[5] Pearlmutter, B & Parra, L (1996). A context-sensitive generalization of ICA. In ICONIP’96,151–157.[6] Bell, AJ & Sejnowski, TJ (1995). An information maximization approach to blind separationand blind deconvolution. Neural Comput. 7, 1129–1159.[7] Barlow, HB (1989). Unsupervised learning. Neural Comput. 1, 295–311.[8] Linsker, R (1992). Local synaptic learning rules suffice to maximize mutual information in alinear network. Neural Comput. 4, 691–702.[9] Parra, L, Deco, G, & Miesbach, S (1996). Statistical independence and novelty detection withinformation preserving nonlinear maps. Neural Comput. 8, 260–269.[10] Amari, S, Cichocki, A & Yang, H (1996). A new learning algorithm for blind signal separation.Advances in Neural Information Processing Systems 8, 757–763.[11] Lewicki, MS & Sejnowski, TJ (2000). Learning overcomplete representations. Neural Compu-tation 12 337–365.

An Information Maximization Approach to Overcomplete and Recurrent Representations

Kosmopolis

University of Pennsylvania
ScholarlyCommons
Departmental Papers (ESE) Department of Electrical & Systems Engineering
November 2000
An Information Maximization Approach to
Overcomplete and Recurrent Representations
Oren Shriki
Hebrew University
Haim Sompolinsky
Hebrew University
Daniel D. Lee
University of Pennsylvania, ddlee@seas.upenn.edu
Follow this and additional works at: http://repository.upenn.edu/ese_papers
Reprinted from 12th Conference on Neural Information Processing Systems (NIPS 2000), November 2000, pages 87-93.
NOTE: At the time of publication, author Daniel D. Lee was affiliated with Bell Laboratories. Currently (March 2005), he is a faculty member in the
Department of Electrical and Systems Engineering at the University of Pennsylvania.
This paper is posted at ScholarlyCommons. http://repository.upenn.edu/ese_papers/85
For more information, please contact repository@pobox.upenn.edu.
Recommended Citation
Oren Shriki, Haim Sompolinsky, and Daniel D. Lee, "An Information Maximization Approach to Overcomplete and Recurrent
Representations", . November 2000.
An Information Maximization Approach to Overcomplete and Recurrent
Representations
Abstract
The principle of maximizing mutual information is applied to learning overcomplete and recurrent
representations. The underlying model consists of a network of input units driving a larger number of output
units with recurrent interactions. In the limit of zero noise, the network is deterministic and the mutual
information can be related to the entropy of the output units. Maximizing this entropy with respect to both
the feedforward connections as well as the recurrent interactions results in simple learning rules for both sets
of parameters. The conventional independent components (ICA) learning algorithm can be recovered as a
special case where there is an equal number of output units and no recurrent connections. The application of
these new learning rules is illustrated on a simple two-dimensional input example.
Comments
Reprinted from 12th Conference on Neural Information Processing Systems (NIPS 2000), November 2000, pages
87-93.
NOTE: At the time of publication, author Daniel D. Lee was affiliated with Bell Laboratories. Currently
(March 2005), he is a faculty member in the Department of Electrical and Systems Engineering at the
University of Pennsylvania.
This conference paper is available at ScholarlyCommons: http://repository.upenn.edu/ese_papers/85
An Information Maximization Approach to
Overcomplete and Recurrent Representations
Oren Shriki and Haim Sompolinsky
Racah Institute of Physics and
Center for Neural Computation
Hebrew University
Jerusalem, 91904, Israel
Daniel D. Lee
Bell Laboratories
Lucent Technologies
Murray Hill, NJ 07974
Abstract
The principle of maximizing mutual information is applied to learning
overcomplete and recurrent representations. The underlying model con-
sists of a network of input units driving a larger number of output units
with recurrent interactions. In the limit of zero noise, the network is de-
terministic and the mutual information can be related to the entropy of
the output units. Maximizing this entropy with respect to both the feed-
forward connections as well as the recurrent interactions results in simple
learning rules for both sets of parameters. The conventional independent
components (ICA) learning algorithm can be recovered as a special case
where there is an equal number of output units and no recurrent con-
nections. The application of these new learning rules is illustrated on a
simple two-dimensional input example.
Introduction
Many unsupervised learning algorithms such as principal component analysis, vector quan-
tization, self-organizing feature maps, and others use the principle of minimizing recon-
struction error to learn appropriate features from multivariate data [1, 2]. Independent
components analysis (ICA) can similarly be understood as maximizing the likelihood of
the data under a non-Gaussian generative model, and thus is related to minimizing a re-
construction cost [3, 4, 5]. On the other hand, the same ICA algorithm can also be derived
without regard to a particular generative model by maximizing the mutual information be-
tween the data and a nonlinearly transformed version of the data [6]. This principle of
information maximization has also been previously applied to explain optimal properties
for single units, linear networks, and symplectic transformations [7, 8, 9].
In these proceedings, we show how the principle of maximizing mutual information can
be generalized to overcomplete as well as recurrent representations. In the limit of zero
noise, we derive gradient descent learning rules for both the feedforward and recurrent
weights. Finally, we show the application of these learning rules to some simple illustrative
examples.
M output variables
N input variables
x1 xN
s1 s2 s3 sM
11W MNW
12K 3MK
Feedforward
Weights
Recurrent
Weights
Figure 1: Network diagram of an overcomplete, recurrent representation.x are input data
which influence the output signalss through feedforward connectionsW . The signalss
also interact with each other through the recurrent interactionsK.
Information Maximization
The “Infomax” formulation of ICA considers the problem of maximizing the mutual in-
formation betweenN -dimensional data observationsfxg which are input to a network
resulting inN -dimensional output signalsfsg [6]. Here, we consider the general problem
where the signalsareM -dimensional withM  N . Thus, the representation is overcom-
plete because there are more signal components than data components. We also consider
the situation where a signal componentsi can influence another componentsj through a
recurrent interactionKji. As a network, this is diagrammed in Fig. 1 with the feedfor-
ward connections described by theM N matrixW and the recurrent connections by the
M M matrixK. The network responses is a deterministic function of the inputx:
si = g
0
@ NX
j=1
Wijxj +
MX
k=1
Kiksk
1
A (1)
whereg is some nonlinear squashing function. In this case, the mutual information between
the inputsx and outputss is functionally only dependent on the entropy of the outputs:
I(s;x) = H(s) H(sjx)  H(s): (2)
The distribution ofs is aN -dimensional manifold embedded in aM -dimensional vector
space and nominally has a negatively divergent entropy. However, as shown in Appendix
1, the probability density ofs can be related to the input distribution via the relation:
P (s) /
P (x)p
det(T)
(3)
where the susceptibility (or Jacobian) matrix is defined as:
ij =
@si
@xj
: (4)
This result can be understood in terms of the singular value decomposition (SVD) of the
matrix . The transformation performed by can be decomposed into a series of three
transformations: an orthogonal transformation that rotates the axes, a diagonal transfor-
mation that scales each axis, followed by another orthogonal transformation. A volume
element in the input space is mapped onto a volume element in the output space, and its
volume change is described by the diagonal scaling operation. This scale change is given
by the product of the square roots of the eigenvalues ofT. Thus, the relationship be-
tween the probability distribution in the input and output spaces includes the proportionality
factor,
p
det(T), as formally derived in Appendix 1.
We now get the following expression for the entropy of the outputs:
H(s)   
Z
dxP (x) log
 
P (x)p
det(T)
!
=
1
2


log det(T)

+H(x); (5)
where the brackets indicate averaging over the input distribution.
Learning rules
From Eq. (5), we see that minimizing the following cost function:
E =  
1
2
Tr hlog(T)i; (6)
is equivalent to maximizing the mutual information. We first note that the susceptibility
satisfies the following recursion relation:
ij = g
0
i 
 
Wij +
X
k
Kikkj
!
= (GW +GK)ij ; (7)
whereGij = Æijg0i andg
0
i  g
0
P
j Wijxj +
P
kKiksk

.
Solving for in Eq. (7) yields the result:
 = (G 1  K) 1W = W; (8)
where 1  G 1  K. ij can be interpreted as the sensitivity in the recurrent network
of theith unit’s output to changes in the total input of thejth unit.
We next derive the learning rules for the network parameters using gradient descent, as
shown in detail in Appendix 2. The resulting expression for the learning rule for the feed-
forward weights is:
W =  
@E
@W
= 


 T +T xT

(9)
where is the learning rate, the matrix  is defined as
  = (T) 1T (10)
and the vector is given by
i = ( )ii
g00i
(g0i)
3
: (11)
Multiplying the gradient in Eq. (9) by the matrix(WW T ) yields an expression analogous
to the “natural” gradient learning rule [10]:
W = W
 
I +


T xT

: (12)
Similarly, the learning rule for the recurrent interactions is
K =  
@E
@K
= 


( )T +T sT

: (13)
In the case when there are equal numbers of input and output units,M = N , and there
are no recurrent interactions,K = 0, most of the previous expressions simplify. The
susceptibility matrix is diagonal, = G, and  = W 1. Substituting back into Eq. (9)
for the learning rule forW results in the update rule:
W = 

(W T ) 1 +


zx
T

; (14)
wherezi = g00i =g
0
i. Thus, the well-known Infomax ICA learning rule is recovered as a
special case of Eq. (9) [6].
(a) (b) (c)
Figure 2: Results of fitting 3 filters to a 2-dimensional hexagon distribution with 10000
sample points.
Examples
We now apply the preceding learning algorithms to a simple two-dimensional (N = 2)
input example. Each input point is generated by a linear combination of three (two-
dimensional) unit vectors with angles of0Æ, 120Æ and240Æ. The coefficients are taken
from a uniform distribution on the unit interval. The resulting distribution has the shape
of a unit hexagon, which is slightly more dense close to the origin than at the boundaries.
Samples of the input distribution are shown in Fig. 2. The second order cross correlations
vanish, so that all the structure in the data is described only by higher order correlations.
We fix the sigmoidal nonlinearity to beg(x) = tanh(x).
Feedforward weights
A set ofM = 3 overcomplete filters forW are learned by applying the update rule in
Eq. (9) to random normalized initial conditions while keeping the recurrent interactions
fixed atK = 0. The length of the rows ofW were constrained to be identical so that the
filters are projections along certain directions in the two-dimensional space. The algorithm
converged after about 20 iterations. Examples of the resulting learned filters are shown
by plotting the rows ofW as vectors in Fig. 2. As shown in the figure, there are several
different local minimum solutions. If the lengths of the rows ofW are left unconstrained,
slight deviations from these solutions occur, but relative orientation differences of60Æ or
120Æ between the various filters are preserved.
Recurrent interactions
To investigate the effect of recurrent interactions on the representation, we fixed the feed-
forward weights inW to point in the directions shown in Fig. 2(a), and learned the optimal
recurrent interactionsK using Eq. (13). Depending upon the length of the rows ofW
which scaled the input patterns, different optimal values are seen for the recurrent connec-
tions. This is shown in Fig. 3 by plotting the value of the cost function against the strength
of the uniform recurrent interaction. For small scaled inputs, the optimal recurrent strength
is negative which effectively amplifies the output signals since the 3 signals are negatively
correlated. With large scaled inputs, the optimal recurrent strength is positive which tend to
decrease the outputs. Thus, in this example, optimizing the recurrent connections performs
gain control on the inputs.
-1.5 -1 -0.5 0 0.5 1 1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
k
C
os
t
|W|=1 |W|=5
Figure 3: Effect of adding recurrent interactions to the representation. The cost function
is plotted as a function of the recurrent interaction strength, for two different input scaling
parameters.
Discussion
The learned feedforward weights are similar to the results of another ICA model that can
learn overcomplete representations [11]. Our algorithm, however, does not need to perform
approximate inference on a generative model. Instead, it directly maximizes the mutual in-
formation between the outputs and inputs of a nonlinear network. Our method also has
the advantage of being able to learn recurrent connections that can enhance the representa-
tional power of the network. We also note that this approach can be easily generalized to
undercomplete representations by simply changing the order of the matrix product in the
cost function. However, more work still needs to be done in order to understand technical
issues regarding speed of convergence and local minima in larger applications. Possible
extensions of this work would be to optimize the nonlinearity that is used, or to adaptively
change the number of output units to best match the input distribution.
We acknowledge the financial support of Bell Laboratories, Lucent Technologies, and the
US-Israel Binational Science Foundation.
Appendix 1: Relationship between input and output distributions
In general, the relation between the input and output distributions is given by
P (s) =
Z
dxP (x)P (sjx): (15)
Since we use a deterministic mapping, the conditional distribution of the response given
the input is given byP (sjx) = Æ(s   g(Wx + Ks)). By adding independent Gaussian
noise to the responses of the output units and considering the limit where the variance of
the noise goes to zero, we can write this term as
P (sjx) = lim
!0
1
(22)
N=2
e 
1
22
ks g(Wx+Ks)k2 : (16)
The output space can be partitioned into those points which belong to the image of the
input space, and those which are not. For points outside the image of the input space,
P (s) = 0. Consider a points inside the image. This means that there existsx0 uch that
s = g(Wx0 +Ks). For small, we can expandg(Wx +Ks)   s ' Æx, where is
defined in Eq. (4), andÆx = x  x0. We then get
P (sjx) = lim
!0
1
(22)
N=2
e
  1
2
ÆxT

T
2

Æx
=
1p
det (T)
2
664 lim!0 e
  1
2
ÆxT

T
2

Æx
(2)N=2
r
det

2 (T)
 1

3
775 : (17)
The expression in the square brackets is a delta function inx aroundx0. Using Eq. (15) we
finally get
P (s) =
P (x)p
det(T)

(s) (18)
where the characteristic function
(s) is 1 if s belongs to the image of the input space
and is zero otherwise. Note that for the case when is a square matrix (M = N ), this
expression reduces to the relationP (s) = P (x)=j det()j.
Appendix 2: Derivation of the learning rules
To derive the appropriate learning rules, we need to calculate the derivatives ofE with
respect to some set of parameters. In general, these derivatives are obtained from the
expression:
@E
@
=  
1
2
Tr

(T) 1
@(T)
@

=  Tr

(T) 1T
@
@

: (19)
Feedforward weights
In order to derive the learning rule for the weightsW , we first calculate
@ab
@Wlm
=
X
c

ac
@Wcb
@Wlm
+
@ac
@Wlm
Wcb

= alÆbm +
X
c
@ac
@Wlm
Wcb: (20)
From the definition of, we see that:
@ac
@Wlm
=  
X
ij
ai
@G 1ij
@Wlm
jc (21)
and
@G 1ij
@Wlm
=  
Æij
(g0i)
2
@g0i
@Wlm
=  Æij
g00i
(g0i)
3
@si
@Wlm
; (22)
whereg00i  g
00
P
j Wijxj +
P
kKiksk

.
The derivatives ofs also satisfy a recursion relation similar to Eq. (7):
@si
@Wlm
= g0i 
0
@Æilxm +X
j
Kij
@sj
@Wlm
1
A ; (23)
which has the solution:
@si
@Wlm
= ilxm: (24)
Putting all these results together in Eq. (19) and taking the trace, we get the gradient descent
rule in Eq. (9).
Recurrent interactions
To derive the learning rules for the recurrent weightsK, we first calculate the derivatives
of ab with respect toKlm:
@ab
@Klm
=
X
c
@ac
@Klm
Wcb =  
X
c;i;j
ai
@ 1ij
@Klm
jcWcb: (25)
From the definition of, we obtain:
@ 1ij
@Klm
=  
Æij
(g0i)
2
@g0i
@Klm
  ÆilÆjm: (26)
The derivatives ofg0 are obtained from the following relations:
@g0i
@Klm
=
g00i
g0i
@si
@Klm
(27)
and
@si
@Klm
= ilsm: (28)
which results from a recursion relation similar to Eq. (23). Finally, after combining these
results and calculating the trace, we get the gradient descent learning rule in Eq. (13).
References
[1] Jolliffe, IT (1986).Principal Component Analysis.New York: Springer-Verlag.
[2] Haykin, S (1999).Neural networks: a comprehensive foundation.2 d ed., Prentice-Hall, Upper
Saddle River, NJ.
[3] Jutten, C & Herault, J (1991). Blind separation of sources, part I: An adaptive algorithm based
on neuromimetic architecture.Signal Processing24,1–10.
[4] Hinton, G & Ghahramani, Z (1997). Generative models for discovering sparse distributed rep-
resentations.Philosophical Transactions Royal Society B352, 1177–1190.
[5] Pearlmutter, B & Parra, L (1996). A context-sensitive generalization of ICA. In ICONIP’96,
151–157.
[6] Bell, AJ & Sejnowski, TJ (1995). An information maximization approach to blind separation
and blind deconvolution.Neural Comput.7, 1129–1159.
[7] Barlow, HB (1989). Unsupervised learning.Neural Comput.1, 295–311.
[8] Linsker, R (1992). Local synaptic learning rules suffice to maximize mutual information in a
linear network.Neural Comput.4, 691–702.
[9] Parra, L, Deco, G, & Miesbach, S (1996). Statistical independence and novelty detection with
information preserving nonlinear maps.Neural Comput.8, 260–269.
[10] Amari, S, Cichocki, A & Yang, H (1996). A new learning algorithm for blind signal separation.
Advances in Neural Information Processing Systems8, 757–763.
[11] Lewicki, MS & Sejnowski, TJ (2000). Learning overcomplete representations.Neural Compu-
tation12337–365.


https://repository.upenn.edu/cgi/viewcontent.cgi?article=1084&amp;context=ese_papers

An Information Maximization Approach to Overcomplete and Recurrent Representations

Abstract

Similar works

Full text

Available Versions

ScholarlyCommons@Penn

Kosmopolis