The remarkable collective computational properties of the Hopfield model for neural networks [Proc. Nat. Acad. Sci. USA 79, 2554 (1982)] are reviewed. These include recognition from partial input, robustness, and error-correction capability. Features of the model that make its optical implementation attractive are discussed, and specific optical implementation schemes are given

Farhat, Nabil

Psaltis, Demetri

English

Caltech Authors - Main

98 OPTICS LETTERS / Vol. 10, No. 2 / February 1985
Optical information processing based on an associative-memory
model of neural nets with thresholding and feedback
Demetri Psaltis and Nabil Farhat*
Department of Electrical Engineering, California Institute of Technology, Pasadena, California 91125
Received July 9, 1984; accepted November 15, 1984
The remarkable collective computational properties of the Hopfield model for neural networks [Proc. Nat. Acad.
Sci. USA 79, 2554 (1982)] are reviewed. These include recognition from partial input, robustness, and error-correc-
tion capability. Features of the model that make its optical implementation attractive are discussed, and specific
optical implementation schemes are given.
Optical information-processing systems can have high
processing power because of the large degree of paral-
lelism as well as the interconnection capability that is
achievable. Typically, more than 106 parallel pro-
cessing channels are available in the optical system, and
furthermore each of these channels can be optically
interconnected (broadcasted) to 106 other channels.
The majority of optical processors are analog systems,
designed to perform linear operations. The accuracy
of an analog processor is limited by the linear dynamic
range of the devices used (detectors, light modulators).
In principle, the accuracy and the repertoire of achiev-
able operations can be improved with systems that
perform nonlinear operations on binary encoded data
using bistable optical devices. Optical bistability is a
subject that has received considerable attention recently
as a means of achieving efficient high-speed logic, and
it has been demonstrated with several nonlinear optical
materials and devices. If we are to use such bistable
devices to realize powerful, nonlinear optical computers,
it is important to find algorithms that are well matched
to the characteristics of the optical processor and utilize
effectively its parallelism and interconnection capa-
bility. In this Letter we examine a method for syn-
thesizing optical processing systems, based on optical
associative memory and threshold logic, that appears
to meet these requirements well.
Associative (or content-addressable) memories are
of interest in computer science, and it is theorized that
information is stored in the human brain in this manner.
Holographic associative memories have been described
by Gabor,' who also commented on the similarity of the
holographic memory to the way information may be
stored in the human brain. More recently, Hopfield2
introduced an associative-memory model to describe
the collective behavior of neural networks. Hopfield's
model consists basically of an associative memory
similar to the holographic, with the addition of
threshold and feedback. The incorporation of non-
linear feedback enhances dramatically the error-cor-
recting capability of the holographic memory.
Let vi (in be a binary word that is N bits long. M such
words are stored in a matrix Tij according to
M
TJ = E |2 0m-][V~)1 if i lj
if i = J
If T-1 is multiplied by one of the stored binary vectors
v(m) the product Oi (mO) is an estimate of the stored
vector [2vi (mO) - 1]:
(gMO) = E TijVi (MO) = NO[2vi (mO) -1]
N
+ E z [2vj(m) - 11vjmO)
m54mO i
X [2vi ) - 1] -Mvi (m), (2)
where the last term accounts for Tij = 0 and No is the
number of l's in vUi(O). We assume that for m # mo
the binary words vi (m) are statistically described in the
following simple manner:
P[Vi(M) = 1] = 1/2, (3)
where vim) are independent for all i and m. Then
E[Oi mO)] = (N/2)[2vimO) - 1] and var[Oi(iOl = N(M
- 1)/2. We define the signal-to-noise ratio (SNR) of
the estimate 6i (mO) as the ratio of the magnitude of the
expected value of Oi (mO) to the standard deviation of the
estimate:
SNR = {l E[6i(mO)]I1/varf[O (mO)]J1/2
= [N/2(M - 1)]1/2. (4)
If N is sufficiently larger than M, then with high prob-
ability TH[Oi(iO)l = vj(in), where TH[bi(mO)J = 1 if
Ui (mO) > 0 and zero otherwise. Thus the vector-matrix
product in Eq. (2) combined with the thresholding op-
eration results in a pseudoeigensystem in that the out-
put vector equals the input. Now suppose that the full
vector vi (mO) is in fact such a pseudoeigenvector of the
system but that only N1 of N bits (N1 < N) of vi (mO) are
known. In this case we define an input vector consist-
ing of the N1 known bits, and -the rest are set equal to
zero. When this vector is multiplied by the matrix Tij,
an estimate of the complete vector [2vi (mO) - 1] is ob-
tained. The SNR of the estimate is now SNR = [N1!
2(M - 1)11/2. If N1 becomes sufficiently small, then,
with high probability, TH[6i (mO)] Fs vi (mO) for some of
the values of i. Let N2 be the number of correct bits in
THl[OimO)]. If N2 > N1, we can multiply this thresh-
olded estimate by Tij and obtain a new estimate with
a higher SNR. This procedure can be continued until
the number of correct bits in the thresholded vector is
equal to N. The crucial issue is under what conditions
N 2 will be higher than N 1. If the SNR of the initial
0146-9592/85/020098-03$2.00/0 © 1985, Optical Society of America
P [Vi (M) = 01 = 1/2,
February 1985 / Vol. 10, No. 2 / OPTICS LETTERS 99
estimate I[N/(2M - 1)]1/2} is sufficiently large, then the
probability of N2 being bigger than N 1 will be high; in
this case the nonlinear, iterative procedure described
above will be likely to converge to the correct vector
Vi (mO). Ideally, each of the M stored binary words is a
pseudoeigenvector of the nonlinear system. Notice that
each pseudoeigenstate is a stable state of the system,
whereas any other input vector (state) will cause a
change to occur in the next cycle. In general, the system
converges to the stable state that is at the shortest
Hamming distance away from the initial state.
This model has been studied computationally by
Hopfield. 2 In simulations, correct convergence was
obtained reliably for M $ 0.15N and N1 0.75N, taking
N = 30. At present there is no (adequate) theoretical
prediction of the maximum number of words that can
be stored or the maximum Hamming distance between
the input vector and one of the stored words that is re-
quired for convergence. Several interesting properties
were observed. The model does not require synchro-
nism. Convergence can be obtained if the output vector
is fed back to the input as a whole or, randomly, one
element at a time. There is some evidence that asyn-
chronous operation is actually preferable. The system
is quite insensitive to imperfections such as nonuni-
formities, the exact form of the threshold operation, and
errors in the Tij matrix. Convergence to the correct
vector was obtained even when the Tij matrix was
thresholded. Such properties are most desirable when
an optical implementation is considered.
One possible optical implementation of the Hopfield
model is through the arrangements shown in Fig. 1, in
which the array of light-emitting diodes (LED's) rep-
resents N logic elements with binary states vj = 0, 1, j
= 0, 1, ... .N (LED on or off), which are to be intercon-
nected in accordance with the model. This is achieved
by the addition of nonlinear feedback (feedback,
thresholding, and gain) to the well-known optical vec-
tor-matrix multiplier. 3 Gain is included in the feedback
loop to compensate for losses. Two possible feedback
schemes are shown. One uses electronic wiring and the
other is optical, with the thresholding (point nonlin-
earity) and the gain concentrated between the photo-
diode (PD) array and the LED array, which can be
fabricated monolithically on GaAs. Furthermore, with
the accelerating pace of research in thin-film nonlinear
light amplifiers4 and optical bistable devices,5 it can be
possible to substitute a single distributed bistable
light-amplifier device for the PD/LED arrays and the
intervening thresholding and amplifying electronics.
Multiplication of the vector vj by the Tij matrix in
these schemes is accomplished by horizontal imaging
and vertical smearing of v; using anamorphic optics
(omitted from Fig. 1 for simplicity). A bipolar Tij can
be realized optoelectronically with incoherent light by
assigning its negative and positive values to adjacent
rows. Light passing through each row is focused onto
adjacent pairs of photodiodes of the PD array that are
electronically connected in opposition, as shown in Fig.
2. Here the positive and negative elements of each row
of the Tij matrix are separated into two subrows, one for
positive values and one for negative. The light trans-
mitted through the two subrows is integrated horizon-
SN~ G I. V" * t .
,)j m.V- r-0 
PO ARRAY
ARRAY MEMORY MASK
LED PD
FEEODBACK.GAIN, AN0 THRESHOLOING
(a) ELECTRONIC FEEDBACK SCHEME
Mt
seMuORI M2
MASK ~ ~ SREEN~_
R <Xt ~~~~MISS
9 4 to > ' '- IMAGE ROTATOR
LZ /4
M3
(b) OPTICAL FEEDBACK SCHEME
Fig. 1. Two schemes for adding nonlinear feedback to an
optical vector-matrix multiplier utilizing (a) electronic feed-
back and (b) optical feedback.
tally with the aid of another set of anamorphic lenses
(omitted from Figs. 1 and 2) and brought to focus on two
adjacent photodiodes of the PD array connected in
opposition. The output of the first diode-pair circuit
will be proportional, u1 = 2j T 1jvj. This output is ap-
plied through an electronic thresholding circuit to the
first element of the LED array, as shown in Fig. 1.
Similar connections are made between other detector
pairs of the photodector array and corresponding ele-
ments in the LED array. Thus each LED assesses the
state of its input vi = 2j Trove and fires according to
whether vi exceeds the threshold or not.
We now consider the possibility of optically storing
two-dimensional (2-D) functions (images). Let V0m)(i,
i') be the bipolar binary (1, -1) images to be stored. If
we directly extend the Hopfield model to two dimen-
sions, then these images must be stored in a four-di-
mensional function in the following general form:
M
T(i, i', j j') = E v (m)(i, iV)v(m)( j').
m
(5)
In order to implement a 2-D Hopfield memory opti-
cally, we need to realize a 2-D, linear optical system
whose spatial impulse response is the four-dimensional
function defined in Eq. (5). Since we have only two
spatial coordinates to work with in an optical system,
it is difficult to implement such a system directly for the
nonseparable, shift-variant kernel defined in Eq. (5).
One possible solution is the use of wavelength multi-
plexing and/or time-domain processing to obtain ad-
ditional independent variables. Another solution is
based on holographic associative memories, as we have
discussed earlier.6 Here we present an implementation
based on spatial-frequency multiplexing.
The entire optical system, including nonlinear feed-
back, is shown in Fig. 3. The system accepts a 2-D
100 OPTICS LETTERS / Vol. 10, No. 2 / February 1985
Tij MASK
Fig. 2. Scheme for realizing bipolar mask transmittance with
incoherent light.
Fig. 3. Coherent optical implementation for 2-D inputs.
input V mO)(j, i'), which illuminates the system from the
left in Fig. 3. The 2-D interconnection pattern between
the planes P1 and P4 that is prescribed by Eq. (5) is
stored on two separate optical transparencies, denoted
T1 and T2 in Fig. 3. Each image v(m)(i, i') is placed on
a separate spatial-frequency carrier, and the images are
added to form the first transparency T,. When T1 is
illuminated with an input image, the products between
the input and all the stored images are formed. The
lens L, produces the Fourier transforms of all the
products and displays them spatially separated at the
back focal plane of L1 (plane P2). An array of pinholes
is placed at P1, the position of each pinhole being at the
spatial frequency of each carrier used in the recording
of transparency T1. Therefore the amplitude of the
light transmitted through each individual pinhole is
proportional to the integral of the product of the input
image and the corresponding image stored in T,. In the
discrete notation used in this Letter, the amplitude of
the light transmitted through the mth pinhole is pro-
portional to 12 O(mO)(i, I)v(m)(i, j). The second
transparency, T2(k, k'), consists of a 2-D array of Fou-
rier-transform holograms of all the stored images. Each
hologram is formed at a separate position on the holo-
gram, the transform of the mth image being centered
at the location of the corresponding mth pinhole. Light
emerging from each pinhole illuminates only the cor-
responding hologram. The lens L2 takes the Fourier
transform between planes P3 and P4, thereby recon-
structing all the images stored in T2. The light ampli-
tude at P4 is a weighted sum of all the stored images, the
weights being proportional to the inner product between
the input and stored images. This is precisely the de-
sired output that is produced by the interconnection
prescription given in Eq. (5). The modulation of the
light at plane P4 will in general be bipolar, and it is in-
terferometrically detected by the nonlinear optical
amplifier at P4, which performs the thresholding op-
eration. If interferometric detection proves to be too
cumbersome, it is possible to modify the interconnection
pattern such that the output is always positive. This
results in a loss of storage capacity by a factor of 2, but
it may be a welcome trade-off. The thresholded image
is fed back to the input through mirrors and imaging
optics (lenses L3 and L4). The output of the system is
taken at the beam splitter. Optical gain must be in-
cluded in the cavity (preferably through the bistable
optical element) to compensate for losses that are due
to the passive components. The requirement that T(i,
i', j, j') = 0 for i = i', j = j' must also be satisfied because
otherwise the diagonal elements always become equal
to M, whereas the off-diagonal elements have an average
value of -/IM. The result is that for large M the system
becomes an imaging system; any input is replicated at
the output. This is avoided by forming each of the
Fourier-transform holograms in P3 with a randomly
chosen, uniform phase.
We have described several specific optical imple-
mentations of the Hopfield model; undoubtedly others
are also possible. The most important feature of all
such implementations is the robustness of a system that
utilizes nonlinear feedback. The systems that we have
described behave basically as associative memories (the
whole is retrieved from a partial input), even with
open-loop operation. However, the nonlinear feedback
can correct errors of the open-loop system since it forces
the state of the system to change continuously until a
stable condition is reached. The nonlinearity plays a
crucial role; if linear feedback were used, the system
would either be unstable or converge to the eigenstate
of the open-loop system with the highest eigenvalue,
independently of the initial condition.
This error-correcting capability can provide the ac-
curacy that is lacking from analog optical processors
without, however, sacrificing the processing power that
can be derived from the global processing capability of
optics; the class of processors that we described are fully
interconnected optical systems and hence utilize fully
the parallelism and the interconnective capability of
optics. In general, there is an excellent match between
the global, linear operations and local, point non-
linearities that are required for the implementation of
the Hopfield model, and the capabilities and limitations
of optical techniques.
The authors thank John Hong and Yaser Abu-Mos-
tafa for many helpful discussions on this subject.
* On scholarly leave from the University of Penn-
sylvania, Philadelphia, Pennsylvania 19104.
References
1. D. Gabor, IBM J. Res. Dev. 13, 156 (1969).
2. J. J. Hopfield, Proc. Nat. Acad. Sci. USA 79, 2554
(1982).
3. J. Goodman, A. R. Dias, and I. M. Woody, Opt. Lett. 2, 1
(1978).
4. Z. Porada, Thin Solid Films 109, 213 (1983).
5. H. M. Gibbs, S. L. McCall, and T. N. C. Venkatesan, Opt.
News 5(3), 6 (1979).
6. D. Psaltis and N. Farhat, presented at the Meeting of the
International Commission for Optics, ICO-13, Sapporo,
Japan, August 1984.


Optical information processing based on an associative-memory model of neural nets with thresholding and feedback

presented at the Meeting

Thin Solid Films 109,

https://authors.library.caltech.edu/3265/1/PSAol85.pdf

Optical information processing based on an associative-memory model of neural nets with thresholding and feedback

Abstract

Similar works

Full text

Available Versions

Caltech Authors - Main