921 research outputs found
Neural Distributed Autoassociative Memories: A Survey
Introduction. Neural network models of autoassociative, distributed memory
allow storage and retrieval of many items (vectors) where the number of stored
items can exceed the vector dimension (the number of neurons in the network).
This opens the possibility of a sublinear time search (in the number of stored
items) for approximate nearest neighbors among vectors of high dimension. The
purpose of this paper is to review models of autoassociative, distributed
memory that can be naturally implemented by neural networks (mainly with local
learning rules and iterative dynamics based on information locally available to
neurons). Scope. The survey is focused mainly on the networks of Hopfield,
Willshaw and Potts, that have connections between pairs of neurons and operate
on sparse binary vectors. We discuss not only autoassociative memory, but also
the generalization properties of these networks. We also consider neural
networks with higher-order connections and networks with a bipartite graph
structure for non-binary data with linear constraints. Conclusions. In
conclusion we discuss the relations to similarity search, advantages and
drawbacks of these techniques, and topics for further research. An interesting
and still not completely resolved question is whether neural autoassociative
memories can search for approximate nearest neighbors faster than other index
structures for similarity search, in particular for the case of very high
dimensional vectors.Comment: 31 page
Transient dynamics for sequence processing neural networks
An exact solution of the transient dynamics for a sequential associative
memory model is discussed through both the path-integral method and the
statistical neurodynamics. Although the path-integral method has the ability to
give an exact solution of the transient dynamics, only stationary properties
have been discussed for the sequential associative memory. We have succeeded in
deriving an exact macroscopic description of the transient dynamics by
analyzing the correlation of crosstalk noise. Surprisingly, the order parameter
equations of this exact solution are completely equivalent to those of the
statistical neurodynamics, which is an approximation theory that assumes
crosstalk noise to obey the Gaussian distribution. In order to examine our
theoretical findings, we numerically obtain cumulants of the crosstalk noise.
We verify that the third- and fourth-order cumulants are equal to zero, and
that the crosstalk noise is normally distributed even in the non-retrieval
case. We show that the results obtained by our theory agree with those obtained
by computer simulations. We have also found that the macroscopic unstable state
completely coincides with the separatrix.Comment: 21 pages, 4 figure
An associative network with spatially organized connectivity
We investigate the properties of an autoassociative network of
threshold-linear units whose synaptic connectivity is spatially structured and
asymmetric. Since the methods of equilibrium statistical mechanics cannot be
applied to such a network due to the lack of a Hamiltonian, we approach the
problem through a signal-to-noise analysis, that we adapt to spatially
organized networks. The conditions are analyzed for the appearance of stable,
spatially non-uniform profiles of activity with large overlaps with one of the
stored patterns. It is also shown, with simulations and analytic results, that
the storage capacity does not decrease much when the connectivity of the
network becomes short range. In addition, the method used here enables us to
calculate exactly the storage capacity of a randomly connected network with
arbitrary degree of dilution.Comment: 27 pages, 6 figures; Accepted for publication in JSTA
A neural network-based estimator for the mixture ratio of the Space Shuttle Main Engine
In order to properly utilize the available fuel and oxidizer of a liquid propellant rocket engine, the mixture ratio is closed loop controlled during main stage (65 percent - 109 percent power) operation. However, because of the lack of flight-capable instrumentation for measuring mixture ratio, the value of mixture ratio in the control loop is estimated using available sensor measurements such as the combustion chamber pressure and the volumetric flow, and the temperature and pressure at the exit duct on the low pressure fuel pump. This estimation scheme has two limitations. First, the estimation formula is based on an empirical curve fitting which is accurate only within a narrow operating range. Second, the mixture ratio estimate relies on a few sensor measurements and loss of any of these measurements will make the estimate invalid. In this paper, we propose a neural network-based estimator for the mixture ratio of the Space Shuttle Main Engine. The estimator is an extension of a previously developed neural network based sensor failure detection and recovery algorithm (sensor validation). This neural network uses an auto associative structure which utilizes the redundant information of dissimilar sensors to detect inconsistent measurements. Two approaches have been identified for synthesizing mixture ratio from measurement data using a neural network. The first approach uses an auto associative neural network for sensor validation which is modified to include the mixture ratio as an additional output. The second uses a new network for the mixture ratio estimation in addition to the sensor validation network. Although mixture ratio is not directly measured in flight, it is generally available in simulation and in test bed firing data from facility measurements of fuel and oxidizer volumetric flows. The pros and cons of these two approaches will be discussed in terms of robustness to sensor failures and accuracy of the estimate during typical transients using simulation data
Comparing Information-Theoretic Measures of Complexity in Boltzmann Machines
In the past three decades, many theoretical measures of complexity have been
proposed to help understand complex systems. In this work, for the first time,
we place these measures on a level playing field, to explore the qualitative
similarities and differences between them, and their shortcomings.
Specifically, using the Boltzmann machine architecture (a fully connected
recurrent neural network) with uniformly distributed weights as our model of
study, we numerically measure how complexity changes as a function of network
dynamics and network parameters. We apply an extension of one such
information-theoretic measure of complexity to understand incremental Hebbian
learning in Hopfield networks, a fully recurrent architecture model of
autoassociative memory. In the course of Hebbian learning, the total
information flow reflects a natural upward trend in complexity as the network
attempts to learn more and more patterns.Comment: 16 pages, 7 figures; Appears in Entropy, Special Issue "Information
Geometry II
Stochastic transitions of attractors in associative memory models with correlated noise
We investigate dynamics of recurrent neural networks with correlated noise to
analyze the noise's effect. The mechanism of correlated firing has been
analyzed in various models, but its functional roles have not been discussed in
sufficient detail. Aoyagi and Aoki have shown that the state transition of a
network is invoked by synchronous spikes. We introduce two types of noise to
each neuron: thermal independent noise and correlated noise. Due to the effects
of correlated noise, the correlation between neural inputs cannot be ignored,
so the behavior of the network has sample dependence. We discuss two types of
associative memory models: one with auto- and weak cross-correlation
connections and one with hierarchically correlated patterns. The former is
similar in structure to Aoyagi and Aoki's model. We show that stochastic
transition can be presented by correlated rather than thermal noise. In the
latter, we show stochastic transition from a memory state to a mixture state
using correlated noise. To analyze the stochastic transitions, we derive a
macroscopic dynamic description as a recurrence relation form of a probability
density function when the correlated noise exists. Computer simulations agree
with theoretical results.Comment: 21 page
Real-time diagnostics for a reusable rocket engine
A hierarchical, decentralized diagnostic system is proposed for the Real-Time Diagnostic System component of the Intelligent Control System (ICS) for reusable rocket engines. The proposed diagnostic system has three layers of information processing: condition monitoring, fault mode detection, and expert system diagnostics. The condition monitoring layer is the first level of signal processing. Here, important features of the sensor data are extracted. These processed data are then used by the higher level fault mode detection layer to do preliminary diagnosis on potential faults at the component level. Because of the closely coupled nature of the rocket engine propulsion system components, it is expected that a given engine condition may trigger more than one fault mode detector. Expert knowledge is needed to resolve the conflicting reports from the various failure mode detectors. This is the function of the diagnostic expert layer. Here, the heuristic nature of this decision process makes it desirable to use an expert system approach. Implementation of the real-time diagnostic system described above requires a wide spectrum of information processing capability. Generally, in the condition monitoring layer, fast data processing is often needed for feature extraction and signal conditioning. This is usually followed by some detection logic to determine the selected faults on the component level. Three different techniques are used to attack different fault detection problems in the NASA LeRC ICS testbed simulation. The first technique employed is the neural network application for real-time sensor validation which includes failure detection, isolation, and accommodation. The second approach demonstrated is the model-based fault diagnosis system using on-line parameter identification. Besides these model based diagnostic schemes, there are still many failure modes which need to be diagnosed by the heuristic expert knowledge. The heuristic expert knowledge is implemented using a real-time expert system tool called G2 by Gensym Corp. Finally, the distributed diagnostic system requires another level of intelligence to oversee the fault mode reports generated by component fault detectors. The decision making at this level can best be done using a rule-based expert system. This level of expert knowledge is also implemented using G2
- …