A layered neural network with three-state neurons optimizing the mutual information

The time evolution of an exactly solvable layered feedforward neural network with three-state neurons and optimizing the mutual information is studied for arbitrary synaptic noise (temperature). Detailed stationary temperature-capacity and capacity-activity phase diagrams are obtained. The model exhibits pattern retrieval, pattern-fluctuation retrieval and spin-glass phases. It is found that there is an improved performance in the form of both a larger critical capacity and information content compared with three-state Ising-type layered network models. Flow diagrams reveal that saddle-point solutions associated with fluctuation overlaps slow down considerably the flow of the network states towards the stable fixed-points.Comment: 17 pages Latex including 6 eps-figure

Synchronous versus sequential updating in the three-state Ising neural network with variable dilution

The three-state Ising neural network with synchronous updating and variable dilution is discussed starting from the appropriate Hamiltonians. The thermodynamic and retrieval properties are examined using replica mean-field theory. Capacity-temperature phase diagrams are derived for several values of the pattern activity and different gradations of dilution, and the information content is calculated. The results are compared with those for sequential updating. The effect of self-coupling is established. Also the dynamics is studied using the generating function technique for both synchronous and sequential updating. Typical flow diagrams for the overlap order parameter are presented. The differences with the signal-to-noise approach are outlined.Comment: 21 pages Latex, 12 eps figures and 1 ps figur

Correlated patterns in non-monotonic graded-response perceptrons

The optimal capacity of graded-response perceptrons storing biased and spatially correlated patterns with non-monotonic input-output relations is studied. It is shown that only the structure of the output patterns is important for the overall performance of the perceptrons.Comment: 4 pages, 4 figure

Retrieval behavior and thermodynamic properties of symmetrically diluted Q-Ising neural networks

The retrieval behavior and thermodynamic properties of symmetrically diluted Q-Ising neural networks are derived and studied in replica-symmetric mean-field theory generalizing earlier works on either the fully connected or the symmetrical extremely diluted network. Capacity-gain parameter phase diagrams are obtained for the Q=3, Q=4 and $Q=\infty$ state networks with uniformly distributed patterns of low activity in order to search for the effects of a gradual dilution of the synapses. It is shown that enlarged regions of continuous changeover into a region of optimal performance are obtained for finite stochastic noise and small but finite connectivity. The de Almeida-Thouless lines of stability are obtained for arbitrary connectivity, and the resulting phase diagrams are used to draw conclusions on the behavior of symmetrically diluted networks with other pattern distributions of either high or low activity.Comment: 21 pages, revte

Thresholds in layered neural networks with variable activity

The inclusion of a threshold in the dynamics of layered neural networks with variable activity is studied at arbitrary temperature. In particular, the effects on the retrieval quality of a self-controlled threshold obtained by forcing the neural activity to stay equal to the activity of the stored paterns during the whole retrieval process, are compared with those of a threshold chosen externally for every loading and every temperature through optimisation of the mutual information content of the network. Numerical results, mostly concerning low activity networks are discussed.Comment: 15 pages, Latex2e, 6 eps figure

Statics and dynamics of an Ashkin-Teller neural network with low loading

An Ashkin-Teller neural network, allowing for two types of neurons is considered in the case of low loading as a function of the strength of the respective couplings between these neurons. The storage and retrieval of embedded patterns built from the two types of neurons, with different degrees of (in)dependence is studied. In particular, thermodynamic properties including the existence and stability of Mattis states are discussed. Furthermore, the dynamic behaviour is examined by deriving flow equations for the macroscopic overlap. It is found that for linked patterns the model shows better retrieval properties than a corresponding Hopfield model.Comment: 20 pages, 6 figures, Latex with postscript figures in one tar.gz fil

Thermodynamic properties of extremely diluted symmetric Q-Ising neural networks

Using the replica-symmetric mean-field theory approach the thermodynamic and retrieval properties of extremely diluted {\it symmetric} $Q$-Ising neural networks are studied. In particular, capacity-gain parameter and capacity-temperature phase diagrams are derived for $Q=3, 4$ and $Q=\infty$. The zero-temperature results are compared with those obtained from a study of the dynamics of the model. Furthermore, the de Almeida-Thouless line is determined. Where appropriate, the difference with other $Q$-Ising architectures is outlined.Comment: 16 pages Latex including 6 eps-figures. Corrections, also in most of the figures have been mad

A canonical ensemble approach to graded-response perceptrons

Perceptrons with graded input-output relations and a limited output precision are studied within the Gardner-Derrida canonical ensemble approach. Soft non- negative error measures are introduced allowing for extended retrieval properties. In particular, the performance of these systems for a linear and quadratic error measure, corresponding to the perceptron respectively the adaline learning algorithm, is compared with the performance for a rigid error measure, simply counting the number of errors. Replica-symmetry-breaking effects are evaluated.Comment: 26 pages, 10 ps figure

The signal-to-noise analysis of the Little-Hopfield model revisited

Using the generating functional analysis an exact recursion relation is derived for the time evolution of the effective local field of the fully connected Little-Hopfield model. It is shown that, by leaving out the feedback correlations arising from earlier times in this effective dynamics, one precisely finds the recursion relations usually employed in the signal-to-noise approach. The consequences of this approximation as well as the physics behind it are discussed. In particular, it is pointed out why it is hard to notice the effects, especially for model parameters corresponding to retrieval. Numerical simulations confirm these findings. The signal-to-noise analysis is then extended to include all correlations, making it a full theory for dynamics at the level of the generating functional analysis. The results are applied to the frequently employed extremely diluted (a)symmetric architectures and to sequence processing networks.Comment: 26 pages, 3 figure

Small-world hypergraphs on a bond-disordered Bethe lattice

We study the thermodynamic properties of spin systems with bond-disorder on small-world hypergraphs, obtained by superimposing a one-dimensional Ising chain onto a random Bethe graph with p-spin interactions. Using transfer-matrix techniques, we derive fixed-point equations describing the relevant order parameters and the free energy, both in the replica symmetric and one step replica symmetry breaking approximation. We determine the static and dynamic ferromagnetic transition and the spinglass transition within replica symmetry for all temperatures, and demonstrate corrections to these results when one step replica symmetry breaking is taken into account. The results obtained are in agreement with Monte-Carlo simulations.Comment: 9 pages, 4 figure