On-Line Learning with Restricted Training Sets: An Exactly Solvable Case

We solve the dynamics of on-line Hebbian learning in large perceptrons exactly, for the regime where the size of the training set scales linearly with the number of inputs. We consider both noiseless and noisy teachers. Our calculation cannot be extended to non-Hebbian rules, but the solution provides a convenient and welcome benchmark with which to test more general and advanced theories for solving the dynamics of learning with restricted training sets.Comment: 19 pages, eps figures included, uses epsfig macr

Dynamics of Learning with Restricted Training Sets I: General Theory

We study the dynamics of supervised learning in layered neural networks, in the regime where the size $p$ of the training set is proportional to the number $N$ of inputs. Here the local fields are no longer described by Gaussian probability distributions and the learning dynamics is of a spin-glass nature, with the composition of the training set playing the role of quenched disorder. We show how dynamical replica theory can be used to predict the evolution of macroscopic observables, including the two relevant performance measures (training error and generalization error), incorporating the old formalism developed for complete training sets in the limit $\alpha=p/N\to\infty$ as a special case. For simplicity we restrict ourselves in this paper to single-layer networks and realizable tasks.Comment: 39 pages, LaTe

Dynamics of on-line Hebbian learning with structurally unrealizable restricted training sets

We present an exact solution for the dynamics of on-line Hebbian learning in neural networks, with restricted and unrealizable training sets. In contrast to other studies on learning with restricted training sets, unrealizability is here caused by structural mismatch, rather than data noise: the teacher machine is a perceptron with a reversed wedge-type transfer function, while the student machine is a perceptron with a sigmoidal transfer function. We calculate the glassy dynamics of the macroscopic performance measures, training error and generalization error, and the (non-Gaussian) student field distribution. Our results, which find excellent confirmation in numerical simulations, provide a new benchmark test for general formalisms with which to study unrealizable learning processes with restricted training sets.Comment: 7 pages including 3 figures, using IOP latex2e preprint class fil

Discovery of a very X-ray luminous galaxy cluster at z=0.89 in the WARPS survey

We report the discovery of the galaxy cluster ClJ1226.9+3332 in the Wide Angle ROSAT Pointed Survey (WARPS). At z=0.888 and L_X=1.1e45 erg/s (0.5-2.0 keV, h_0=0.5) ClJ1226.9+3332 is the most distant X-ray luminous cluster currently known. The mere existence of this system represents a huge problem for Omega_0=1 world models. At the modest (off-axis) resolution of the ROSAT PSPC observation in which the system was detected, ClJ1226.9+3332 appears relaxed; an off-axis HRI observation confirms this impression and rules out significant contamination from point sources. However, in moderately deep optical images (R and I band) the cluster exhibits signs of substructure in its apparent galaxy distribution. A first crude estimate of the velocity dispersion of the cluster galaxies based on six redshifts yields a high value of 1650 km/s, indicative of a very massive cluster and/or the presence of substructure along the line of sight. While a more accurate assessment of the dynamical state of this system requires much better data at both optical and X-ray wavelengths, the high mass of the cluster has already been unambiguously confirmed by a very strong detection of the Sunyaev-Zel'dovich effect in its direction (Joy et al. 2001). Using ClJ1226.9+3332 and ClJ0152.7-1357 (z=0.835), the second-most distant X-ray luminous cluster currently known and also a WARPS discovery, we obtain a first estimate of the cluster X-ray luminosity function at 0.8<z<1.4 and L_X>5e44 erg/s. Using the best currently available data, we find the comoving space density of very distant, massive clusters to be in excellent agreement with the value measured locally (z<0.3), and conclude that negative evolution is not required at these luminosities out to z~1. (truncated)Comment: accepted for publication in ApJ Letters, 6 pages, 2 figures, uses emulateapj.st

The WARPS survey - IV: The X-ray luminosity-temperature relation of high redshift galaxy clusters

We present a measurement of the cluster X-ray luminosity-temperature relation out to high redshift (z~0.8). Combined ROSAT PSPC spectra of 91 galaxy clusters detected in the Wide Angle ROSAT Pointed Survey (WARPS) are simultaneously fit in redshift and luminosity bins. The resulting temperature and luminosity measurements of these bins, which occupy a region of the high redshift L-T relation not previously sampled, are compared to existing measurements at low redshift in order to constrain the evolution of the L-T relation. We find a best fit to low redshift (z1 keV, to be L proportional to T^(3.15\pm0.06). Our data are consistent with no evolution in the normalisation of the L-T relation up to z~0.8. Combining our results with ASCA measurements taken from the literature, we find eta=0.19\pm0.38 (for Omega_0=1, with 1 sigma errors) where L_Bol is proportional to (1 + z)^eta T^3.15, or eta=0.60\pm0.38 for Omega_0=0.3. This lack of evolution is considered in terms of the entropy-driven evolution of clusters. Further implications for cosmological constraints are also discussed.Comment: 11 pages, 7 figures, accepted for publication in MNRA

Order-Parameter Flow in the SK Spin-Glass II: Inclusion of Microscopic Memory Effects

We develop further a recent dynamical replica theory to describe the dynamics of the Sherrington-Kirkpatrick spin-glass in terms of closed evolution equations for macroscopic order parameters. We show how microscopic memory effects can be included in the formalism through the introduction of a dynamic order parameter function: the joint spin-field distribution. The resulting formalism describes very accurately the relaxation phenomena observed in numerical simulations, including the typical overall slowing down of the flow that was missed by the previous simple two-parameter theory. The advanced dynamical replica theory is either exact or a very good approximation.Comment: same as original, but this one is TeXabl

Spin-glass model with partially annealed asymmetric bonds

We have considered the two-spin interaction spherical spin-glass model with asymmetric bonds (coupling constants). Besides the usual interactions between spins and bonds and between the spins and a thermostat with temperature $T_{\sigma}$ there is also an additional factor: the bonds are not assumed random {\it a priori} but interact with some other thermostat at the temperature $T_{J}$. We show that when the bonds are frozen with respect to the spins a first order phase transition to a spin-glass phase occurs, and the temperature of this transition tends to zero if $T_J$ is large. Our analytical results show that a spin-glass phase can exist in mean-field models with nonrelaxational dynamics.Comment: 10 pages, late

Application of two-parameter dynamical replica theory to retrieval dynamics of associative memory with non-monotonic neurons

The two-parameter dynamical replica theory (2-DRT) is applied to investigate retrieval properties of non-monotonic associative memory, a model which lacks thermodynamic potential functions. 2-DRT reproduces dynamical properties of the model quite well, including the capacity and basin of attraction. Superretrieval state is also discussed in the framework of 2-DRT. The local stability condition of the superretrieval state is given, which provides a better estimate of the region in which superretrieval is observed experimentally than the self-consistent signal-to-noise analysis (SCSNA) does.Comment: 16 pages, 19 postscript figure