Phase transitions in soft-committee machines

Equilibrium statistical physics is applied to layered neural networks with differentiable activation functions. A first analysis of off-line learning in soft-committee machines with a finite number (K) of hidden units learning a perfectly matching rule is performed. Our results are exact in the limit of high training temperatures. For K=2 we find a second order phase transition from unspecialized to specialized student configurations at a critical size P of the training set, whereas for K > 2 the transition is first order. Monte Carlo simulations indicate that our results are also valid for moderately low temperatures qualitatively. The limit K to infinity can be performed analytically, the transition occurs after presenting on the order of N K examples. However, an unspecialized metastable state persists up to P= O (N K^2).Comment: 8 pages, 4 figure

Generalizing with perceptrons in case of structured phase- and pattern-spaces

We investigate the influence of different kinds of structure on the learning behaviour of a perceptron performing a classification task defined by a teacher rule. The underlying pattern distribution is permitted to have spatial correlations. The prior distribution for the teacher coupling vectors itself is assumed to be nonuniform. Thus classification tasks of quite different difficulty are included. As learning algorithms we discuss Hebbian learning, Gibbs learning, and Bayesian learning with different priors, using methods from statistics and the replica formalism. We find that the Hebb rule is quite sensitive to the structure of the actual learning problem, failing asymptotically in most cases. Contrarily, the behaviour of the more sophisticated methods of Gibbs and Bayes learning is influenced by the spatial correlations only in an intermediate regime of $\alpha$, where $\alpha$ specifies the size of the training set. Concerning the Bayesian case we show, how enhanced prior knowledge improves the performance.Comment: LaTeX, 32 pages with eps-figs, accepted by J Phys

Correlation of internal representations in feed-forward neural networks

Feed-forward multilayer neural networks implementing random input-output mappings develop characteristic correlations between the activity of their hidden nodes which are important for the understanding of the storage and generalization performance of the network. It is shown how these correlations can be calculated from the joint probability distribution of the aligning fields at the hidden units for arbitrary decoder function between hidden layer and output. Explicit results are given for the parity-, and-, and committee-machines with arbitrary number of hidden nodes near saturation.Comment: 6 pages, latex, 1 figur

Motion of four-dimensional rigid body around a fixed point: an elementary approach. I

The goal of this note is to give the explicit solution of Euler-Frahm equations for the Manakov four-dimensional case by elementary means. For this, we use some results from the original papers by Schottky [Sch 1891], Koetter [Koe 1892], Weber [We 1878], and Caspary [Ca 1893]. We hope that such approach will be useful for the solution of the problem of $n$-dimensional top.Comment: LaTeX, 9 page

Wave-packet Formalism of Full Counting Statistics

We make use of the first-quantized wave-packet formulation of the full counting statistics to describe charge transport of noninteracting electrons in a mesoscopic device. We derive various expressions for the characteristic function generating the full counting statistics, accounting for both energy and time dependence in the scattering process and including exchange effects due to finite overlap of the incoming wave packets. We apply our results to describe the generic statistical properties of a two-fermion scattering event and find, among other features, sub-binomial statistics for nonentangled incoming states (Slater rank 1), while entangled states (Slater rank 2) may generate super-binomial (and even super-Poissonian) noise, a feature that can be used as a spin singlet-triplet detector. Another application is concerned with the constant-voltage case, where we generalize the original result of Levitov-Lesovik to account for energy-dependent scattering and finite measurement time, including short time measurements, where Pauli blocking becomes important.Comment: 20 pages, 5 figures; major update, new figures and explanations included as well as a discussion about finite temperatures and subleading logarithmic term

AC conductance and non-symmetrized noise at finite frequency in quantum wires and carbon nanotubes

We calculate the AC conductance and the finite-frequency non-symmetrized noise in interacting quantum wires and single-wall carbon nanotubes in the presence of an impurity. We observe a strong asymmetry in the frequency spectrum of the non-symmetrized excess noise, even in the presence of the metallic leads. We find that this asymmetry is proportional to the differential excess AC conductance of the system, defined as the difference between the AC differential conductances at finite and zero voltage, and thus disappears for a linear system. In the quantum regime, for temperatures much smaller than the frequency and the applied voltage, we find that the emission noise is exactly equal to the impurity partition noise. For the case of a weak impurity we expand our results for the AC conductance and the noise perturbatively. In particular, if the impurity is located in the middle of the wire or at one of the contacts, our calculations show that the noise exhibits oscillations with respect to frequency, whose period is directly related to the value of the interaction parameter $g$

Excess Noise in Biased Superconducting Weak Links

Non-equilibrium excess noise of a short quasi one-dimensional constriction between two superconductors is considered. A general expression for the current-current correlation function valid for arbitrary temperatures and bias voltages is derived. This formalism is applied to a current-carrying quantum channel with perfect transparency. Contrary to a transparent channel separating two normal conductors, a weak link between two superconductors exhibits a finite level of noise. The source of noise is fractional Andreev scattering of quasiparticles with energies $|E|$ greater than the half-width $\Delta$ of the superconducting gap. For high bias voltages, $V \gg \Delta /e$, the relation between the zero-frequency limit of the noise spectrum, $S(0)$, and the excess current $I_{\text{exc}}$ reads $S(0)=(1/5)|e|I_{\text{exc}}$. As $\Delta \rightarrow 0$ both the excess noise and the excess current vanish linearly in $\Delta$, %$\propto \Delta$, their ratio being constant.Comment: 8 pages (Latex), 1 figur

Correlations between hidden units in multilayer neural networks and replica symmetry breaking

We consider feed-forward neural networks with one hidden layer, tree architecture and a fixed hidden-to-output Boolean function. Focusing on the saturation limit of the storage problem the influence of replica symmetry breaking on the distribution of local fields at the hidden units is investigated. These field distributions determine the probability for finding a specific activation pattern of the hidden units as well as the corresponding correlation coefficients and therefore quantify the division of labor among the hidden units. We find that although modifying the storage capacity and the distribution of local fields markedly replica symmetry breaking has only a minor effect on the correlation coefficients. Detailed numerical results are provided for the PARITY, COMMITTEE and AND machines with K=3 hidden units and nonoverlapping receptive fields.Comment: 9 pages, 3 figures, RevTex, accepted for publication in Phys. Rev.

Retarded Learning: Rigorous Results from Statistical Mechanics

We study learning of probability distributions characterized by an unknown symmetry direction. Based on an entropic performance measure and the variational method of statistical mechanics we develop exact upper and lower bounds on the scaled critical number of examples below which learning of the direction is impossible. The asymptotic tightness of the bounds suggests an asymptotically optimal method for learning nonsmooth distributions.Comment: 8 pages, 1 figur