Learning Curves for Mutual Information Maximization

An unsupervised learning procedure based on maximizing the mutual information between the outputs of two networks receiving different but statistically dependent inputs is analyzed (Becker and Hinton, Nature, 355, 92, 161). For a generic data model, I show that in the large sample limit the structure in the data is recognized by mutual information maximization. For a more restricted model, where the networks are similar to perceptrons, I calculate the learning curves for zero-temperature Gibbs learning. These show that convergence can be rather slow, and a way of regularizing the procedure is considered.Comment: 13 pages, to appear in Phys.Rev.

Comment on "Finite size scaling in Neural Networks"

We use a binary search tree and the simplex algorithm to measure the fraction of patterns that can be stored by an Ising perceptron. The algorithm is much faster than exhaustive search and allows us to obtain accurate statistics up to a system size of N=42. The results show that the finite size scaling ansatz Nadler and Fink suggest in [1] cannot be applied to estimate accurately the storage capacity from small systems. [1] W.Nadler and W.Fink: Phys.Rev.Lett. 78, 555 (1997)Comment: LaTeX with 1 postscript figure, using REVTe

Noisy regression and classification with continuous multilayer networks

We investigate zero temperature Gibbs learning for two classes of unrealizable rules which play an important role in practical applications of multilayer neural networks with differentiable activation functions: classification problems and noisy regression problems. Considering one step of replica symmetry breaking, we surprisingly find that for sufficiently large training sets the stable state is replica symmetric even though the target rule is unrealizable. Further, the classification problem is shown to be formally equivalent to the noisy regression problem.Comment: 7 pages, including 2 figure

An improved algorithm for stoichiometric network analysis: theory and applications

Motivation: Genome scale analysis of the metabolic network of a microorganism is a major challenge in bioinformatics. The combinatorial explosion, which occurs during the construction of elementary fluxes (non-redundant pathways) requires sophisticated and efficient algorithms to tackle the problem. Results: Mathematically, the calculation of elementary fluxes amounts to characterizing the space of solutions to a mixed system of linear equalities, given by the stoichiometry matrix, and linear inequalities, arising from the irreversibility of some or all of the reactions in the network. Previous approaches to this problem have iteratively solved for the equalities while satisfying the inequalities throughout the process. In an extension of previous work, here we consider the complementary approach and derive an algorithm which satisfies the inequalities one by one while staying in the space of solution of the equality constraints. Benchmarks on different subnetworks of the central carbon metabolism of Escherichia coli show that this new approach yields a significant reduction in the execution time of the calculation. This reduction arises since the odds that an intermediate elementary flux already fulfills an additional inequality are larger than when having to satisfy an additional equality constraint. Availability: The code is available upon request. Supplementary information: Pseudo code and a Mathematica implementation of the algorithm is on the OUP server. Contact: [email protected]; [email protected]

Functional stoichiometric analysis of metabolic networks

Motivation: An important tool in Systems Biology is the stoichiometric modeling of metabolic networks, where the stationary states of the network are described by a high-dimensional polyhedral cone, the so-called flux cone. Exhaustive descriptions of the metabolism can be obtained by computing the elementary vectors of this cone but, owing to a combinatorial explosion of the number of elementary vectors, this approach becomes computationally intractable for genome scale networks. Result: Hence, we propose to instead focus on the conversion cone, a projection of the flux cone, which describes the interaction of the metabolism with its external chemical environment. We present a direct method for calculating the elementary vectors of this cone and, by studying the metabolism of Saccharomyces cerevisiae, we demonstrate that such an analysis is computationally feasible even for genome scale networks. Contact: [email protected]

Statistical mechanics of mutual information maximization

An unsupervised learning procedure based on maximizing the mutual information between the outputs of two networks receiving different but statistically dependent inputs is analyzed (Becker S. and Hinton G., Nature, 355 (1992) 161). By exploiting a formal analogy to supervised learning in parity machines, the theory of zero-temperature Gibbs learning for the unsupervised procedure is presented for the case that the networks are perceptrons and for the case of fully connected committees

Statistical physics of independent component analysis

Statistical physics is used to investigate independent component analysis with polynomial contrast functions. While the replica method fails, an adapted cavity approach yields valid results. The learning curves, obtained in a suitable thermodynamic limit, display a first order phase transition from poor to perfect generalization.Comment: 7 pages, 1 figure, to appear in Europhys. Lett

Phase transitions in soft-committee machines

Equilibrium statistical physics is applied to layered neural networks with differentiable activation functions. A first analysis of off-line learning in soft-committee machines with a finite number (K) of hidden units learning a perfectly matching rule is performed. Our results are exact in the limit of high training temperatures. For K=2 we find a second order phase transition from unspecialized to specialized student configurations at a critical size P of the training set, whereas for K > 2 the transition is first order. Monte Carlo simulations indicate that our results are also valid for moderately low temperatures qualitatively. The limit K to infinity can be performed analytically, the transition occurs after presenting on the order of N K examples. However, an unspecialized metastable state persists up to P= O (N K^2).Comment: 8 pages, 4 figure