120 research outputs found
Learning Curves for Mutual Information Maximization
An unsupervised learning procedure based on maximizing the mutual information
between the outputs of two networks receiving different but statistically
dependent inputs is analyzed (Becker and Hinton, Nature, 355, 92, 161). For a
generic data model, I show that in the large sample limit the structure in the
data is recognized by mutual information maximization. For a more restricted
model, where the networks are similar to perceptrons, I calculate the learning
curves for zero-temperature Gibbs learning. These show that convergence can be
rather slow, and a way of regularizing the procedure is considered.Comment: 13 pages, to appear in Phys.Rev.
Comment on "Finite size scaling in Neural Networks"
We use a binary search tree and the simplex algorithm to measure the fraction
of patterns that can be stored by an Ising perceptron. The algorithm is much
faster than exhaustive search and allows us to obtain accurate statistics up to
a system size of N=42. The results show that the finite size scaling ansatz
Nadler and Fink suggest in [1] cannot be applied to estimate accurately the
storage capacity from small systems.
[1] W.Nadler and W.Fink: Phys.Rev.Lett. 78, 555 (1997)Comment: LaTeX with 1 postscript figure, using REVTe
Noisy regression and classification with continuous multilayer networks
We investigate zero temperature Gibbs learning for two classes of
unrealizable rules which play an important role in practical applications of
multilayer neural networks with differentiable activation functions:
classification problems and noisy regression problems. Considering one step of
replica symmetry breaking, we surprisingly find that for sufficiently large
training sets the stable state is replica symmetric even though the target rule
is unrealizable. Further, the classification problem is shown to be formally
equivalent to the noisy regression problem.Comment: 7 pages, including 2 figure
An improved algorithm for stoichiometric network analysis: theory and applications
Motivation: Genome scale analysis of the metabolic network of a microorganism is a major challenge in bioinformatics. The combinatorial explosion, which occurs during the construction of elementary fluxes (non-redundant pathways) requires sophisticated and efficient algorithms to tackle the problem. Results: Mathematically, the calculation of elementary fluxes amounts to characterizing the space of solutions to a mixed system of linear equalities, given by the stoichiometry matrix, and linear inequalities, arising from the irreversibility of some or all of the reactions in the network. Previous approaches to this problem have iteratively solved for the equalities while satisfying the inequalities throughout the process. In an extension of previous work, here we consider the complementary approach and derive an algorithm which satisfies the inequalities one by one while staying in the space of solution of the equality constraints. Benchmarks on different subnetworks of the central carbon metabolism of Escherichia coli show that this new approach yields a significant reduction in the execution time of the calculation. This reduction arises since the odds that an intermediate elementary flux already fulfills an additional inequality are larger than when having to satisfy an additional equality constraint. Availability: The code is available upon request. Supplementary information: Pseudo code and a Mathematica implementation of the algorithm is on the OUP server. Contact: [email protected]; [email protected]
Functional stoichiometric analysis of metabolic networks
Motivation: An important tool in Systems Biology is the stoichiometric modeling of metabolic networks, where the stationary states of the network are described by a high-dimensional polyhedral cone, the so-called flux cone. Exhaustive descriptions of the metabolism can be obtained by computing the elementary vectors of this cone but, owing to a combinatorial explosion of the number of elementary vectors, this approach becomes computationally intractable for genome scale networks. Result: Hence, we propose to instead focus on the conversion cone, a projection of the flux cone, which describes the interaction of the metabolism with its external chemical environment. We present a direct method for calculating the elementary vectors of this cone and, by studying the metabolism of Saccharomyces cerevisiae, we demonstrate that such an analysis is computationally feasible even for genome scale networks. Contact: [email protected]
Statistical mechanics of mutual information maximization
An unsupervised learning procedure based on maximizing the mutual information between the outputs of two networks receiving different but statistically dependent inputs is analyzed (Becker S. and Hinton G., Nature, 355 (1992) 161). By exploiting a formal analogy to supervised learning in parity machines, the theory of zero-temperature Gibbs learning for the unsupervised procedure is presented for the case that the networks are perceptrons and for the case of fully connected committees
Statistical physics of independent component analysis
Statistical physics is used to investigate independent component analysis
with polynomial contrast functions. While the replica method fails, an adapted
cavity approach yields valid results. The learning curves, obtained in a
suitable thermodynamic limit, display a first order phase transition from poor
to perfect generalization.Comment: 7 pages, 1 figure, to appear in Europhys. Lett
Phase transitions in soft-committee machines
Equilibrium statistical physics is applied to layered neural networks with
differentiable activation functions. A first analysis of off-line learning in
soft-committee machines with a finite number (K) of hidden units learning a
perfectly matching rule is performed. Our results are exact in the limit of
high training temperatures. For K=2 we find a second order phase transition
from unspecialized to specialized student configurations at a critical size P
of the training set, whereas for K > 2 the transition is first order. Monte
Carlo simulations indicate that our results are also valid for moderately low
temperatures qualitatively. The limit K to infinity can be performed
analytically, the transition occurs after presenting on the order of N K
examples. However, an unspecialized metastable state persists up to P= O (N
K^2).Comment: 8 pages, 4 figure
Online Learning with Ensembles
Supervised online learning with an ensemble of students randomized by the
choice of initial conditions is analyzed. For the case of the perceptron
learning rule, asymptotically the same improvement in the generalization error
of the ensemble compared to the performance of a single student is found as in
Gibbs learning. For more optimized learning rules, however, using an ensemble
yields no improvement. This is explained by showing that for any learning rule
a transform exists, such that a single student using
has the same generalization behaviour as an ensemble of
-students.Comment: 8 pages, 1 figure. Submitted to J.Phys.
- …