Correlations between hidden units in multilayer neural networks and replica symmetry breaking

We consider feed-forward neural networks with one hidden layer, tree architecture and a fixed hidden-to-output Boolean function. Focusing on the saturation limit of the storage problem the influence of replica symmetry breaking on the distribution of local fields at the hidden units is investigated. These field distributions determine the probability for finding a specific activation pattern of the hidden units as well as the corresponding correlation coefficients and therefore quantify the division of labor among the hidden units. We find that although modifying the storage capacity and the distribution of local fields markedly replica symmetry breaking has only a minor effect on the correlation coefficients. Detailed numerical results are provided for the PARITY, COMMITTEE and AND machines with K=3 hidden units and nonoverlapping receptive fields.Comment: 9 pages, 3 figures, RevTex, accepted for publication in Phys. Rev.

Correlation of internal representations in feed-forward neural networks

Feed-forward multilayer neural networks implementing random input-output mappings develop characteristic correlations between the activity of their hidden nodes which are important for the understanding of the storage and generalization performance of the network. It is shown how these correlations can be calculated from the joint probability distribution of the aligning fields at the hidden units for arbitrary decoder function between hidden layer and output. Explicit results are given for the parity-, and-, and committee-machines with arbitrary number of hidden nodes near saturation.Comment: 6 pages, latex, 1 figur

Statistical Mechanics of Learning: A Variational Approach for Real Data

Using a variational technique, we generalize the statistical physics approach of learning from random examples to make it applicable to real data. We demonstrate the validity and relevance of our method by computing approximate estimators for generalization errors that are based on training data alone.Comment: 4 pages, 2 figure

Storage capacity of correlated perceptrons

We consider an ensemble of $K$ single-layer perceptrons exposed to random inputs and investigate the conditions under which the couplings of these perceptrons can be chosen such that prescribed correlations between the outputs occur. A general formalism is introduced using a multi-perceptron costfunction that allows to determine the maximal number of random inputs as a function of the desired values of the correlations. Replica-symmetric results for $K=2$ and $K=3$ are compared with properties of two-layer networks of tree-structure and fixed Boolean function between hidden units and output. The results show which correlations in the hidden layer of multi-layer neural networks are crucial for the value of the storage capacity.Comment: 16 pages, Latex2

Statistical Mechanical Development of a Sparse Bayesian Classifier

The demand for extracting rules from high dimensional real world data is increasing in various fields. However, the possible redundancy of such data sometimes makes it difficult to obtain a good generalization ability for novel samples. To resolve this problem, we provide a scheme that reduces the effective dimensions of data by pruning redundant components for bicategorical classification based on the Bayesian framework. First, the potential of the proposed method is confirmed in ideal situations using the replica method. Unfortunately, performing the scheme exactly is computationally difficult. So, we next develop a tractable approximation algorithm, which turns out to offer nearly optimal performance in ideal cases when the system size is large. Finally, the efficacy of the developed classifier is experimentally examined for a real world problem of colon cancer classification, which shows that the developed method can be practically useful.Comment: 13 pages, 6 figure

Training a perceptron in a discrete weight space

On-line and batch learning of a perceptron in a discrete weight space, where each weight can take $2 L+1$ different values, are examined analytically and numerically. The learning algorithm is based on the training of the continuous perceptron and prediction following the clipped weights. The learning is described by a new set of order parameters, composed of the overlaps between the teacher and the continuous/clipped students. Different scenarios are examined among them on-line learning with discrete/continuous transfer functions and off-line Hebb learning. The generalization error of the clipped weights decays asymptotically as $exp(-K \alpha^2)$/$exp(-e^{|\lambda| \alpha})$ in the case of on-line learning with binary/continuous activation functions, respectively, where $\alpha$ is the number of examples divided by N, the size of the input vector and $K$ is a positive constant that decays linearly with 1/L. For finite $N$ and $L$, a perfect agreement between the discrete student and the teacher is obtained for $\alpha \propto \sqrt{L \ln(NL)}$. A crossover to the generalization error $\propto 1/\alpha$, characterized continuous weights with binary output, is obtained for synaptic depth $L > O(\sqrt{N})$.Comment: 10 pages, 5 figs., submitted to PR

Replica theory for learning curves for Gaussian processes on random graphs

Statistical physics approaches can be used to derive accurate predictions for the performance of inference methods learning from potentially noisy data, as quantified by the learning curve defined as the average error versus number of training examples. We analyse a challenging problem in the area of non-parametric inference where an effectively infinite number of parameters has to be learned, specifically Gaussian process regression. When the inputs are vertices on a random graph and the outputs noisy function values, we show that replica techniques can be used to obtain exact performance predictions in the limit of large graphs. The covariance of the Gaussian process prior is defined by a random walk kernel, the discrete analogue of squared exponential kernels on continuous spaces. Conventionally this kernel is normalised only globally, so that the prior variance can differ between vertices; as a more principled alternative we consider local normalisation, where the prior variance is uniform

Evidence for Alternative Hypotheses

Most researchers want evidence for the direction of an effect, not evidence against a point null hypothesis. Such evidence is ideally on a scale that is easily in- terpretable, with an accompanying standard error. Further, the evidence from iden- tical experiments should be repeatable, and evidence from independent experiments should be easily combined, such as required in meta-analysis. Such a measure of evidence exists and has been shown to be closely related to the Kullback-Leibler symmetrized distance between null and alternative hypotheses for exponential fam- ilies. Here we provide more examples of the latter phenomenon, for distributions ly- ing outside the class of exponential families, including the non-central chi-squared family with unknown non-centrality parameter

Climate change effects on phytoplankton depend on cell size and food web structure

We investigated the effects of warming on a natural phytoplankton community from the Baltic Sea, based on six mesocosm experiments conducted 2005–2009. We focused on differences in the dynamics of three phytoplankton size groups which are grazed to a variable extent by different zooplankton groups. While small-sized algae were mostly grazer-controlled, light and nutrient availability largely determined the growth of medium- and large-sized algae. Thus, the latter groups dominated at increased light levels. Warming increased mesozooplankton grazing on medium-sized algae, reducing their biomass. The biomass of small-sized algae was not affected by temperature, probably due to an interplay between indirect effects spreading through the food web. Thus, under the higher temperature and lower light levels anticipated for the next decades in the southern Baltic Sea, a higher share of smaller phytoplankton is expected. We conclude that considering the size structure of the phytoplankton community strongly improves the reliability of projections of climate change effects

Thiolutin is a zinc chelator that inhibits the Rpn11 and other JAMM metalloproteases

Thiolutin is a disulfide-containing antibiotic and anti-angiogenic compound produced by Streptomyces. Its biological targets are not known. We show that reduced thiolutin is a zinc chelator that inhibits the JAB1/MPN/Mov34 (JAMM) domain–containing metalloprotease Rpn11, a deubiquitinating enzyme of the 19S proteasome. Thiolutin also inhibits the JAMM metalloproteases Csn5, the deneddylase of the COP9 signalosome; AMSH, which regulates ubiquitin-dependent sorting of cell-surface receptors; and BRCC36, a K63-specific deubiquitinase of the BRCC36-containing isopeptidase complex and the BRCA1–BRCA2-containing complex. We provide evidence that other dithiolopyrrolones also function as inhibitors of JAMM metalloproteases