Optimisation of on-line principal component analysis

Different techniques, used to optimise on-line principal component analysis, are investigated by methods of statistical mechanics. These include local and global optimisation of node-dependent learning-rates which are shown to be very efficient in speeding up the learning process. They are investigated further for gaining insight into the learning rates' time-dependence, which is then employed for devising simple practical methods to improve training performance. Simulations demonstrate the benefit gained from using the new methods.Comment: 10 pages, 5 figure

Towards Designing Artificial Universes for Artificial Agents under Interaction Closure

We are interested in designing artificial universes for artificial agents. We view artificial agents as networks of highlevel processes on top of of a low-level detailed-description system. We require that the high-level processes have some intrinsic explanatory power and we introduce an extension of informational closure namely interaction closure to capture this. Then we derive a method to design artificial universes in the form of finite Markov chains which exhibit high-level processes that satisfy the property of interaction closure. We also investigate control or information transfer which we see as an building block for networks representing artificial agent

Functional Optimisation of Online Algorithms in Multilayer Neural Networks

We study the online dynamics of learning in fully connected soft committee machines in the student-teacher scenario. The locally optimal modulation function, which determines the learning algorithm, is obtained from a variational argument in such a manner as to maximise the average generalisation error decay per example. Simulations results for the resulting algorithm are presented for a few cases. The symmetric phase plateaux are found to be vastly reduced in comparison to those found when online backpropagation algorithms are used. A discussion of the implementation of these ideas as practical algorithms is given

Phase transitions in soft-committee machines

Equilibrium statistical physics is applied to layered neural networks with differentiable activation functions. A first analysis of off-line learning in soft-committee machines with a finite number (K) of hidden units learning a perfectly matching rule is performed. Our results are exact in the limit of high training temperatures. For K=2 we find a second order phase transition from unspecialized to specialized student configurations at a critical size P of the training set, whereas for K > 2 the transition is first order. Monte Carlo simulations indicate that our results are also valid for moderately low temperatures qualitatively. The limit K to infinity can be performed analytically, the transition occurs after presenting on the order of N K examples. However, an unspecialized metastable state persists up to P= O (N K^2).Comment: 8 pages, 4 figure

Analysis of dropout learning regarded as ensemble learning

Deep learning is the state-of-the-art in fields such as visual object recognition and speech recognition. This learning uses a large number of layers, huge number of units, and connections. Therefore, overfitting is a serious problem. To avoid this problem, dropout learning is proposed. Dropout learning neglects some inputs and hidden units in the learning process with a probability, p, and then, the neglected inputs and hidden units are combined with the learned network to express the final output. We find that the process of combining the neglected hidden units with the learned network can be regarded as ensemble learning, so we analyze dropout learning from this point of view.Comment: 9 pages, 8 figures, submitted to Conferenc

Finite-size effects in on-line learning of multilayer neural networks

We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating fluctuations possessed by finite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time

Differentiable Kernels in Generalized Matrix Learning Vector Quantization

In the present paper we investigate the application of differentiable kernel for generalized matrix learning vector quantization as an alternative kernel-based classifier, which additionally provides classification dependent data visualization. We show that the concept of differentiable kernels allows a prototype description in the data space but equipped with the kernel metric. Moreover, using the visualization properties of the original matrix learning vector quantization we are able to optimize the class visualization by inherent visualization mapping learning also in this new kernel-metric data space

Analysis of 101 nuclear transcriptomes reveals 23 distinct regulons and their relationship to metabolism, chromosomal gene distribution and co-ordination of nuclear and plastid gene expression

Post-endosymbiotic evolution of the proto-chloroplast was characterized by gene transfer to the nucleus. Hence, most chloroplast proteins are nuclear-encoded and the regulation of chloroplast functions includes nuclear transcriptional control. The expression profiles of 3292 nuclear Arabidopsis genes, most of them encoding chloroplast proteins, were determined from 101 different conditions and have been deposited at the GEO database (http://www.ncbi.nlm.nih.gov/geo/) under GSE1160-GSE1260. The 1590 most-regulated genes fell into 23 distinct groups of co-regulated genes (regulons). Genes of some regulons are not evenly distributed among the five Arabidopsis chromosomes and pairs of adjacent, co-expressed genes exist. Except regulons 1 and 2, regulons are heterogeneous and consist of genes coding for proteins with different subcellular locations or contributing to several biochemical functions. This implies that different organelles and/or metabolic pathways are co-ordinated at the nuclear transcriptional level, and a prototype for this is regulon 12 which contains genes with functions in amino acid and carbohydrate metabolism, as well as genes associated with transport or transcription. The co-expression of nuclear genes coding for subunits of the photosystems or encoding proteins involved in the transcription/translation of plastome genes (particularly ribosome polypeptides) (regulons 1 and 2, respectively) implies the existence of a novel mechanism that co-ordinates plastid and nuclear gene expression and involves nuclear control of plastid ribosome abundance. The co-regulation of genes for photosystem and plastid ribosome proteins escapes a previously described general control of nuclear chloroplast proteins imposed by a transcriptional master switch, highlighting a mode of transcriptional regulation of photosynthesis which is different compared to other chloroplast functions. From the evolutionary standpoint, the results provided indicate that functional integration of the proto-chloroplast into the eukaryotic cell was associated with the establishment of different layers of nuclear transcriptional control