Dynamics of Supervised Learning with Restricted Training Sets

We study the dynamics of supervised learning in layered neural networks, in the regime where the size $p$ of the training set is proportional to the number $N$ of inputs. Here the local fields are no longer described by Gaussian probability distributions. We show how dynamical replica theory can be used to predict the evolution of macroscopic observables, including the relevant performance measures, incorporating the old formalism in the limit $\alpha=p/N\to\infty$ as a special case. For simplicity we restrict ourselves to single-layer networks and realizable tasks.Comment: 36 pages, latex2e, 12 eps figures (to be publ in: Proc Newton Inst Workshop on On-Line Learning '97

Superfluidity of Dense 4^4He in Vycor

We calculate properties of a model of $^4$He in Vycor using the Path Integral Monte Carlo method. We find that $^4$He forms a distinct layered structure with a highly localized first layer, a disordered second layer with some atoms delocalized and able to give rise to the observed superfluid response, and higher layers nearly perfect crystals. The addition of a single $^3$He atom was enough to bring down the total superfluidity by blocking the exchange in the second layer. Our results are consistent with the persistent liquid layer model to explain the observations. Such a model may be relevant to the experiments on bulk solid $^4$He, if there is a fine network of grain boundaries in those systems.Comment: 4 pages, 4 figure

Dynamics of Learning with Restricted Training Sets I: General Theory

We study the dynamics of supervised learning in layered neural networks, in the regime where the size $p$ of the training set is proportional to the number $N$ of inputs. Here the local fields are no longer described by Gaussian probability distributions and the learning dynamics is of a spin-glass nature, with the composition of the training set playing the role of quenched disorder. We show how dynamical replica theory can be used to predict the evolution of macroscopic observables, including the two relevant performance measures (training error and generalization error), incorporating the old formalism developed for complete training sets in the limit $\alpha=p/N\to\infty$ as a special case. For simplicity we restrict ourselves in this paper to single-layer networks and realizable tasks.Comment: 39 pages, LaTe

Solutions for certain classes of Riccati differential equation

We derive some analytic closed-form solutions for a class of Riccati equation y'(x)-\lambda_0(x)y(x)\pm y^2(x)=\pm s_0(x), where \lambda_0(x), s_0(x) are C^{\infty}-functions. We show that if \delta_n=\lambda_n s_{n-1}-\lambda_{n-1}s_n=0, where \lambda_{n}= \lambda_{n-1}^\prime+s_{n-1}+\lambda_0\lambda_{n-1} and s_{n}=s_{n-1}^\prime+s_0\lambda_{k-1}, n=1,2,..., then The Riccati equation has a solution given by y(x)=\mp s_{n-1}(x)/\lambda_{n-1}(x). Extension to the generalized Riccati equation y'(x)+P(x)y(x)+Q(x)y^2(x)=R(x) is also investigated.Comment: 10 page

Effective photon mass and exact translating quantum relativistic structures

Using a variation of the celebrated Volkov solution, the Klein-Gordon equation for a charged particle is reduced to a set of ordinary differential equations, exactly solvable in specific cases. The new quantum relativistic structures can reveal a localization in the radial direction perpendicular to the wave packet propagation, thanks to a non-vanishing scalar potential. The external electromagnetic field, the particle current density and the charge density are determined. The stability analysis of the solutions is performed by means of numerical simulations. The results are useful for the description of a charged quantum test particle in the relativistic regime, provided spin effects are not decisive

Analysis of dropout learning regarded as ensemble learning

Deep learning is the state-of-the-art in fields such as visual object recognition and speech recognition. This learning uses a large number of layers, huge number of units, and connections. Therefore, overfitting is a serious problem. To avoid this problem, dropout learning is proposed. Dropout learning neglects some inputs and hidden units in the learning process with a probability, p, and then, the neglected inputs and hidden units are combined with the learned network to express the final output. We find that the process of combining the neglected hidden units with the learned network can be regarded as ensemble learning, so we analyze dropout learning from this point of view.Comment: 9 pages, 8 figures, submitted to Conferenc

Exhibiting cross-diffusion-induced patterns for reaction-diffusion systems on evolving domains and surfaces

The aim of this manuscript is to present for the first time the application of the finite element method for solving reaction-diffusion systems with cross-diffusion on continuously evolving domains and surfaces. Furthermore we present pattern formation generated by the reaction-diffusion systemwith cross-diffusion on evolving domains and surfaces. A two-component reaction-diffusion system with linear cross-diffusion in both u and v is presented. The finite element method is based on the approximation of the domain or surface by a triangulated domain or surface consisting of a union of triangles. For surfaces, the vertices of the triangulation lie on the continuous surface. A finite element space of functions is then defined by taking the continuous functions which are linear affine on each simplex of the triangulated domain or surface. To demonstrate the role of cross-diffusion to the theory of pattern formation, we compute patterns with model kinetic parameter values that belong only to the cross-diffusion parameter space; these do not belong to the standard parameter space for classical reaction-diffusion systems. Numerical results exhibited show the robustness, flexibility, versatility, and generality of our methodology; the methodology can deal with complicated evolution laws of the domain and surface, and these include uniform isotropic and anisotropic growth profiles as well as those profiles driven by chemical concentrations residing in the domain or on the surface

Optimization of the Asymptotic Property of Mutual Learning Involving an Integration Mechanism of Ensemble Learning

We propose an optimization method of mutual learning which converges into the identical state of optimum ensemble learning within the framework of on-line learning, and have analyzed its asymptotic property through the statistical mechanics method.The proposed model consists of two learning steps: two students independently learn from a teacher, and then the students learn from each other through the mutual learning. In mutual learning, students learn from each other and the generalization error is improved even if the teacher has not taken part in the mutual learning. However, in the case of different initial overlaps(direction cosine) between teacher and students, a student with a larger initial overlap tends to have a larger generalization error than that of before the mutual learning. To overcome this problem, our proposed optimization method of mutual learning optimizes the step sizes of two students to minimize the asymptotic property of the generalization error. Consequently, the optimized mutual learning converges to a generalization error identical to that of the optimal ensemble learning. In addition, we show the relationship between the optimum step size of the mutual learning and the integration mechanism of the ensemble learning.Comment: 13 pages, 3 figures, submitted to Journal of Physical Society of Japa