Statistical Mechanics of Time Domain Ensemble Learning

Conventional ensemble learning combines students in the space domain. On the other hand, in this paper we combine students in the time domain and call it time domain ensemble learning. In this paper, we analyze the generalization performance of time domain ensemble learning in the framework of online learning using a statistical mechanical method. We treat a model in which both the teacher and the student are linear perceptrons with noises. Time domain ensemble learning is twice as effective as conventional space domain ensemble learning.Comment: 10 pages, 10 figure

On-line Learning of an Unlearnable True Teacher through Mobile Ensemble Teachers

On-line learning of a hierarchical learning model is studied by a method from statistical mechanics. In our model a student of a simple perceptron learns from not a true teacher directly, but ensemble teachers who learn from the true teacher with a perceptron learning rule. Since the true teacher and the ensemble teachers are expressed as non-monotonic perceptron and simple ones, respectively, the ensemble teachers go around the unlearnable true teacher with the distance between them fixed in an asymptotic steady state. The generalization performance of the student is shown to exceed that of the ensemble teachers in a transient state, as was shown in similar ensemble-teachers models. Further, it is found that moving the ensemble teachers even in the steady state, in contrast to the fixed ensemble teachers, is efficient for the performance of the student.Comment: 18 pages, 8 figure

Statistical Mechanics of Nonlinear On-line Learning for Ensemble Teachers

We analyze the generalization performance of a student in a model composed of nonlinear perceptrons: a true teacher, ensemble teachers, and the student. We calculate the generalization error of the student analytically or numerically using statistical mechanics in the framework of on-line learning. We treat two well-known learning rules: Hebbian learning and perceptron learning. As a result, it is proven that the nonlinear model shows qualitatively different behaviors from the linear model. Moreover, it is clarified that Hebbian learning and perceptron learning show qualitatively different behaviors from each other. In Hebbian learning, we can analytically obtain the solutions. In this case, the generalization error monotonically decreases. The steady value of the generalization error is independent of the learning rate. The larger the number of teachers is and the more variety the ensemble teachers have, the smaller the generalization error is. In perceptron learning, we have to numerically obtain the solutions. In this case, the dynamical behaviors of the generalization error are non-monotonic. The smaller the learning rate is, the larger the number of teachers is; and the more variety the ensemble teachers have, the smaller the minimum value of the generalization error is.Comment: 13 pages, 9 figure

Statistical Mechanics of Linear and Nonlinear Time-Domain Ensemble Learning

Conventional ensemble learning combines students in the space domain. In this paper, however, we combine students in the time domain and call it time-domain ensemble learning. We analyze, compare, and discuss the generalization performances regarding time-domain ensemble learning of both a linear model and a nonlinear model. Analyzing in the framework of online learning using a statistical mechanical method, we show the qualitatively different behaviors between the two models. In a linear model, the dynamical behaviors of the generalization error are monotonic. We analytically show that time-domain ensemble learning is twice as effective as conventional ensemble learning. Furthermore, the generalization error of a nonlinear model features nonmonotonic dynamical behaviors when the learning rate is small. We numerically show that the generalization performance can be improved remarkably by using this phenomenon and the divergence of students in the time domain.Comment: 11 pages, 7 figure

Oscillation Phenomena in the disk around the massive black hole Sagittarius A*

We report the detection of radio QPOs with structure changes using the Very Long Baseline Array (VLBA) at 43 GHz. We found conspicuous patterned changes of the structure with P = 16.8 +- 1.4, 22.2 +- 1.4, 31.2 +- 1.5, 56.4 +- 6 min, very roughly in a 3:4:6:10 ratio. The first two periods show a rotating one-arm structure, while the P = 31.4 min shows a rotating 3-arm structure, as if viewed edge-on. At the central 50 microasec the P = 56.4 min period shows a double amplitude variation of those in its surroundings. Spatial distributions of the oscillation periods suggest that the disk of SgrA* is roughly edge-on, rotating around an axis with PA = -10 degree. Presumably, the observed VLBI images of SgrA* at 43 GHz retain several features of the black hole accretion disk of SgrA* in spite of being obscured and broadened by scattering of surrounding plasma.Comment: 24 pages, 20 figures, revised version submitted to MN main journal (2010, Jan., 12th

Average profiles of the solar wind and outer radiation belt during the extreme flux enhancement of relativistic electrons at geosynchronous orbit

We report average profiles of the solar wind and outer radiation belt during the extreme flux enhancement of relativistic electrons at geosynchronous orbit (GEO). It is found that seven of top ten extreme events at GEO during solar cycle 23 are associated with the magnetosphere inflation during the storm recovery phase as caused by the large-scale solar wind structure of very low dynamic pressure (<1.0 nPa) during rapid speed decrease from very high (>650 km/s) to typical (400–500 km/s) in a few days. For the seven events, the solar wind parameters, geomagnetic activity indices, and relativistic electron flux and geomagnetic field at GEO are superposed at the local noon period of GOES satellites to investigate the physical cause. The average profiles support the "double inflation" mechanism that the rarefaction of the solar wind and subsequent magnetosphere inflation are one of the best conditions to produce the extreme flux enhancement at GEO because of the excellent magnetic confinement of relativistic electrons by reducing the drift loss of trapped electrons at dayside magnetopause

Optimization of the Asymptotic Property of Mutual Learning Involving an Integration Mechanism of Ensemble Learning

We propose an optimization method of mutual learning which converges into the identical state of optimum ensemble learning within the framework of on-line learning, and have analyzed its asymptotic property through the statistical mechanics method.The proposed model consists of two learning steps: two students independently learn from a teacher, and then the students learn from each other through the mutual learning. In mutual learning, students learn from each other and the generalization error is improved even if the teacher has not taken part in the mutual learning. However, in the case of different initial overlaps(direction cosine) between teacher and students, a student with a larger initial overlap tends to have a larger generalization error than that of before the mutual learning. To overcome this problem, our proposed optimization method of mutual learning optimizes the step sizes of two students to minimize the asymptotic property of the generalization error. Consequently, the optimized mutual learning converges to a generalization error identical to that of the optimal ensemble learning. In addition, we show the relationship between the optimum step size of the mutual learning and the integration mechanism of the ensemble learning.Comment: 13 pages, 3 figures, submitted to Journal of Physical Society of Japa

Ensemble learning of linear perceptron; Online learning theory

Within the framework of on-line learning, we study the generalization error of an ensemble learning machine learning from a linear teacher perceptron. The generalization error achieved by an ensemble of linear perceptrons having homogeneous or inhomogeneous initial weight vectors is precisely calculated at the thermodynamic limit of a large number of input elements and shows rich behavior. Our main findings are as follows. For learning with homogeneous initial weight vectors, the generalization error using an infinite number of linear student perceptrons is equal to only half that of a single linear perceptron, and converges with that of the infinite case with O(1/K) for a finite number of K linear perceptrons. For learning with inhomogeneous initial weight vectors, it is advantageous to use an approach of weighted averaging over the output of the linear perceptrons, and we show the conditions under which the optimal weights are constant during the learning process. The optimal weights depend on only correlation of the initial weight vectors.Comment: 14 pages, 3 figures, submitted to Physical Review