48 research outputs found
On-line Learning of an Unlearnable True Teacher through Mobile Ensemble Teachers
On-line learning of a hierarchical learning model is studied by a method from
statistical mechanics. In our model a student of a simple perceptron learns
from not a true teacher directly, but ensemble teachers who learn from the true
teacher with a perceptron learning rule. Since the true teacher and the
ensemble teachers are expressed as non-monotonic perceptron and simple ones,
respectively, the ensemble teachers go around the unlearnable true teacher with
the distance between them fixed in an asymptotic steady state. The
generalization performance of the student is shown to exceed that of the
ensemble teachers in a transient state, as was shown in similar
ensemble-teachers models. Further, it is found that moving the ensemble
teachers even in the steady state, in contrast to the fixed ensemble teachers,
is efficient for the performance of the student.Comment: 18 pages, 8 figure
Statistical Mechanics of Linear and Nonlinear Time-Domain Ensemble Learning
Conventional ensemble learning combines students in the space domain. In this
paper, however, we combine students in the time domain and call it time-domain
ensemble learning. We analyze, compare, and discuss the generalization
performances regarding time-domain ensemble learning of both a linear model and
a nonlinear model. Analyzing in the framework of online learning using a
statistical mechanical method, we show the qualitatively different behaviors
between the two models. In a linear model, the dynamical behaviors of the
generalization error are monotonic. We analytically show that time-domain
ensemble learning is twice as effective as conventional ensemble learning.
Furthermore, the generalization error of a nonlinear model features
nonmonotonic dynamical behaviors when the learning rate is small. We
numerically show that the generalization performance can be improved remarkably
by using this phenomenon and the divergence of students in the time domain.Comment: 11 pages, 7 figure
Statistical Mechanics of Time Domain Ensemble Learning
Conventional ensemble learning combines students in the space domain. On the
other hand, in this paper we combine students in the time domain and call it
time domain ensemble learning. In this paper, we analyze the generalization
performance of time domain ensemble learning in the framework of online
learning using a statistical mechanical method. We treat a model in which both
the teacher and the student are linear perceptrons with noises. Time domain
ensemble learning is twice as effective as conventional space domain ensemble
learning.Comment: 10 pages, 10 figure
Machine learning: statistical physics based theory and smart industry applications
The increasing computational power and the availability of data have made it possible to train ever-bigger artificial neural networks. These so-called deep neural networks have been used for impressive applications, like advanced driver assistance and support in medical diagnoses. However, various vulnerabilities have been revealed and there are many open questions concerning the workings of neural networks. Theoretical analyses are therefore essential for further progress. One current question is: why is it that networks with Rectified Linear Unit (ReLU) activation seemingly perform better than networks with sigmoidal activation?We contribute to the answer to this question by comparing ReLU networks with sigmoidal networks in diverse theoretical learning scenarios. In contrast to analysing specific datasets, we use a theoretical modelling using methods from statistical physics. They give the typical learning behaviour for chosen model scenarios. We analyse both the learning behaviour on a fixed dataset and on a data stream in the presence of a changing task. The emphasis is on the analysis of the network’s transition to a state wherein specific concepts have been learnt. We find significant benefits of ReLU networks: they exhibit continuous increases of their performance and adapt more quickly to changing tasks.In the second part of the thesis we treat applications of machine learning: we design a quick quality control method for material in a production line and study the relationship with product faults. Furthermore, we introduce a methodology for the interpretable classification of time series data
The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks
We derive a simple unified framework giving closed-form estimates for the
test risk and other generalization metrics of kernel ridge regression (KRR).
Relative to prior work, our derivations are greatly simplified and our final
expressions are more readily interpreted. These improvements are enabled by our
identification of a sharp conservation law which limits the ability of KRR to
learn any orthonormal basis of functions. Test risk and other objects of
interest are expressed transparently in terms of our conserved quantity
evaluated in the kernel eigenbasis. We use our improved framework to: i)
provide a theoretical explanation for the "deep bootstrap" of Nakkiran et al
(2020), ii) generalize a previous result regarding the hardness of the classic
parity problem, iii) fashion a theoretical tool for the study of adversarial
robustness, and iv) draw a tight analogy between KRR and a well-studied system
in statistical physics
Pastplay: Teaching and Learning History with Technology
In the field of history, the Web and other technologies have become important tools in research and teaching of the past. Yet the use of these tools is limited—many historians and history educators have resisted adopting them because they fail to see how digital tools supplement and even improve upon conventional tools (such as books). In Pastplay, a collection of essays by leading history and humanities researchers and teachers, editor Kevin Kee works to address these concerns head-on. How should we use technology? Playfully, Kee contends. Why? Because doing so helps us think about the past in new ways; through the act of creating technologies, our understanding of the past is re-imagined and developed. From the insights of numerous scholars and teachers, Pastplay argues that we should play with technology in history because doing so enables us to see the past in new ways by helping us understand how history is created; honoring the roots of research, teaching, and technology development; requiring us to model our thoughts; and then allowing us to build our own understanding