4 research outputs found
Port-Hamiltonian Approach to Neural Network Training
Neural networks are discrete entities: subdivided into discrete layers and
parametrized by weights which are iteratively optimized via difference
equations. Recent work proposes networks with layer outputs which are no longer
quantized but are solutions of an ordinary differential equation (ODE);
however, these networks are still optimized via discrete methods (e.g. gradient
descent). In this paper, we explore a different direction: namely, we propose a
novel framework for learning in which the parameters themselves are solutions
of ODEs. By viewing the optimization process as the evolution of a
port-Hamiltonian system, we can ensure convergence to a minimum of the
objective function. Numerical experiments have been performed to show the
validity and effectiveness of the proposed methods.Comment: To appear in the Proceedings of the 58th IEEE Conference on Decision
and Control (CDC 2019). The first two authors contributed equally to the wor
Thermodynamics of learning physical phenomena
Thermodynamics could be seen as an expression of physics at a high epistemic
level. As such, its potential as an inductive bias to help machine learning
procedures attain accurate and credible predictions has been recently realized
in many fields. We review how thermodynamics provides helpful insights in the
learning process. At the same time, we study the influence of aspects such as
the scale at which a given phenomenon is to be described, the choice of
relevant variables for this description or the different techniques available
for the learning process
Structure-preserving deep learning
Over the past few years, deep learning has risen to the foreground as a topic
of massive interest, mainly as a result of successes obtained in solving
large-scale image processing tasks. There are multiple challenging mathematical
problems involved in applying deep learning: most deep learning methods require
the solution of hard optimisation problems, and a good understanding of the
tradeoff between computational effort, amount of data and model complexity is
required to successfully design a deep learning approach for a given problem. A
large amount of progress made in deep learning has been based on heuristic
explorations, but there is a growing effort to mathematically understand the
structure in existing deep learning methods and to systematically design new
deep learning methods to preserve certain types of structure in deep learning.
In this article, we review a number of these directions: some deep neural
networks can be understood as discretisations of dynamical systems, neural
networks can be designed to have desirable properties such as invertibility or
group equivariance, and new algorithmic frameworks based on conformal
Hamiltonian systems and Riemannian manifolds to solve the optimisation problems
have been proposed. We conclude our review of each of these topics by
discussing some open problems that we consider to be interesting directions for
future research