1,273 research outputs found
On Second Order Behaviour in Augmented Neural ODEs.
Neural Ordinary Differential Equations (NODEs) are a new class of models that
transform data continuously through infinite-depth architectures. The
continuous nature of NODEs has made them particularly suitable for learning the
dynamics of complex physical systems. While previous work has mostly been
focused on first order ODEs, the dynamics of many systems, especially in
classical physics, are governed by second order laws. In this work, we take a
closer look at Second Order Neural ODEs (SONODEs). We show how the adjoint
sensitivity method can be extended to SONODEs and prove that an alternative
first order optimisation method is computationally more efficient. Furthermore,
we extend the theoretical understanding of the broader class of Augmented NODEs
(ANODEs) by showing they can also learn higher order dynamics, but at the cost
of interpretability. This indicates that the advantages of ANODEs go beyond the
extra space offered by the augmented dimensions, as originally thought.
Finally, we compare SONODEs and ANODEs on synthetic and real dynamical systems
and demonstrate that the inductive biases of the former generally result in
faster training and better performance.Comment: Contains 27 pages, 14 figure
Neural ODEs with stochastic vector field mixtures
It was recently shown that neural ordinary differential equation models
cannot solve fundamental and seemingly straightforward tasks even with
high-capacity vector field representations. This paper introduces two other
fundamental tasks to the set that baseline methods cannot solve, and proposes
mixtures of stochastic vector fields as a model class that is capable of
solving these essential problems. Dynamic vector field selection is of critical
importance for our model, and our approach is to propagate component
uncertainty over the integration interval with a technique based on forward
filtering. We also formalise several loss functions that encourage desirable
properties on the trajectory paths, and of particular interest are those that
directly encourage fewer expected function evaluations. Experimentally, we
demonstrate that our model class is capable of capturing the natural dynamics
of human behaviour; a notoriously volatile application area. Baseline
approaches cannot adequately model this problem
Multiscale Computations on Neural Networks: From the Individual Neuron Interactions to the Macroscopic-Level Analysis
We show how the Equation-Free approach for multi-scale computations can be
exploited to systematically study the dynamics of neural interactions on a
random regular connected graph under a pairwise representation perspective.
Using an individual-based microscopic simulator as a black box coarse-grained
timestepper and with the aid of simulated annealing we compute the
coarse-grained equilibrium bifurcation diagram and analyze the stability of the
stationary states sidestepping the necessity of obtaining explicit closures at
the macroscopic level. We also exploit the scheme to perform a rare-events
analysis by estimating an effective Fokker-Planck describing the evolving
probability density function of the corresponding coarse-grained observables
Optimising network interactions through device agnostic models
Physically implemented neural networks hold the potential to achieve the
performance of deep learning models by exploiting the innate physical
properties of devices as computational tools. This exploration of physical
processes for computation requires to also consider their intrinsic dynamics,
which can serve as valuable resources to process information. However, existing
computational methods are unable to extend the success of deep learning
techniques to parameters influencing device dynamics, which often lack a
precise mathematical description. In this work, we formulate a universal
framework to optimise interactions with dynamic physical systems in a fully
data-driven fashion. The framework adopts neural stochastic differential
equations as differentiable digital twins, effectively capturing both
deterministic and stochastic behaviours of devices. Employing differentiation
through the trained models provides the essential mathematical estimates for
optimizing a physical neural network, harnessing the intrinsic temporal
computation abilities of its physical nodes. To accurately model real devices'
behaviours, we formulated neural-SDE variants that can operate under a variety
of experimental settings. Our work demonstrates the framework's applicability
through simulations and physical implementations of interacting dynamic
devices, while highlighting the importance of accurately capturing system
stochasticity for the successful deployment of a physically defined neural
network
Mechanistic Neural Networks for Scientific Machine Learning
This paper presents Mechanistic Neural Networks, a neural network design for
machine learning applications in the sciences. It incorporates a new
Mechanistic Block in standard architectures to explicitly learn governing
differential equations as representations, revealing the underlying dynamics of
data and enhancing interpretability and efficiency in data modeling. Central to
our approach is a novel Relaxed Linear Programming Solver (NeuRLP) inspired by
a technique that reduces solving linear ODEs to solving linear programs. This
integrates well with neural networks and surpasses the limitations of
traditional ODE solvers enabling scalable GPU parallel processing. Overall,
Mechanistic Neural Networks demonstrate their versatility for scientific
machine learning applications, adeptly managing tasks from equation discovery
to dynamic systems modeling. We prove their comprehensive capabilities in
analyzing and interpreting complex scientific data across various applications,
showing significant performance against specialized state-of-the-art methods
- …