Embedding Capabilities of Neural ODEs

A class of neural networks that gained particular interest in the last years are neural ordinary differential equations (neural ODEs). We study input-output relations of neural ODEs using dynamical systems theory and prove several results about the exact embedding of maps in different neural ODE architectures in low and high dimension. The embedding capability of a neural ODE architecture can be increased by adding, for example, a linear layer, or augmenting the phase space. Yet, there is currently no systematic theory available and our work contributes towards this goal by developing various embedding results as well as identifying situations, where no embedding is possible. The mathematical techniques used include as main components iterative functional equations, Morse functions and suspension flows, as well as several further ideas from analysis. Although practically, mainly universal approximation theorems are used, our geometric dynamical systems viewpoint on universal embedding provides a fundamental understanding, why certain neural ODE architectures perform better than others

Geometric Blow-up for Folded Limit Cycle Manifolds in Three Time-Scale Systems

$\textit{Geometric singular perturbation theory}$ provides a powerful mathematical framework for the analysis of 'stationary' multiple time-scale systems which possess a $\textit{critical manifold}$, i.e. a smooth manifold of steady states for the limiting fast subsystem, particularly when combined with a method of desingularization known as $\textit{blow-up}$. The theory for 'oscillatory' multiple time-scale systems which possess a limit cycle manifold instead of (or in addition to) a critical manifold is less developed, particularly in the non-normally hyperbolic regime. We show that the blow-up method can be applied to analyse the global oscillatory transition near a regular folded limit cycle manifold in a class of three time-scale systems with two small parameters. The systems considered behave like oscillatory systems as the smallest perturbation parameter tends to zero, and stationary systems as both perturbation parameters tend to zero. The additional time-scale structure is crucial for the applicability of the blow-up method, which cannot be applied directly to the two time-scale counterpart of the problem. Our methods allow us to describe the asymptotics and strong contractivity of all solutions which traverse a neighbourhood of the global singularity. Our results cover a range of different cases with respect to the relative time-scale of the angular dynamics and the parameter drift