The Synthesis of Arbitrary Stable Dynamics in Non-linear Neural Networks II: Feedback and Universality

We wish to construct a realization theory of stable neural networks and use this theory to model the variety of stable dynamics apparent in natural data. Such a theory should have numerous applications to constructing specific artificial neural networks with desired dynamical behavior. The networks used in this theory should have well understood dynamics yet be as diverse as possible to capture natural diversity. In this article, I describe a parameterized family of higher order, gradient-like neural networks which have known arbitrary equilibria with unstable manifolds of known specified dimension. Moreover, any system with hyperbolic dynamics is conjugate to one of these systems in a neighborhood of the equilibrium points. Prior work on how to synthesize attractors using dynamical systems theory, optimization, or direct parametric. fits to known stable systems, is either non-constructive, lacks generality, or has unspecified attracting equilibria. More specifically, We construct a parameterized family of gradient-like neural networks with a simple feedback rule which will generate equilibrium points with a set of unstable manifolds of specified dimension. Strict Lyapunov functions and nested periodic orbits are obtained for these systems and used as a method of synthesis to generate a large family of systems with the same local dynamics. This work is applied to show how one can interpolate finite sets of data, on nested periodic orbits.Air Force Office of Scientific Research (90-0128

The Essential Stability of Local Error Control for Dynamical Systems

Although most adaptive software for initial value problems is designed with an accuracy requirement—control of the local error—it is frequently observed that stability is imparted by the adaptation. This relationship between local error control and numerical stability is given a firm theoretical underpinning. The dynamics of numerical methods with local error control are studied for three classes of ordinary differential equations: dissipative, contractive, and gradient systems. Dissipative dynamical systems are characterised by having a bounded absorbing set B which all trajectories eventually enter and remain inside. The exponentially contractive problems studied have a unique, globally exponentially attracting equilibrium point and thus they are also dissipative since the absorbing set B may be chosen to be a ball of arbitrarily small radius around the equilibrium point. The gradient systems studied are those for which the set of equilibria comprises isolated points and all trajectories are bounded so that each trajectory converges to an equilibrium point as t → ∞. If the set of equilibria is bounded then the gradient systems are also dissipative. Conditions under which numerical methods with local error control replicate these large-time dynamical features are described. The results are proved without recourse to asymptotic expansions for the truncation error. Standard embedded Runge–Kutta pairs are analysed together with several nonstandard error control strategies. Both error per step and error per unit step strategies are considered. Certain embedded pairs are identified for which the sequence generated can be viewed as coming from a small perturbation of an algebraically stable scheme, with the size of the perturbation proportional to the tolerance τ. Such embedded pairs are defined to be essentially algebraically stable and explicit essentially stable pairs are identified. Conditions on the tolerance τ are identified under which appropriate discrete analogues of the properties of the underlying differential equation may be proved for certain essentially stable embedded pairs. In particular, it is shown that for dissipative problems the discrete dynamical system has an absorbing set B_τ and is hence dissipative. For exponentially contractive problems the radius of B_τ is proved to be proportional to τ. For gradient systems the numerical solution enters and remains in a small ball about one of the equilibria and the radius of the ball is proportional to τ. Thus the local error control mechanisms confer desirable global properties on the numerical solution. It is shown that for error per unit step strategies the conditions on the tolerance τ are independent of initial data while for error per step strategies the conditions are initial-data dependent. Thus error per unit step strategies are considerably more robust

Solving Structured Hierarchical Games Using Differential Backward Induction

Many real-world systems possess a hierarchical structure where a strategic plan is forwarded and implemented in a top-down manner. Examples include business activities in large companies or policy making for reducing the spread during pandemics. We introduce a novel class of games that we call structured hierarchical games (SHGs) to capture these strategic interactions. In an SHG, each player is represented as a vertex in a multi-layer decision tree and controls a real-valued action vector reacting to orders from its predecessors and influencing its descendants' behaviors strategically based on its own subjective utility. SHGs generalize extensive form games as well as Stackelberg games. For general SHGs with (possibly) nonconvex payoffs and high-dimensional action spaces, we propose a new solution concept which we call local subgame perfect equilibrium. By exploiting the hierarchical structure and strategic dependencies in payoffs, we derive a back propagation-style gradient-based algorithm which we call Differential Backward Induction to compute an equilibrium. We theoretically characterize the convergence properties of DBI and empirically demonstrate a large overlap between the stable points reached by DBI and equilibrium solutions. Finally, we demonstrate the effectiveness of our algorithm in finding \emph{globally} stable solutions and its scalability for a recently introduced class of SHGs for pandemic policy making

A striking correspondence between the dynamics generated by the vector fields and by the scalar parabolic equations

The purpose of this paper is to enhance a correspondence between the dynamics of the differential equations $\dot y(t)=g(y(t))$ on $\mathbb{R}^d$ and those of the parabolic equations $\dot u=\Delta u +f(x,u,\nabla u)$ on a bounded domain $\Omega$. We give details on the similarities of these dynamics in the cases $d=1$, $d=2$ and $d\geq 3$ and in the corresponding cases $\Omega=(0,1)$, $\Omega=\mathbb{T}^1$ and dim($\Omega$)$\geq 2$ respectively. In addition to the beauty of such a correspondence, this could serve as a guideline for future research on the dynamics of parabolic equations