Differential Dynamic Programming for time-delayed systems

Trajectory optimization considers the problem of deciding how to control a dynamical system to move along a trajectory which minimizes some cost function. Differential Dynamic Programming (DDP) is an optimal control method which utilizes a second-order approximation of the problem to find the control. It is fast enough to allow real-time control and has been shown to work well for trajectory optimization in robotic systems. Here we extend classic DDP to systems with multiple time-delays in the state. Being able to find optimal trajectories for time-delayed systems with DDP opens up the possibility to use richer models for system identification and control, including recurrent neural networks with multiple timesteps in the state. We demonstrate the algorithm on a two-tank continuous stirred tank reactor. We also demonstrate the algorithm on a recurrent neural network trained to model an inverted pendulum with position information only.Comment: 7 pages, 6 figures, conference, Decision and Control (CDC), 2016 IEEE 55th Conference o

Time-delayed feedback control of unstable periodic orbits near a subcritical Hopf bifurcation

We show that Pyragas delayed feedback control can stabilize an unstable periodic orbit (UPO) that arises from a generic subcritical Hopf bifurcation of a stable equilibrium in an n-dimensional dynamical system. This extends results of Fiedler et al. [PRL 98, 114101 (2007)], who demonstrated that such feedback control can stabilize the UPO associated with a two-dimensional subcritical Hopf normal form. Pyragas feedback requires an appropriate choice of a feedback gain matrix for stabilization, as well as knowledge of the period of the targeted UPO. We apply feedback in the directions tangent to the two-dimensional center manifold. We parameterize the feedback gain by a modulus and a phase angle, and give explicit formulae for choosing these two parameters given the period of the UPO in a neighborhood of the bifurcation point. We show, first heuristically, and then rigorously by a center manifold reduction for delay differential equations, that the stabilization mechanism involves a highly degenerate Hopf bifurcation problem that is induced by the time-delayed feedback. When the feedback gain modulus reaches a threshold for stabilization, both of the genericity assumptions associated with a two-dimensional Hopf bifurcation are violated: the eigenvalues of the linearized problem do not cross the imaginary axis as the bifurcation parameter is varied, and the real part of the cubic coefficient of the normal form vanishes. Our analysis of this degenerate bifurcation problem reveals two qualitatively distinct cases when unfolded in a two-parameter plane. In each case, Pyragas-type feedback successfully stabilizes the branch of small-amplitude UPOs in a neighborhood of the original bifurcation point, provided that the phase angle satisfies a certain restriction.Comment: 35 pages, 19 figure

On the approximation of delayed systems by Taylor series expansion

It is known that stability properties of delay-differential equations are not preserved by Taylor series expansion of the delayed term. Still, this technique is often used to approximate delayed systems by ordinary differential equations in different engineering and biological applications. In this brief, it is demonstrated through some simple second-order scalar systems that low-order Taylor series expansion of the delayed term approximates the asymptotic behavior of the original delayed system only for certain parameter regions, while for high-order expansions, the approximate system is unstable independently of the system parameters

Path integrals and symmetry breaking for optimal control theory

This paper considers linear-quadratic control of a non-linear dynamical system subject to arbitrary cost. I show that for this class of stochastic control problems the non-linear Hamilton-Jacobi-Bellman equation can be transformed into a linear equation. The transformation is similar to the transformation used to relate the classical Hamilton-Jacobi equation to the Schr\"odinger equation. As a result of the linearity, the usual backward computation can be replaced by a forward diffusion process, that can be computed by stochastic integration or by the evaluation of a path integral. It is shown, how in the deterministic limit the PMP formalism is recovered. The significance of the path integral approach is that it forms the basis for a number of efficient computational methods, such as MC sampling, the Laplace approximation and the variational approximation. We show the effectiveness of the first two methods in number of examples. Examples are given that show the qualitative difference between stochastic and deterministic control and the occurrence of symmetry breaking as a function of the noise.Comment: 21 pages, 6 figures, submitted to JSTA

Derivation of Delay Equation Climate Models Using the Mori-Zwanzig Formalism

Models incorporating delay have been frequently used to understand climate variability phenomena, but often the delay is introduced through an ad-hoc physical reasoning, such as the propagation time of waves. In this paper, the Mori-Zwanzig formalism is introduced as a way to systematically derive delay models from systems of partial differential equations and hence provides a better justification for using these delay-type models. The Mori-Zwanzig technique gives a formal rewriting of the system using a projection onto a set of resolved variables, where the rewritten system contains a memory term. The computation of this memory term requires solving the orthogonal dynamics equation, which represents the unresolved dynamics. For nonlinear systems, it is often not possible to obtain an analytical solution to the orthogonal dynamics and an approximate solution needs to be found. Here, we demonstrate the Mori-Zwanzig technique for a two-strip model of the El Nino Southern Oscillation (ENSO) and explore methods to solve the orthogonal dynamics. The resulting nonlinear delay model contains an additional term compared to previously proposed ad-hoc conceptual models. This new term leads to a larger ENSO period, which is closer to that seen in observations.Comment: Submitted to Proceedings of the Royal Society A, 25 pages, 10 figure

Model order reduction of time-delay systems using a laguerre expansion technique

The demands for miniature sized circuits with higher operating speeds have increased the complexity of the circuit, while at high frequencies it is known that effects such as crosstalk, attenuation and delay can have adverse effects on signal integrity. To capture these high speed effects a very large number of system equations is normally required and hence model order reduction techniques are required to make the simulation of the circuits computationally feasible. This paper proposes a higher order Krylov subspace algorithm for model order reduction of time-delay systems based on a Laguerre expansion technique. The proposed technique consists of three sections i.e., first the delays are approximated using the recursive relation of Laguerre polynomials, then in the second part, the reduced order is estimated for the time-delay system using a delay truncation in the Laguerre domain and in the third part, a higher order Krylov technique using Laguerre expansion is computed for obtaining the reduced order time-delay system. The proposed technique is validated by means of real world numerical examples