Brune, Christoph

Brunton, Steven L.

Kalia, Manu

Kutz, J. Nathan

Meijer, Hil Gaétan Ellart

English

A long-standing goal in dynamical systems is to construct reduced-order models for high-dimensional spatiotemporal data that capture the underlying parametric dependence of the data. In this work, we introduce a deep learning technique to discover a single low-dimensional model for such data that captures the underlying parametric dependence in terms of a normal form. A normal form is a symbolic expression, or universal unfolding, that describes how a the reduced-order differential equation model varies with respect to a bifurcation parameter. Our approach introduces coupled autoencoders for the state and parameter, with the latent variables constrained to adhere to a given normal form. We demonstrate our method on one-parameter bifurcations that occur in the canonical Lorenz96 equations and a neural field equation. This method demonstrates how normal forms can be leveraged as canonical and universal building blocks in deep learning approaches for model discovery and reduced-order modeling

University of Twente Research Information

Deep learning of normal form autoencoders foruniversal, parameter-dependent dynamicsManu KaliaDepartment of Applied MathematicsUniversity of Twentem.kalia@utwente.nlSteven L. BruntonDepartment of Mechanical EngineeringUniversity of WashingtonHil G.E. MeijerDepartment of Applied MathematicsUniversity of TwenteChristoph BruneDepartment of Applied MathematicsUniversity of TwenteJ. Nathan KutzDepartment of Applied MathematicsUniversity of WashingtonAbstractA long-standing goal in dynamical systems is to construct reduced-order modelsfor high-dimensional spatiotemporal data that capture the underlying parametricdependence of the data. In this work, we introduce a deep learning techniqueto discover a single low-dimensional model for such data that captures the un-derlying parametric dependence in terms of a normal form. A normal form is asymbolic expression, or universal unfolding, that describes how a the reduced-orderdifferential equation model varies with respect to a bifurcation parameter. Ourapproach introduces coupled autoencoders for the state and parameter, with thelatent variables constrained to adhere to a given normal form. We demonstrateour method on one-parameter bifurcations that occur in the canonical Lorenz96equations and a neural field equation. This method demonstrates how normalforms can be leveraged as canonical and universal building blocks in deep learningapproaches for model discovery and reduced-order modeling.1 IntroductionDiscovery of reduced-order models from high-dimensional spatiotemporal data is central to a quali-tative understanding of underlying characteristics in dynamics [1, 2]. Current approaches includemodal decomposition techniques, such as proper orthogonal decomposition (POD) and dynamicmode decomposition (DMD), which approximate low-dimensional projections by using linear mapsbetween snapshots in time [3, 4, 5, 6, 7, 8, 9]. These methods are reinforced by model discov-ery techniques, such as sparse identification of nonlinear dynamics (SINDy), which can uncoverparsimonious nonlinear models from data [10, 11]. However, the techniques above depend on apredefined candidate library of basis functions. Recent work has focused on combining modeldiscovery and reduced-order modeling methods with neural networks and deep learning to removethis dependency [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25].These methods, however, do not directly deal with temporal datasets originating from a singlephysical system across different experimental conditions and parametrizations. Such data can exhibittopological inequivalence [26], where two time-traces from the data cannot be mapped onto each1st NeurIPS workshop on Interpretable Inductive Biases and Physically Structured Learning (2020), virtual....  ......Figure 1: Overview of the state-of-the-art in the reduced-order modeling of parametric systems (left),resulting in isolated models for each parameter value. Our novel approach (right), uncovers a singleparameterized equation (i.e., a normal form) that captures the parametric dependence across the data.other by continuous invertible transformations. This presents a challenge for the aforementionedmethods, as observations from a single physical system might yield irreconcilably different low-dimensional models. In this work, we present a first deep learning approach that extracts a singlelow-dimensional model, and its parametric dependence, from high dimensional temporal data of asingle physical system across different experimental conditions.A summary is presented in Fig. 1. The resulting low-dimensional models are normal form equations,which describe universal behaviour of transitions in topological equivalence (called bifurcations) [1,26]. We do so by using neural networks that transform observations of states and parameterssimultaneously while constraining the transformed variables to normal form equations. In thiswork, we focus on observations corresponding to a single bifurcation only. Moreover, the resultingtransformation can be interpreted as a restriction of high-dimensional dynamics to the underlyinglow-dimensional theoretical center manifold, on which the interesting transitions occur.2 MethodsFigure 2 describes the neural network architecture, the loss functions and the normal forms weconsider. The architecture is composed of two fully connected adjoint autoencoders (Φ1, ϕ1) and(Φ2, ϕ2). They encode state x and parameter α to the respective latent state z and parameter β suchthat ẋ = f(x, α), and ż = g(z, β) is a normal form equation. The architecture is inspired from [19]where we now add an extra autoencoder for the parameters α. The various normal forms, with theirequations and representative phase portraits are also presented in Fig. 2.Loss function: We also show the loss function, that is composed of three kinds of terms. The‘decoder loss’ refers to the two autoencoder loss terms. The ‘consistency loss’ couples the twoautoencoders by constraining the latent variables z and β to the normal form equation ż = g(z, β).Inductive biases are introduced in the ‘orientation loss’ that is composed of two terms, the zero-loss(λ5‖Φ1(0)‖1) and the parameter-orientation loss. We motivate these terms here.Figure 2: Summary of the method. We show the network architecture, the various normal formsconsidered in this work, and the loss function we use to constrain the data to the normal form.Throughout the work, we set λ1 = λ2 = λ5 = λ6 = 1, and λ2 = λ3 = 10−3.Existence and uniqueness: First, we note that the theory of normal forms and bifurcations guaranteea solution for the neural network parameters Φi, ϕi. For any high-dimensional system undergoing2Figure 3: Results for the 1D system Eq. 1. For each bifurcation, 10 test traces are concatenated andpresented. In all cases, training and test data contain 500 and 20 traces, respectively. The gray areashows an ensemble of 50 simulated trajectories. Φ1 and ϕ1 have widths [20, 20] and [10, 10].a bifurcation, there exists a smooth transformation to a low-dimensional center manifold such thatthe dynamics on this manifold are described by the normal form equations [1, 26]. In our case,the transformation is smooth as we always use elu activation functions. Moreover, as a byproductthe autoencoder determines the center manifold. In classical literature on obtaining normal formequations [26], the center manifold is explicitly constructed which is only feasible for low-dimensionalmodels. In order to use center manifold theory, the bifurcation in the original system ẋ = f(x, α)is always shifted to (x, α) = (0, 0). In our training and test data we ensure this by appropriatelytranslating the dataset. However, the center manifold is not unique. Thus, there may exist multiplesolutions Φi, ϕi .Inductive bias: In order to deal with non-uniqueness and underlying local minima, we introduce thezero loss and the parameter-orientation loss. The zero loss forces the autoencoder to recognize that azero solution in the original dynamics must correspond to a zero solution in the latent dynamics. Thisis necessary to make sure that we take into account the existence of the trivial solution in the normalform equation as an equilibrium in most cases. The parameter orientation loss fixes the direction ofthe bifurcation for the latent dynamics, by making sure that α and β have the same sign, therebyeliminating several local minima.Training: Initial conditions and parameters are chosen from a uniform distribution. They are shuffledand paired together, and then used together to simulate time traces, that form test and training data.Training is performed for 200 epochs with batch size 100 or till the test loss stabilizes, which is to≈ 1× 10−2 in all examples.3 ResultsIn this section we present the use of our approach on a few examples, showcasing the most importantlocal codimension one bifurcations: saddle-node, transcritical, pitchfork and Hopf.1D system with several bifurcations. We start by looking at the following scalar ODE that containsmultiple bifurcations,ẋ = γx(α− αpf − x2)(α− αsn + (x− xsn)2), (1)where we set γ = 0.01, xsn = αsn = −6 and αpf = 6. Fig. 3 shows the bifurcation diagram forEq. (1). The system exhibits saddle-node (at (x, α) = (−6,−6)), pitchfork (at (x, α) = (0, 6)) andtranscritical (at (x, α) = (0,−30)) bifurcations. Using affine linear transformations of x and α, wereorient Eq. 1 around each bifurcation point. We construct training and test data by simulating forvarious parameter values; 10 such test traces clubbed together are presented for each bifurcationscenario (in blue) in Fig. 3.Results are presented in Fig. 3. We show learned test data (blue) and normal form simulations(gray), together for 10 traces. These traces correspond to different initial values and parameters. Thesimulations are performed by using the learned parameter and initial condition from the learned timetraces. We observe agreement across different cases, for all bifurcation scenarios. We note that theparameter orientation loss was not used (λ5 = 0) in the transcritical case, to allow the parameters(blue) to orient themselves properly with respect to the normal form (α 7→ −α). In the remainingtwo bifurcations, the learned (blue) and original (orange) parameters have the same sign. Moreover,3Figure 4: Results for the two high-dimensional systems, Eq.2 and Eq. 3. In both cases, training andtest data contain 1000 and 20 traces, respectively. Φ1 has widths [32, 16] (Lorenz96) and [64, 32, 16](Neural field). ϕ1 has widths [16, 16] in both cases.in the case of pitchfork and transcritical bifurcations, the zero loss ensures that the trivial equilibriumx = 0 is preserved.Hopf bifurcation in the Lorenz96 system. The Lorenz96 equations [27] are used widely in modeldiscovery and data assimilation problems. The equations are given byẋj = −xj−1(xj−2 − xj+1)− xj + α (2)for j = 1, 2, 3...N with boundary conditions x1 = xN and x2 = xN−1. In this work we set N = 64.For this choice of N , the trivial equilibrium x = α undergoes a supercritical Hopf bifurcation withrespect to α at α = α0 = 0.84975 [28]. Using affine linear transformations of x and α, we firstreorient Eq. 2 around (x, α) = (α0, α0) such that the bifurcation now occurs at (x, α) = (0, 0).Simulations of the system before and after the bifurcation point are shown in Fig. 4. For α < 0, weobserve the existence of a stable stationary solution, while for α > 0 we observe a spatiotemporalstripe pattern, which is interpreted as a travelling wave.Results are presented in Fig. 4. For two choices of α on different sides of the bifurcation point, weplot learned (blue) and simulated time traces from test data, that match well qualitatively. The highdimensional periodic pattern is encoded to a stable periodic solution, and the stationary solutionis encoded into a damped oscillation. The encoding of the time traces match well with parameterencoding, as the sign of the original parameters (orange) is the same as the learned parameters (blue).Hopf bifurcation in a neural field equation. Next, we consider a neural field equation describingthe neuronal potential for a one-dimensional continuum of neural tissue [29, 30]. The spatiotemporaldynamics due to an input inhomogeneity lead to a Hopf bifurcation of a stationary pattern leading tobreathers when varying the input strength [31, 32]. The equations are given by,u̇ = −u− κa+ (w ∗ f(u)) + I(x),ȧ = (−a+ u)/τnf . (3)The operator ∗ represents a spatial convolution. w(x) ≡ w(x − y) = we exp(−((x − y)/σe)2)is the spatial connectivity kernel and f(u) is the potential-dependent sigmoidal firing rate givenby f(u) = (1 + exp(βnf (u− uthr ))). The spatially non-uniform input I(x) is given by I(x) =α exp(−(x/σ)2).We fix κ = 2.75, τnf = 10, we = 1, σe = 1, βnf = 6, uthr = 0.375, σ = 1.2. The parameter αis used as the bifurcation parameter. A supercritical Hopf bifurcation with respect to a stationarybump response occurs at α = 0.8040. States u, a are discretized over a uniform grid of size 64 each.Like before, we first reorient Eq. 3 such that the bifurcation occurs at (u, a, α) = (0, 0, 0). Settingα < 0 produces the globally asymptotically stable stationary bump response. This is shown for thevariable u in Fig. 4. For α > 0, we see that the stationary bump loses stability and the emergingasymptotically stable periodic solution expands and contracts, referred to as ‘breathers’ [32].Results are presented in Fig. 4. Similar to the Lorenz96 case, we present learned test data for twocases, α < 0 and α > 0. Once again, the qualitative behaviour of the time-traces match the Hopfnormal form dynamics and are supported by the parameter encodings. The sign of the learnedparameters are the same as the original parameters. In both Lorenz96 and the neural field equations,fixing the orientation loss helps avoid local minima and other symmetry-induced feasible solutionsby fixing the sign of the learned parameters and thus, the direction of the bifurcation.4ConclusionWe have demonstrated a deep learning approach to uncover a coordinate transformation into a single,parameterized normal form equation that describes the parametric dependence of a data set over arange of values. Our approach has consequences for dynamical systems theory and data-driven modeldiscovery alike. The approach can be extended to discover underlying low-dimensional models byusing normal forms as building blocks. On the other hand, this presents a novel method to computecenter manifold restrictions, which is an important problem in applied dynamical systems theory.References[1] John Guckenheimer and Philip Holmes. Nonlinear Oscillations, Dynamical Systems, andBifurcations of Vector Fields. Springer New York, 1983.[2] P. J. Holmes, J. L. Lumley, G. Berkooz, and C. W. Rowley. Turbulence, coherent structures,dynamical systems and symmetry. Cambridge Monographs in Mechanics. Cambridge UniversityPress, Cambridge, England, 2nd edition, 2012.[3] Gal Berkooz, Philip Holmes, and John L Lumley. The proper orthogonal decomposition in theanalysis of turbulent flows. Annual review of fluid mechanics, 25(1):539–575, 1993.[4] K. Willcox and J. Peraire. Balanced model reduction via the proper orthogonal decomposition.AIAA Journal, 40(11):2323–2330, 2002.[5] Clarence W Rowley. Model reduction for fluids, using balanced proper orthogonal decomposi-tion. International Journal of Bifurcation and Chaos, 15(03):997–1013, 2005.[6] Peter J Schmid. Dynamic mode decomposition of numerical and experimental data. Journal offluid mechanics, 656:5–28, 2010.[7] Jonathan H. Tu, Clarence W. Rowley, Dirk M. Luchtenburg, Steven L. Brunton, and J. NathanKutz. On dynamic mode decomposition: Theory and applications. Journal of ComputationalDynamics, 1(2158-2491-2014-2-391):391, 2014.[8] Qianxiao Li, Felix Dietrich, Erik M. Bollt, and Ioannis G. Kevrekidis. Extended dynamic modedecomposition with dictionary learning: A data-driven adaptive spectral decomposition of thekoopman operator. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(10):103111,2017.[9] Kunihiko Taira, Steven L Brunton, Scott Dawson, Clarence W Rowley, Tim Colonius, Beverley JMcKeon, Oliver T Schmidt, Stanislav Gordeyev, Vassilios Theofilis, and Lawrence S Ukeiley.Modal analysis of fluid flows: An overview. AIAA Journal, 55(12):4013–4041, 2017.[10] Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equationsfrom data by sparse identification of nonlinear dynamical systems. Proceedings of the nationalacademy of sciences, 113(15):3932–3937, 2016.[11] Samuel H. Rudy, Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Data-drivendiscovery of partial differential equations. Science Advances, 3(4):e1602614, apr 2017.[12] Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant sub-spaces for dynamic mode decomposition. In Advances in Neural Information ProcessingSystems, pages 1130–1140, 2017.[13] Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representationsfor koopman operators of nonlinear dynamical systems. arXiv preprint arXiv:1708.06850, 2017.[14] Christoph Wehmeyer and Frank Noé. Time-lagged autoencoders: Deep learning of slowcollective variables for molecular kinetics. The Journal of Chemical Physics, 148(241703):1–9,2018.[15] Andreas Mardt, Luca Pasquali, Hao Wu, and Frank Noé. VAMPnets: Deep learning of molecularkinetics. Nature Communications, 9(5), 2018.[16] Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Deep learning for universal linearembeddings of nonlinear dynamics. Nature Communications, 9(1):4950, dec 2018.[17] Pantelis R Vlachas, Wonmin Byeon, Zhong Y Wan, Themistoklis P Sapsis, and Petros Koumout-sakos. Data-driven forecasting of high-dimensional chaotic systems with long short-termmemory networks. Proc. R. Soc. A, 474(2213):20170844, 2018.5[18] Jaideep Pathak, Brian Hunt, Michelle Girvan, Zhixin Lu, and Edward Ott. Model-free predictionof large spatiotemporally chaotic systems from data: a reservoir computing approach. Physicalreview letters, 120(2):024102, 2018.[19] Kathleen Champion, Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Data-drivendiscovery of coordinates and governing equations. Proceedings of the National Academy ofSciences, 116(45):22445–22451, nov 2019.[20] Samuel E. Otto and Clarence W. Rowley. Linearly recurrent autoencoder networks for learningdynamics. SIAM Journal on Applied Dynamical Systems, 18(1):558–593, 2019.[21] Francesco Regazzoni, Luca Dedè, and Alfio Quarteroni. Machine learning for fast and re-liable solution of time-dependent differential equations. Journal of Computational Physics,397:108852, 2019.[22] Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-drivendiscretizations for partial differential equations. Proceedings of the National Academy ofSciences, 116(31):15344–15349, 2019.[23] Alec J Linot and Michael D Graham. Deep learning to discover and predict dynamics on aninertial manifold. Physical Review E, 101(6):062209, 2020.[24] Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learningvelocity and pressure fields from flow visualizations. Science, 367(6481):1026–1030, 2020.[25] Kookjin Lee and Kevin T Carlberg. Model reduction of dynamical systems on nonlinear mani-folds using deep convolutional autoencoders. Journal of Computational Physics, 404:108973,2020.[26] Yu. A. Kuznetsov. Elements of Applied Bifurcation Theory, volume 112 of Applied MathematicalSciences. Springer-Verlag, New York, third edition, 2004.[27] Edward N. Lorenz. Predictability – a problem partly solved, page 40–58. Cambridge UniversityPress, 2006.[28] Dirk L. van Kekem and Alef E. Sterk. Travelling waves and their bifurcations in the lorenz-96model. Physica D: Nonlinear Phenomena, 367:38 – 60, 2018.[29] Shun-ichi Amari. Dynamics of pattern formation in lateral-inhibition type neural fields. Biolog-ical cybernetics, 27(2):77–87, 1977.[30] Hugh R Wilson and Jack D Cowan. Excitatory and inhibitory interactions in localized popula-tions of model neurons. Biophysical journal, 12(1):1–24, 1972.[31] Stefanos E. Folias. Nonlinear analysis of breathing pulses in a synaptically coupled neuralnetwork. SIAM Journal on Applied Dynamical Systems, 10(2):744–787, 2011.[32] S. Coombes and M. R. Owen. Bumps, breathers, and waves in a neural network with spikefrequency adaptation. Phys. Rev. Lett., 94:148102, Apr 2005.6

Deep learning of normal form autoencoders for universal, parameter-dependent dynamics

https://ris.utwente.nl/ws/files/249970585/46.pdf

Deep learning of normal form autoencoders for universal, parameter-dependent dynamics

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information