Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed!

Implicit layer deep learning techniques, like Neural Differential Equations, have become an important modeling framework due to their ability to adapt to new problems automatically. Training a neural differential equation is effectively a search over a space of plausible dynamical systems. However, controlling the computational cost for these models is difficult since it relies on the number of steps the adaptive solver takes. Most prior works have used higher-order methods to reduce prediction timings while greatly increasing training time or reducing both training and prediction timings by relying on specific training algorithms, which are harder to use as a drop-in replacement due to strict requirements on automatic differentiation. In this manuscript, we use internal cost heuristics of adaptive differential equation solvers at stochastic time points to guide the training toward learning a dynamical system that is easier to integrate. We "close the black-box" and allow the use of our method with any adjoint technique for gradient calculations of the differential equation solution. We perform experimental studies to compare our method to global regularization to show that we attain similar performance numbers without compromising the flexibility of implementation on ordinary differential equations (ODEs) and stochastic differential equations (SDEs). We develop two sampling strategies to trade off between performance and training time. Our method reduces the number of function evaluations to 0.556-0.733x and accelerates predictions by 1.3-2x

Reduced Order Model for Chemical Kinetics: A case study with Primordial Chemical Network

Chemical kinetics plays an important role in governing the thermal evolution in reactive flows problems. The possible interactions between chemical species increase drastically with the number of species considered in the system. Various ways have been proposed before to simplify chemical networks with an aim to reduce the computational complexity of the chemical network. These techniques oftentimes require domain-knowledge experts to handcraftedly identify important reaction pathways and possible simplifications. Here, we propose a combination of autoencoder and neural ordinary differential equation to model the temporal evolution of chemical kinetics in a reduced subspace. We demonstrated that our model has achieved a close-to 10-fold speed-up compared to commonly used astro-chemistry solver for a 9-species primordial network, while maintaining 1 percent accuracy across a wide-range of density and temperature.Comment: 10 pages, 8 figures, accepted to the ICML 2022 Machine Learning for Astrophysics worksho

On the convergence of spectral deferred correction methods

In this work we analyze the convergence properties of the Spectral Deferred Correction (SDC) method originally proposed by Dutt et al. [BIT, 40 (2000), pp. 241--266]. The framework for this high-order ordinary differential equation (ODE) solver is typically described wherein a low-order approximation (such as forward or backward Euler) is lifted to higher order accuracy by applying the same low-order method to an error equation and then adding in the resulting defect to correct the solution. Our focus is not on solving the error equation to increase the order of accuracy, but on rewriting the solver as an iterative Picard integral equation solver. In doing so, our chief finding is that it is not the low-order solver that picks up the order of accuracy with each correction, but it is the underlying quadrature rule of the right hand side function that is solely responsible for picking up additional orders of accuracy. Our proofs point to a total of three sources of errors that SDC methods carry: the error at the current time point, the error from the previous iterate, and the numerical integration error that comes from the total number of quadrature nodes used for integration. The second of these two sources of errors is what separates SDC methods from Picard integral equation methods; our findings indicate that as long as difference between the current and previous iterate always gets multiplied by at least a constant multiple of the time step size, then high-order accuracy can be found even if the underlying "solver" is inconsistent the underlying ODE. From this vantage, we solidify the prospects of extending spectral deferred correction methods to a larger class of solvers to which we present some examples.Comment: 29 page

A simple, efficient, mass-conservative approach to solving Richards' equation (openRE, v1.0)

A simple numerical solution procedure – namely the method of lines combined with an off-the-shelf ordinary differential equation (ODE) solver – was shown in previous work to provide efficient, mass-conservative solutions to the pressure-head form of Richards' equation. We implement such a solution in our model openRE. We developed a novel method to quantify the boundary fluxes that reduce water balance errors without negative impacts on model runtimes – the solver flux output method (SFOM). We compare this solution with alternatives, including the classic modified Picard iteration method and the Hydrus 1D model. We reproduce a set of benchmark solutions with all models. We find that Celia's solution has the best water balance, but it can incur significant truncation errors in the simulated boundary fluxes, depending on the time steps used. Our solution has comparable runtimes to Hydrus and better water balance performance (though both models have excellent water balance closure for all the problems we considered). Our solution can be implemented in an interpreted language, such as MATLAB or Python, making use of off-the-shelf ODE solvers. We evaluated alternative SciPy ODE solvers that are available in Python and make practical recommendations about the best way to implement them for Richards' equation. There are two advantages of our approach: (i) the code is concise, making it ideal for teaching purposes; and (ii) the method can be easily extended to represent alternative properties (e.g., novel ways to parameterize the K(ψ) relationship) and processes (e.g., it is straightforward to couple heat or solute transport), making it ideal for testing alternative hypotheses

Reverse-Shooting versus Forward-Shooting over a Range of Dimensionalities

This paper investigates the properties of dynamic solutions that have been derived using the well-known reverse-shooting and forwardshooting algorithms. Given an arbitrary large-scale model about which we have limited information, how successful are the algorithms likely to be in solving this model? We address this question using a range of investment models, both linear and non-linear. By extending the investment models to allow for multi-dimensional specifications of the capital stock, we are able to examine the computational efficiency of the competing algorithms as the dimensionality of the capital stock is allowed to increase. Our approach provides insights into how the complexity of the solutions to a broad range of macroeconomic models increases with the dimensionality of the models.Macroeconomics; Reverse-shooting; Forward-shooting; Saddlepath instability; Computational techniques; Investment models.