Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations

While there is currently a lot of enthusiasm about "big data", useful data is usually "small" and expensive to acquire. In this paper, we present a new paradigm of learning partial differential equations from {\em small} data. In particular, we introduce \emph{hidden physics models}, which are essentially data-efficient learning machines capable of leveraging the underlying laws of physics, expressed by time dependent and nonlinear partial differential equations, to extract patterns from high-dimensional data generated from experiments. The proposed methodology may be applied to the problem of learning, system identification, or data-driven discovery of partial differential equations. Our framework relies on Gaussian processes, a powerful tool for probabilistic inference over functions, that enables us to strike a balance between model complexity and data fitting. The effectiveness of the proposed approach is demonstrated through a variety of canonical problems, spanning a number of scientific domains, including the Navier-Stokes, Schr\"odinger, Kuramoto-Sivashinsky, and time dependent linear fractional equations. The methodology provides a promising new direction for harnessing the long-standing developments of classical methods in applied mathematics and mathematical physics to design learning machines with the ability to operate in complex domains without requiring large quantities of data

A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems

We propose a new composite neural network (NN) that can be trained based on multi-fidelity data. It is comprised of three NNs, with the first NN trained using the low-fidelity data and coupled to two high-fidelity NNs, one with activation functions and another one without, in order to discover and exploit nonlinear and linear correlations, respectively, between the low-fidelity and the high-fidelity data. We first demonstrate the accuracy of the new multi-fidelity NN for approximating some standard benchmark functions but also a 20-dimensional function. Subsequently, we extend the recently developed physics-informed neural networks (PINNs) to be trained with multi-fidelity data sets (MPINNs). MPINNs contain four fully-connected neural networks, where the first one approximates the low-fidelity data, while the second and third construct the correlation between the low- and high-fidelity data and produce the multi-fidelity approximation, which is then used in the last NN that encodes the partial differential equations (PDEs). Specifically, in the two high-fidelity NNs a relaxation parameter is introduced, which can be optimized to combine the linear and nonlinear sub-networks. By optimizing this parameter, the present model is capable of learning both the linear and complex nonlinear correlations between the low- and high-fidelity data adaptively. By training the MPINNs, we can:(1) obtain the correlation between the low- and high-fidelity data, (2) infer the quantities of interest based on a few scattered data, and (3) identify the unknown parameters in the PDEs. In particular, we employ the MPINNs to learn the hydraulic conductivity field for unsaturated flows as well as the reactive models for reactive transport. The results demonstrate that MPINNs can achieve relatively high accuracy based on a very small set of high-fidelity data

A Tunably-Accurate Laguerre Petrov-Galerkin Spectral Method for Multi-Term Fractional Differential Equations on the Half Line

We present a new tunably-accurate Laguerre Petrov-Galerkin spectral method for solving linear multi-term fractional initial value problems with derivative orders at most one and constant coefficients on the half line. Our method results in a matrix equation of special structure which can be solved in $\mathcal{O}(N \log N)$ operations. We also take advantage of recurrence relations for the generalized associated Laguerre functions (GALFs) in order to derive explicit expressions for the entries of the stiffness and mass matrices, which can be factored into the product of a diagonal matrix and a lower-triangular Toeplitz matrix. The resulting spectral method is efficient for solving multi-term fractional differential equations with arbitrarily many terms. We apply this method to a distributed order differential equation, which is approximated by linear multi-term equations through the Gauss-Legendre quadrature rule. We provide numerical examples demonstrating the spectral convergence and linear complexity of the method

A Riesz basis Galerkin method for the tempered fractional Laplacian

The fractional Laplacian $\Delta^{\beta/2}$ is the generator of $\beta$-stable L\'evy process, which is the scaling limit of the L\'evy fight. Due to the divergence of the second moment of the jump length of the L\'evy fight it is not appropriate as a physical model in many practical applications. However, using a parameter $\lambda$ to exponentially temper the isotropic power law measure of the jump length leads to the tempered L\'evy fight, which has finite second moment. For short time the tempered L\'evy fight exhibits the dynamics of L\'evy fight while after sufficiently long time it turns to normal diffusion. The generator of tempered $\beta$-stable L\'evy process is the tempered fractional Laplacian $(\Delta+\lambda)^{\beta/2}$ [W.H. Deng, B.Y. Li, W.Y. Tian, and P.W. Zhang, Multiscale Model. Simul., in press, 2017]. In the current work, we present new computational methods for the tempered fractional Laplacian equation, including the cases with the homogeneous and nonhomogeneous generalized Dirichlet type boundary conditions. We prove the well-posedness of the Galerkin weak formulation and provide convergence analysis of the single scaling B-spline and multiscale Riesz bases finite element methods. We propose a technique for efficiently generating the entries of the dense stiffness matrix and for solving the resulting algebraic equation by preconditioning. We also present several numerical experiments to verify the theoretical results.Comment: 28 pages, 2 figure

Neural-net-induced Gaussian process regression for function approximation and PDE solution

Neural-net-induced Gaussian process (NNGP) regression inherits both the high expressivity of deep neural networks (deep NNs) as well as the uncertainty quantification property of Gaussian processes (GPs). We generalize the current NNGP to first include a larger number of hyperparameters and subsequently train the model by maximum likelihood estimation. Unlike previous works on NNGP that targeted classification, here we apply the generalized NNGP to function approximation and to solving partial differential equations (PDEs). Specifically, we develop an analytical iteration formula to compute the covariance function of GP induced by deep NN with an error-function nonlinearity. We compare the performance of the generalized NNGP for function approximations and PDE solutions with those of GPs and fully-connected NNs. We observe that for smooth functions the generalized NNGP can yield the same order of accuracy with GP, while both NNGP and GP outperform deep NN. For non-smooth functions, the generalized NNGP is superior to GP and comparable or superior to deep NN

Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks

One of the open problems in scientific computing is the long-time integration of nonlinear stochastic partial differential equations (SPDEs). We address this problem by taking advantage of recent advances in scientific machine learning and the dynamically orthogonal (DO) and bi-orthogonal (BO) methods for representing stochastic processes. Specifically, we propose two new Physics-Informed Neural Networks (PINNs) for solving time-dependent SPDEs, namely the NN-DO/BO methods, which incorporate the DO/BO constraints into the loss function with an implicit form instead of generating explicit expressions for the temporal derivatives of the DO/BO modes. Hence, the proposed methods overcome some of the drawbacks of the original DO/BO methods: we do not need the assumption that the covariance matrix of the random coefficients is invertible as in the original DO method, and we can remove the assumption of no eigenvalue crossing as in the original BO method. Moreover, the NN-DO/BO methods can be used to solve time-dependent stochastic inverse problems with the same formulation and computational complexity as for forward problems. We demonstrate the capability of the proposed methods via several numerical examples: (1) A linear stochastic advection equation with deterministic initial condition where the original DO/BO method would fail; (2) Long-time integration of the stochastic Burgers' equation with many eigenvalue crossings during the whole time evolution where the original BO method fails. (3) Nonlinear reaction diffusion equation: we consider both the forward and the inverse problem, including noisy initial data, to investigate the flexibility of the NN-DO/BO methods in handling inverse and mixed type problems. Taken together, these simulation results demonstrate that the NN-DO/BO methods can be employed to effectively quantify uncertainty propagation in a wide range of physical problems

Petrov-Galerkin and Spectral Collocation Methods for distributed Order Differential Equations

Distributed order fractional operators offer a rigorous tool for mathematical modelling of multi-physics phenomena, where the differential orders are distributed over a range of values rather than being just a fixed integer/fraction as it is in standard/fractional ODEs/PDEs. We develop two spectrally-accurate schemes, namely a Petrov-Galerkin spectral method and a spectral collocation method for distributed order fractional differential equations. These schemes are developed based on the fractional Sturm-Liouville eigen-problems (FSLPs). In the Petrov-Galerkin method, we employ fractional (non-polynomial) basis functions, called \textit{Jacobi poly-fractonomials}, which are the eigenfunctions of the FSLP of first kind, while, we employ another space of test functions as the span of poly-fractonomial eigenfunctions of the FSLP of second kind. We define the underlying \textit{distributed Sobolev space} and the associated norms, where we carry out the corresponding discrete stability and error analyses of the proposed scheme. In the collocation scheme, we employ fractional (non-polynomial) Lagrange interpolants satisfying the Kronecker delta property at the collocation points. Subsequently, we obtain the corresponding distributed differentiation matrices to be employed in the discretization of the strong problem. We perform systematic numerical tests to demonstrate the efficiency and conditioning of each method

Inferring solutions of differential equations using noisy multi-fidelity data

For more than two centuries, solutions of differential equations have been obtained either analytically or numerically based on typically well-behaved forcing and boundary conditions for well-posed problems. We are changing this paradigm in a fundamental way by establishing an interface between probabilistic machine learning and differential equations. We develop data-driven algorithms for general linear equations using Gaussian process priors tailored to the corresponding integro-differential operators. The only observables are scarce noisy multi-fidelity data for the forcing and solution that are not required to reside on the domain boundary. The resulting predictive posterior distributions quantify uncertainty and naturally lead to adaptive solution refinement via active learning. This general framework circumvents the tyranny of numerical discretization as well as the consistency and stability issues of time-integration, and is scalable to high-dimensions.Comment: 19 pages, 3 figure

Nonlocal flocking dynamics: Learning the fractional order of PDEs from particle simulations

Flocking refers to collective behavior of a large number of interacting entities, where the interactions between discrete individuals produce collective motion on the large scale. We employ an agent-based model to describe the microscopic dynamics of each individual in a flock, and use a fractional PDE to model the evolution of macroscopic quantities of interest. The macroscopic models with phenomenological interaction functions are derived by applying the continuum hypothesis to the microscopic model. Instead of specifying the fPDEs with an ad hoc fractional order for nonlocal flocking dynamics, we learn the effective nonlocal influence function in fPDEs directly from particle trajectories generated by the agent-based simulations. We demonstrate how the learning framework is used to connect the discrete agent-based model to the continuum fPDEs in 1D and 2D nonlocal flocking dynamics. In particular, a Cucker-Smale particle model is employed to describe the microscale dynamics of each individual, while Euler equations with nonlocal interaction terms are used to compute the evolution of macroscale quantities. The trajectories generated by the particle simulations mimic the field data of tracking logs that can be obtained experimentally. They can be used to learn the fractional order of the influence function using a Gaussian process regression model implemented with the Bayesian optimization. We show that the numerical solution of the learned Euler equations solved by the finite volume scheme can yield correct density distributions consistent with the collective behavior of the agent-based system. The proposed method offers new insights on how to scale the discrete agent-based models to the continuum-based PDE models, and could serve as a paradigm on extracting effective governing equations for nonlocal flocking dynamics directly from particle trajectories.Comment: 22 pages, 7 figure

Hidden Fluid Mechanics: A Navier-Stokes Informed Deep Learning Framework for Assimilating Flow Visualization Data

We present hidden fluid mechanics (HFM), a physics informed deep learning framework capable of encoding an important class of physical laws governing fluid motions, namely the Navier-Stokes equations. In particular, we seek to leverage the underlying conservation laws (i.e., for mass, momentum, and energy) to infer hidden quantities of interest such as velocity and pressure fields merely from spatio-temporal visualizations of a passive scaler (e.g., dye or smoke), transported in arbitrarily complex domains (e.g., in human arteries or brain aneurysms). Our approach towards solving the aforementioned data assimilation problem is unique as we design an algorithm that is agnostic to the geometry or the initial and boundary conditions. This makes HFM highly flexible in choosing the spatio-temporal domain of interest for data acquisition as well as subsequent training and predictions. Consequently, the predictions made by HFM are among those cases where a pure machine learning strategy or a mere scientific computing approach simply cannot reproduce. The proposed algorithm achieves accurate predictions of the pressure and velocity fields in both two and three dimensional flows for several benchmark problems motivated by real-world applications. Our results demonstrate that this relatively simple methodology can be used in physical and biomedical problems to extract valuable quantitative information (e.g., lift and drag forces or wall shear stresses in arteries) for which direct measurements may not be possible