An Ensemble Kalman-Particle Predictor-Corrector Filter for Non-Gaussian Data Assimilation

An Ensemble Kalman Filter (EnKF, the predictor) is used make a large change in the state, followed by a Particle Filer (PF, the corrector) which assigns importance weights to describe non-Gaussian distribution. The weights are obtained by nonparametric density estimation. It is demonstrated on several numerical examples that the new predictor-corrector filter combines the advantages of the EnKF and the PF and that it is suitable for high dimensional states which are discretizations of solutions of partial differential equations.Comment: ICCS 2009, to appear; 9 pages; minor edit

Bridging the ensemble Kalman and particle filter

In many applications of Monte Carlo nonlinear filtering, the propagation step is computationally expensive, and hence, the sample size is limited. With small sample sizes, the update step becomes crucial. Particle filtering suffers from the well-known problem of sample degeneracy. Ensemble Kalman filtering avoids this, at the expense of treating non-Gaussian features of the forecast distribution incorrectly. Here we introduce a procedure which makes a continuous transition indexed by gamma in [0,1] between the ensemble and the particle filter update. We propose automatic choices of the parameter gamma such that the update stays as close as possible to the particle filter update subject to avoiding degeneracy. In various examples, we show that this procedure leads to updates which are able to handle non-Gaussian features of the prediction sample even in high-dimensional situations

Deterministic Mean-field Ensemble Kalman Filtering

The proof of convergence of the standard ensemble Kalman filter (EnKF) from Legland etal. (2011) is extended to non-Gaussian state space models. A density-based deterministic approximation of the mean-field limit EnKF (DMFEnKF) is proposed, consisting of a PDE solver and a quadrature rule. Given a certain minimal order of convergence $\kappa$ between the two, this extends to the deterministic filter approximation, which is therefore asymptotically superior to standard EnKF when the dimension $d<2\kappa$. The fidelity of approximation of the true distribution is also established using an extension of total variation metric to random measures. This is limited by a Gaussian bias term arising from non-linearity/non-Gaussianity of the model, which exists for both DMFEnKF and standard EnKF. Numerical results support and extend the theory

Health monitoring for strongly non‐linear systems using the Ensemble Kalman filter

Many structural engineering problems of practical interest involve pronounced non-linear dynamics the governing laws of which are not always clearly understood. Standard identification and damage detection techniques have difficulties in these situations which feature significant modelling errors and strongly non-Gaussian signals. This paper presents a combination of the ensemble Kalman filter and non-parametric modelling techniques to tackle structural health monitoring for non-linear systems in a manner that can readily accommodate the presence of non-Gaussian noise. Both location and time of occurrence of damage are accurately detected in spite of measurement and modelling noise. A comparison between ensemble and extended Kalman filters is also presented, highlighting the benefits of the present approach. Copyright © 2005 John Wiley & Sons, Ltd

Bridging the ensemble Kalman and particle filters

In many applications of Monte Carlo nonlinear filtering, the propagation step is computationally expensive, and hence the sample size is limited. With small sample sizes, the update step becomes crucial. Particle filtering suffers from the well-known problem of sample degeneracy. Ensemble Kalman filtering avoids this, at the expense of treating non-Gaussian features of the forecast distribution incorrectly. Here we introduce a procedure that makes a continuous transition indexed by γ∈[0,1] between the ensemble and the particle filter update. We propose automatic choices of the parameter γ such that the update stays as close as possible to the particle filter update subject to avoiding degeneracy. In various examples, we show that this procedure leads to updates that are able to handle non-Gaussian features of the forecast sample even in high-dimensional situation

Score-based Data Assimilation

Data assimilation, in its most comprehensive form, addresses the Bayesian inverse problem of identifying plausible state trajectories that explain noisy or incomplete observations of stochastic dynamical systems. Various approaches have been proposed to solve this problem, including particle-based and variational methods. However, most algorithms depend on the transition dynamics for inference, which becomes intractable for long time horizons or for high-dimensional systems with complex dynamics, such as oceans or atmospheres. In this work, we introduce score-based data assimilation for trajectory inference. We learn a score-based generative model of state trajectories based on the key insight that the score of an arbitrarily long trajectory can be decomposed into a series of scores over short segments. After training, inference is carried out using the score model, in a non-autoregressive manner by generating all states simultaneously. Quite distinctively, we decouple the observation model from the training procedure and use it only at inference to guide the generative process, which enables a wide range of zero-shot observation scenarios. We present theoretical and empirical evidence supporting the effectiveness of our method