An error estimate of Gaussian Recursive Filter in 3Dvar problem

Computational kernel of the three-dimensional variational data assimilation (3D-Var) problem is a linear system, generally solved by means of an iterative method. The most costly part of each iterative step is a matrix-vector product with a very large covariance matrix having Gaussian correlation structure. This operation may be interpreted as a Gaussian convolution, that is a very expensive numerical kernel. Recursive Filters (RFs) are a well known way to approximate the Gaussian convolution and are intensively applied in the meteorology, in the oceanography and in forecast models. In this paper, we deal with an oceanographic 3D-Var data assimilation scheme, named OceanVar, where the linear system is solved by using the Conjugate Gradient (GC) method by replacing, at each step, the Gaussian convolution with RFs. Here we give theoretical issues on the discrete convolution approximation with a first order (1st-RF) and a third order (3rd-RF) recursive filters. Numerical experiments confirm given error bounds and show the benefits, in terms of accuracy and performance, of the 3-rd RF.Comment: 9 page

Controlling instabilities along a 3DVar analysis cycle by assimilating in the unstable subspace: a comparison with the EnKF

A hybrid scheme obtained by combining 3DVar with the Assimilation in the Unstable Subspace (3DVar-AUS) is tested in a QG model, under perfect model conditions, with a fixed observational network, with and without observational noise. The AUS scheme, originally formulated to assimilate adaptive observations, is used here to assimilate the fixed observations that are found in the region of local maxima of BDAS vectors (Bred vectors subject to assimilation), while the remaining observations are assimilated by 3DVar. The performance of the hybrid scheme is compared with that of 3DVar and of an EnKF. The improvement gained by 3DVar-AUS and the EnKF with respect to 3DVar alone is similar in the present model and observational configuration, while 3DVar-AUS outperforms the EnKF during the forecast stage. The 3DVar-AUS algorithm is easy to implement and the results obtained in the idealized conditions of this study encourage further investigation toward an implementation in more realistic contexts

Retrieval of moisture from GPS slant-path water vapor observations using 3DVAR and its impact on the prediction of convection initiation and precipitation.

The anisotropic explicit filter is computationally expensive in both CPU time and memory usage. Therefore the implicit recursive filter which is computationally much more efficient is implemented in our 3DVAR system, even though its implementation is significantly more complicated. A similar set of water vapor analysis experiments using the recursive filters is performed. The analyses thus obtained are generally comparable to or better than those obtained using the corresponding explicit filters. In addition, the sensitivity of the analyses to the spatial de-correlation scales of the background error is systematically examined.A set of high-resolution numerical experiments is conducted using the Advanced Regional Prediction System (ARPS) for a case that occurred on 12 June, 2002 and involved multiple initiations of convection. The results are verified against the radar composite reflectivity in detail. It is shown that the model performs reasonably well on predicting the initiation timing and location and the subsequent storm evolution for up to 7 hours. Using the most realistic simulation of this case as the 'truth', simulated SWV data and surface moisture observations are generated to perform a set of Observing System Simulation Experiments (OSSEs) using our 3DVAR system with recursive filters. The preliminary results illustrate that convection initiation (CI) without strong low-level mesoscale forcing is highly sensitive to the moisture initial condition and the use of SWV and surface data improves the moisture analysis and thus the prediction of CI and precipitation. The enhanced moisture analysis obtained from the use of anisotropic background error further improves the precipitation forecast though it does not lead to positive impact on the prediction of exact timing and location of the CI due to its high sensitivity to very small-scale moisture structures.The 3DVAR system developed in this study is based on a terrain-following coordinate. A non-negative water vapor weak constraint is included in the cost function. The background term and its associated background error covariance are considered in the system and the latter is modeled using explicit or implicit recursive spatial filters. Most importantly, a direct way to estimate a flow-dependent background error covariance based on the idea of Riishojgaard is proposed for the moisture analysis. The explicit spatial filter first is implemented with both isotropic and anisotropic options. It is demonstrated that this system is robust on deriving mesoscale moisture structures from the GPS SWV and surface observations and the analysis is improved when the anisotropic background error covariance is used. Sensitivity experiments show that surface moisture data are important for the analysis near ground and a vertical filter is essential to obtain an accurate analysis near the surface. The positive impact of flow-dependent background error is enhanced when the density of GPS receiver network is lower.The accurate prediction of convection initiation and the subsequent precipitation in a cloud-resolving numerical model is highly dependent on the precise estimate of the three-dimensional moisture in the initial condition because water vapor is directly involved in the formation of clouds and precipitation. However, the water vapor is currently poorly characterized due to its high variability in space and time. A three-dimensional variational analysis system (3DVAR) is developed in this dissertation to retrieve the moisture field from simulated ground-based GPS slant-path integrated water vapor (SWV) data that are potentially available at high temporal and spatial resolutions

Data Assimilation Fundamentals

This open-access textbook's significant contribution is the unified derivation of data-assimilation techniques from a common fundamental and optimal starting point, namely Bayes' theorem. Unique for this book is the "top-down" derivation of the assimilation methods. It starts from Bayes theorem and gradually introduces the assumptions and approximations needed to arrive at today's popular data-assimilation methods. This strategy is the opposite of most textbooks and reviews on data assimilation that typically take a bottom-up approach to derive a particular assimilation method. E.g., the derivation of the Kalman Filter from control theory and the derivation of the ensemble Kalman Filter as a low-rank approximation of the standard Kalman Filter. The bottom-up approach derives the assimilation methods from different mathematical principles, making it difficult to compare them. Thus, it is unclear which assumptions are made to derive an assimilation method and sometimes even which problem it aspires to solve. The book's top-down approach allows categorizing data-assimilation methods based on the approximations used. This approach enables the user to choose the most suitable method for a particular problem or application. Have you ever wondered about the difference between the ensemble 4DVar and the "ensemble randomized likelihood" (EnRML) methods? Do you know the differences between the ensemble smoother and the ensemble-Kalman smoother? Would you like to understand how a particle flow is related to a particle filter? In this book, we will provide clear answers to several such questions. The book provides the basis for an advanced course in data assimilation. It focuses on the unified derivation of the methods and illustrates their properties on multiple examples. It is suitable for graduate students, post-docs, scientists, and practitioners working in data assimilation

Estimation and Impact of Nonuniform Horizontal Correlation Length Scales for Global Ocean Physical Analyses

Optimally modeling background-error horizontal correlations is crucial in ocean data assimilation. This paper investigates the impact of releasing the assumption of uniform background-error correlations in a global ocean variational analysis system. Spatially varying horizontal correlations are introduced in the recursive filter operator, which is used for modeling horizontal covariances in the Centro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC) analysis system. The horizontal correlation length scales (HCLSs) were defined on the full three-dimensional model space and computed from both a dataset of monthly anomalies with respect to the monthly climatology and through the so-called National Meteorological Center (NMC) method. Different formulas for estimating the correlation length scale are also discussed and applied to the two forecast error datasets. The new formulation is tested within a 12-yr period (2000–11) in the ½° resolution system. The comparison with the data assimilation system using uniform background-error horizontal correlations indicates the superiority of the former, especially in eddy-dominated areas. Verification skill scores report a significant reduction of RMSE, and the use of nonuniform length scales improves the representation of the eddy kinetic energy at midlatitudes, suggesting that uniform, latitude, or Rossby radius-dependent formulations are insufficient to represent the geographical variations of the background-error correlations. Furthermore, a small tuning of the globally uniform value of the length scale was found to have a small impact on the analysis system. The use of either anomalies or NMC-derived correlation length scales also has a marginal effect with respect to the use of nonuniform HCLSs. On the other hand, the application of overestimated length scales has proved to be detrimental to the analysis system in all areas and for all parameters

Data Assimilation Fundamentals

This open-access textbook's significant contribution is the unified derivation of data-assimilation techniques from a common fundamental and optimal starting point, namely Bayes' theorem. Unique for this book is the "top-down" derivation of the assimilation methods. It starts from Bayes theorem and gradually introduces the assumptions and approximations needed to arrive at today's popular data-assimilation methods. This strategy is the opposite of most textbooks and reviews on data assimilation that typically take a bottom-up approach to derive a particular assimilation method. E.g., the derivation of the Kalman Filter from control theory and the derivation of the ensemble Kalman Filter as a low-rank approximation of the standard Kalman Filter. The bottom-up approach derives the assimilation methods from different mathematical principles, making it difficult to compare them. Thus, it is unclear which assumptions are made to derive an assimilation method and sometimes even which problem it aspires to solve. The book's top-down approach allows categorizing data-assimilation methods based on the approximations used. This approach enables the user to choose the most suitable method for a particular problem or application. Have you ever wondered about the difference between the ensemble 4DVar and the "ensemble randomized likelihood" (EnRML) methods? Do you know the differences between the ensemble smoother and the ensemble-Kalman smoother? Would you like to understand how a particle flow is related to a particle filter? In this book, we will provide clear answers to several such questions. The book provides the basis for an advanced course in data assimilation. It focuses on the unified derivation of the methods and illustrates their properties on multiple examples. It is suitable for graduate students, post-docs, scientists, and practitioners working in data assimilation

Long-time asymptotics of the filtering distribution for partially observed chaotic dynamical systems

The filtering distribution is a time-evolving probability distribution on the state of a dynamical system given noisy observations. We study the large-time asymptotics of this probability distribution for discrete-time, randomly initialized signals that evolve according to a deterministic map Ψ. The observations are assumed to comprise a low-dimensional projection of the signal, given by an operator P, subject to additive noise. We address the question of whether these observations contain sufficient information to accurately reconstruct the signal. In a general framework, we establish conditions on Ψ and P under which the filtering distributions concentrate around the signal in the small-noise, long-time asymptotic regime. Linear systems, the Lorenz ’63 and ’96 models, and the Navier–Stokes equation on a two-dimensional torus are within the scope of the theory. Our main findings come as a by-product of computable bounds, of independent interest, for suboptimal filters based on new variants of the 3DVAR filtering algorith

AN EVALUATION OF HYBRID VARIATIONAL-ENSEMBLE DATA ASSIMILATION FOR THE NCEP GFS

Several variants of hybrid data assimilation algorithms have been developed and tested within recent years, particularly for numerical weather prediction (NWP). The hybrid algorithms are designed to combine the strengths of variational and ensemble-based techniques while at the same time attempting to mitigate their weaknesses. One such variational-based algorithm is under development for use with the National Centers for Environmental Prediction's (NCEP) global forecast system (GFS) model. In this work, we attempt to better understand the impact of utilizing a hybrid scheme on the quality of analyses and subsequent forecasts, as well as explore alternative extensions to make better use of the ensemble information within the variational solver. A series of Observing System Simulation Experiments (OSSEs) are carried out. It is demonstrated that analysis and subsequent forecast errors are generally reduced in a 3D-hybrid scheme relative to 3DVAR. Several variational-based 4D extensions are proposed and tested, including the use of a variety of dynamic constraints. A simple approach for hybridizing the 4D-ensemble with a time-invariant contribution is proposed and tested. The 4D variants are shown to be superior to the 3D-hybrid, with positive contributions from static B as well as the dynamic constraint formulations. It is clear from both the 3D and 4D experiments that more sophisticated methods for dealing with inflation and localization in the ensemble update are needed even within the hybrid paradigm. Lastly, a method for applying piecewise scale-dependent weights is proposed and successfully tested. The 3D OSSE-based results are also compared with results from an experiment using real observations to corroborate the findings. It is found that in general, most of the results are comparable, though the positive impact in the real system is more consistent and impressive

3D Variational Analysis In Subsurface Contaminant Transport Model

Modeling of contaminant transport in a subsurface environment by a numerical model deviates from the real world environment because of the highly heterogeneous nature of the subsurface environment. In this study, the data assimilation techniques are integrated with the numerical model and are applied to the subsurface environment to predict the contaminant transport. The Forward Time Center Space (FTCS) model is used as a numerical approach to solve the classical advection-dispersion-reaction transport equation and the Kalman Filter, Ensemble Kalman Filter (EnKF) and 3D Variational (3DVAR) analysis are used for data assimilation purpose. A hybrid scheme, termed as EnKF-3DVAR is developed using EnKF and 3DVAR analysis. The EnKF is a Monte Carlo based sequential data assimilation technique that divides the state vector into N number of ensembles rather than computing one state vector. The 3DVAR analysis uses the EnKF mean state vector as the background state and uses a cost function to find out an optimal estimate of that EnKF mean state vector. The simulation is run using an ensemble size of 100 members. A Root Mean Square Error (RMSE) profile is used to evaluate the prediction accuracy of the models. This study shows that state predictions are better for both the EnKF and EnKF- 3DVAR when compared to those of the numerical and KF solutions

A Revised Scheme to Compute Horizontal Covariances in an Oceanographic 3D-VAR Assimilation System.

We propose an improvement of an oceanographic three dimensional variational assimilation scheme (3D-VAR), named OceanVar, by introducing a recursive filter (RF) with the third order of accuracy (3rd-RF), instead of an RFwith first order of accuracy (1st-RF), to approximate horizontal Gaussian covariances. An advantage of the proposed scheme is that the CPU's time can be substantially reduced with benefits on the large scale applications. Experiments estimating the impact of 3rd-RF are performed by assimilating oceanographic data in two realistic oceanographic applications. The results evince benefits in terms of assimilation process computational time, accuracy of the Gaussian correlation modeling, and show that the 3rd-RF is a suitable tool for operational data assimilation