Inferring unknown unknowns: Regularized bias-aware ensemble Kalman filter

Because of physical assumptions and numerical approximations, low-order models are affected by uncertainties in the state and parameters, and by model biases. Model biases, also known as model errors or systematic errors, are difficult to infer because they are `unknown unknowns', i.e., we do not necessarily know their functional form a priori. With biased models, data assimilation methods may be ill-posed because either (i) they are 'bias-unaware' because the estimators are assumed unbiased, (ii) they rely on an a priori parametric model for the bias, or (iii) they can infer model biases that are not unique for the same model and data. First, we design a data assimilation framework to perform combined state, parameter, and bias estimation. Second, we propose a mathematical solution with a sequential method, i.e., the regularized bias-aware ensemble Kalman Filter (r-EnKF), which requires a model of the bias and its gradient (i.e., the Jacobian). Third, we propose an echo state network as the model bias estimator. We derive the Jacobian of the network, and design a robust training strategy with data augmentation to accurately infer the bias in different scenarios. Fourth, we apply the r-EnKF to nonlinearly coupled oscillators (with and without time-delay) affected by different forms of bias. The r-EnKF infers in real-time parameters and states, and a unique bias. The applications that we showcase are relevant to acoustics, thermoacoustics, and vibrations; however, the r-EnKF opens new opportunities for combined state, parameter and bias estimation for real-time and on-the-fly prediction in nonlinear systems.Comment: 22 Figure

Multivariate Analysis in Metabolomics

Metabolomics aims to provide a global snapshot of all small-molecule metabolites in cells and biological fluids, free of observational biases inherent to more focused studies of metabolism. However, the staggeringly high information content of such global analyses introduces a challenge of its own; efficiently forming biologically relevant conclusions from any given metabolomics dataset indeed requires specialized forms of data analysis. One approach to finding meaning in metabolomics datasets involves multivariate analysis (MVA) methods such as principal component analysis (PCA) and partial least squares projection to latent structures (PLS), where spectral features contributing most to variation or separation are identified for further analysis. However, as with any mathematical treatment, these methods are not a panacea; this review discusses the use of multivariate analysis for metabolomics, as well as common pitfalls and misconceptions

APPLICATIONS OF ENSEMBLE KALMAN FILTER DATA ASSIMILATION: FROM CONVECTIVE THUNDERSTORMS TO HURRICANES

For the numerical prediction of severe thunderstorm and hurricane, data assimilation is one of the necessary tools to obtain accurate initial conditions. Ensemble Kalman filter (EnKF) is a state of the art data assimilation algorithm, with the advantage of using flow-dependent error covariance information and retrieving unobserved model quantities. In this dissertation, EnKF is first employed to assimilate additional surface observation in the presence of radar data for severe thunderstorm analysis and prediction with Observing System Simulation Experiments (OSSEs). The EnKF is then used to assimilate real coastal WSR-88D radar observations for Hurricane Ike (2008), and the impact of radar data on the analysis and forecast is investigated.Due to the earth curvature effect and the non-zero elevation of the lowest scan of ground-based radars, low-level coverage of radar data decreases as distance from the radar increases, causing loss of coverage for important low-level features including the cold pool and gust front. Observations from surface networks are expected to help fill such low-level data gaps. To investigate the impact of additional surface observations on the analysis and forecast of convective storms, a series of OSSEs are performed using the ARPS model and its EnKF data assimilation system.When the radar is located at a significant distance (e.g., the 115 and 185 km distances considered) from the main convective storm, a clear positive impact on the storm analysis and forecast is achieved by assimilating surface observations with a spacing of about 20 km. When the radar is located just 45 km from the storm center, a network spacing of 6 km is needed to achieve any noticeable positive impact. The impact of surface data in terms of relative error reduction increases linearly with decreased surface network spacing until the spacing is close to the grid interval of truth simulation. Assimilating observations from a coarser network over a longer period of time helps to achieve a similar level of impact as would be seen from a network of higher density.The error correlation fields derived from the forecast ensemble exhibit dynamically consistent structures. Through flow-dependent error covariance and dynamical interactions in the prediction model, the surface observations not only correct the surface errors, but also improve analyses of state variables at the mid- and upper levels. Given typical observation error, surface wind observations produce the largest positive impact, followed by temperature measurements. Pressure measurements produce the least impact. Assimilating all surface observation variables together yields the largest impact.The impact of surface data is sustained or even amplified during subsequent forecasts when their impact on the analysis is significant.In the second part of this dissertation, EnKF assimilation and forecasting experiments are performed for the case of Hurricane Ike (2008), the third most destructive hurricane hitting the United States. Data from two coastal WSR-88D radars were carefully quality controlled, including automatic and manual velocity dealiasing. For the control experiment, 32 ensemble members are used in the EnKF system, and reflectivity (Z) and radial velocity (Vr) data from the two coastal radars are assimilated at 10-minute intervals over a 2-hour period shortly before Ike made landfall.Compared to the corresponding NCEP GFS analysis, the assimilation resulted in a much improved vortex intensity at the final analysis time, although it is still weaker than observed. Compared to the forecast starting from GFS analysis at the same initial time, the forecast intensity, track and structure of Ike over a 12 hour period are improved in both deterministic and ensemble mean forecasts. The ensemble spread is well maintained with the help of multiplicative covariance inflation and posterior additive perturbations during the assimilation cycles. Assimilation of either Vr or Z alone leads to improvement in hurricane intensity, track and quantitative precipitation forecast. Vr leads to more improvement in intensity and track forecast, emphasizing more importance of Vr data. Ensemble forecast has shown uncertainty growth in track forecast but not in intensity forecast. 30-minute assimilation interval has the similar results with 10-minute assimilation interval and 60-minute assimilation interval shows weaker intensity forecast.Assimilation of additional minimum mean sea level pressure from best track data together with Z leads to further improvement in intensity and track forecast compared to assimilating Z alone. Assimilating minimum sea level pressure in addition to Vr leads to track forecast improvement but only small improvement in intensity analysis and forecast

Forecasting: theory and practice

Forecasting has always been at the forefront of decision making and planning. The uncertainty that surrounds the future is both exciting and challenging, with individuals and organisations seeking to minimise risks and maximise utilities. The large number of forecasting applications calls for a diverse set of forecasting methods to tackle real-life challenges. This article provides a non-systematic review of the theory and the practice of forecasting. We provide an overview of a wide range of theoretical, state-of-the-art models, methods, principles, and approaches to prepare, produce, organise, and evaluate forecasts. We then demonstrate how such theoretical concepts are applied in a variety of real-life contexts. We do not claim that this review is an exhaustive list of methods and applications. However, we wish that our encyclopedic presentation will offer a point of reference for the rich work that has been undertaken over the last decades, with some key insights for the future of forecasting theory and practice. Given its encyclopedic nature, the intended mode of reading is non-linear. We offer cross-references to allow the readers to navigate through the various topics. We complement the theoretical concepts and applications covered by large lists of free or open-source software implementations and publicly-available databases

Forecasting: theory and practice

Forecasting has always been at the forefront of decision making and planning. The uncertainty that surrounds the future is both exciting and challenging, with individuals and organisations seeking to minimise risks and maximise utilities. The large number of forecasting applications calls for a diverse set of forecasting methods to tackle real-life challenges. This article provides a non-systematic review of the theory and the practice of forecasting. We provide an overview of a wide range of theoretical, state-of-the-art models, methods, principles, and approaches to prepare, produce, organise, and evaluate forecasts. We then demonstrate how such theoretical concepts are applied in a variety of real-life contexts. We do not claim that this review is an exhaustive list of methods and applications. However, we wish that our encyclopedic presentation will offer a point of reference for the rich work that has been undertaken over the last decades, with some key insights for the future of forecasting theory and practice. Given its encyclopedic nature, the intended mode of reading is non-linear. We offer cross-references to allow the readers to navigate through the various topics. We complement the theoretical concepts and applications covered by large lists of free or open-source software implementations and publicly-available databases.info:eu-repo/semantics/publishedVersio

Forecasting: theory and practice

Forecasting has always been in the forefront of decision making and planning. The uncertainty that surrounds the future is both exciting and challenging, with individuals and organisations seeking to minimise risks and maximise utilities. The lack of a free-lunch theorem implies the need for a diverse set of forecasting methods to tackle an array of applications. This unique article provides a non-systematic review of the theory and the practice of forecasting. We offer a wide range of theoretical, state-of-the-art models, methods, principles, and approaches to prepare, produce, organise, and evaluate forecasts. We then demonstrate how such theoretical concepts are applied in a variety of real-life contexts, including operations, economics, finance, energy, environment, and social good. We do not claim that this review is an exhaustive list of methods and applications. The list was compiled based on the expertise and interests of the authors. However, we wish that our encyclopedic presentation will offer a point of reference for the rich work that has been undertaken over the last decades, with some key insights for the future of the forecasting theory and practice

Assimilation of radar reflectivity volumes through a LETKF scheme for a high-resolution numerical weather prediction model

Despite the continuous improvements in numerical weather prediction (NWP) models, the quantitative precipitation forecast (QPF) is still a challenge. A crucial role in the accuracy of QPF is played by data assimilation, the technique whereby initial conditions for an NWP model are generated by combining observations of the state of the atmosphere and a previous forecast of the model itself. In this work, the direct assimilation of radar reflectivity volumes, which is still at a preliminary stage in operational frameworks, is evaluated. This is carried out using a local ensemble transform Kalman filter (LETKF) scheme and employing the COSMO-2I model, the configuration of the convection-permitting model of the COnsortium for Small-scale MOdelling (COSMO) adopted at the Regional Agency for Prevention, Environment and Energy of Emilia-Romagna region (ARPAE) to provide high-resolution weather forecasts over Italy. The crucial aspects of the assimilation of this type of observation are investigated, in particular concerning the length of the assimilation window, the estimation of the observation error and the configuration of the radar operator employed to simulate reflectivity observations from the prognostic model fields. Taking advantage of the results obtained from this investigation, a set-up for the direct assimilation of reflectivity volumes suitable for an operational implementation is defined. Accuracy of QPF and of other forecast variables obtained with this set-up is compared to that obtained with the current operational set-up employed at ARPAE to generate the initial conditions of COSMO-2I, in which radar-estimated precipitation is assimilated through a latent heat nudging scheme. Results of this comparison, which is the most extended ever performed in terms of number of forecasts involved and in the number of verification scores employed, suggest that time is ripe to directly assimilate reflectivity volumes in an operational framework using an ensemble Kalman filter scheme

HYBRID EN3DVAR RADAR DATA ASSIMILATION AND COMPARISONS WITH 3DVAR AND ENKF WITH OSSES AND A REAL CASE

Studies have shown advantages of the hybrid ensemble-variational data assimilation (DA) algorithms over pure ensemble or variational algorithms, although such advantages at the convective scale, in the presence of complex ice microphysics and for radar data assimilation, have not yet been clearly demonstrated, if the advantages do exist. A hybrid ensemble-3DVar (En3DVar) system is developed recently based on the ARPS 3DVar and EnKF systems at the Center for Analysis and Prediction of Storms (CAPS). In this dissertation, hybrid En3DVar is compared with 3DVar, EnKF, and pure En3DVar for radar DA through observing system simulation experiments (OSSEs) under both perfect and imperfect model assumptions. It is also applied to a real case including multiple tornadic supercells. For the real case, radar radial velocity and reflectivity data are assimilated every 5 minutes for 1 hour that is followed by short-term forecasts. DfEnKF that updates a single deterministic background forecast using the EnKF updating algorithm is introduced to have an algorithm-wise parallel comparison between EnKF and pure En3DVar. In the perfect-model OSSEs, DfEnKF and pure En3DVar are compared and are found to perform differently when using the same localization radii. The serial (EnKF) versus global (pure En3DVar) nature of the algorithms, and direct filter update (EnKF) versus variational minimization (En3DVar) are the major reasons for the differences. Hybrid En3DVar for radar DA is also compared with 3DVar, EnKF, DfEnKF, and pure En3DVar. Experiments are conducted first to obtain the optimal configurations for different algorithms before they are compared; the optimal configurations include the optimal background decorrelation scales for 3DVar, optimal localization radii for EnKF, DfEnKF, and pure En3DVar, as well as the optimal hybrid weights for hybrid En3DVar. When the algorithms are tuned optimally, hybrid En3DVar does not outperform EnKF or pure En3DVar, although their analyses are all much better than 3DVar. When ensemble background error covariance is a good estimation of the true error distribution, pure ensemble-based DA methods can do a good job, and the advantage of including static background error covariance B in hybrid DA is not obvious. In the imperfect-model OSSEs, model errors are introduced by using different microphysical schemes in the truth run (Lin scheme) and in the ensemble forecasts (WSM6 scheme). Experiments are conducted to obtain the optimal configurations for different algorithms, similar to those in perfect-model OSSEs. Hybrid En3DVar is then found to outperform EnKF and pure En3DVar (3DVar) for better capturing the hail analyses below the freezing level (intensity of the storm). The advantage of hybrid En3DVar over pure ensemble-based methods is most obvious when ensemble background errors are systematically underestimated. In addition, the impact of adding a mass continuity constraint in 3DVar, pure and hybrid En3DVar is also examined. Overall, adding the mass continuity constraint improving the analyses by producing a little stronger vertical velocity analyses that are much closer to the truth and by smoothing noise present in the velocity and hydrometer fields. Finally, the ARPS hybrid En3DVar system is applied to the assimilation of radar data for a real tornadic supercell storm. Hybrid En3DVar is compared with 3DVar, EnKF, DfEnKF, and pure En3DVar based on both objective verification and the analyses and forecasts of storm intensity and structures. Hybrid En3DVar with 75% weight of static B clearly outperforms 3DVar in better capturing the hook echo structure and rotating updraft in the forecasts, and outperforms EnKF and DfEnKF in better capturing forecast reflectivity between 35 and 45 dBZ. The low-level mesocyclone is better forecast by hybrid En3DVar than by other methods, suggesting stronger rotations and a larger tornado threat in the forecast. In perfect-model OSSEs, only precipitation reflectivity (≥ 5 dBZ) is assimilated. In the imperfect-model OSSEs and the real data case, the clear-air reflectivity (< 5 dBZ) is also assimilated to help suppress spurious storms. The assimilation of clear-air reflectivity by the variational DA algorithms is found to seriously degrade the analyses in storm region, with the intensity of the reflectivity analysis being much weaker than that assimilating reflectivity larger than 5 dBZ only. When using hydrometeor mixing ratios as the control variables, the gradient of the cost function becomes extremely large when background reflectivity is small. In the imperfect-model OSSEs, a double-pass procedure is proposed and found to help alleviate this problem. In the real data case, an alternative way of using the logarithmic hydrometeor mixing ratios as the control variables is found to be a better solution to the problem. In such a case, the excessively large gradient of the cost function is avoided