Controlling overestimation of error covariance in ensemble Kalman filters with sparse observations: A variance limiting Kalman filter

We consider the problem of an ensemble Kalman filter when only partial observations are available. In particular we consider the situation where the observational space consists of variables which are directly observable with known observational error, and of variables of which only their climatic variance and mean are given. To limit the variance of the latter poorly resolved variables we derive a variance limiting Kalman filter (VLKF) in a variational setting. We analyze the variance limiting Kalman filter for a simple linear toy model and determine its range of optimal performance. We explore the variance limiting Kalman filter in an ensemble transform setting for the Lorenz-96 system, and show that incorporating the information of the variance of some un-observable variables can improve the skill and also increase the stability of the data assimilation procedure.Comment: 32 pages, 11 figure

Towards a General Theory of Extremes for Observables of Chaotic Dynamical Systems

In this paper we provide a connection between the geometrical properties of a chaotic dynamical system and the distribution of extreme values. We show that the extremes of so-called physical observables are distributed according to the classical generalised Pareto distribution and derive explicit expressions for the scaling and the shape parameter. In particular, we derive that the shape parameter does not depend on the chosen observables, but only on the partial dimensions of the invariant measure on the stable, unstable, and neutral manifolds. The shape parameter is negative and is close to zero when high-dimensional systems are considered. This result agrees with what was derived recently using the generalized extreme value approach. Combining the results obtained using such physical observables and the properties of the extremes of distance observables, it is possible to derive estimates of the partial dimensions of the attractor along the stable and the unstable directions of the flow. Moreover, by writing the shape parameter in terms of moments of the extremes of the considered observable and by using linear response theory, we relate the sensitivity to perturbations of the shape parameter to the sensitivity of the moments, of the partial dimensions, and of the Kaplan-Yorke dimension of the attractor. Preliminary numerical investigations provide encouraging results on the applicability of the theory presented here. The results presented here do not apply for all combinations of Axiom A systems and observables, but the breakdown seems to be related to very special geometrical configurations.Comment: 16 pages, 3 Figure

Shepherding Hordes of Markov Chains

This paper considers large families of Markov chains (MCs) that are defined over a set of parameters with finite discrete domains. Such families occur in software product lines, planning under partial observability, and sketching of probabilistic programs. Simple questions, like `does at least one family member satisfy a property?', are NP-hard. We tackle two problems: distinguish family members that satisfy a given quantitative property from those that do not, and determine a family member that satisfies the property optimally, i.e., with the highest probability or reward. We show that combining two well-known techniques, MDP model checking and abstraction refinement, mitigates the computational complexity. Experiments on a broad set of benchmarks show that in many situations, our approach is able to handle families of millions of MCs, providing superior scalability compared to existing solutions.Comment: Full version of TACAS'19 submissio

A Novel Stochastic Interacting Particle-Field Algorithm for 3D Parabolic-Parabolic Keller-Segel Chemotaxis System

We introduce an efficient stochastic interacting particle-field (SIPF) algorithm with no history dependence for computing aggregation patterns and near singular solutions of parabolic-parabolic Keller-Segel (KS) chemotaxis system in three space dimensions (3D). The KS solutions are approximated as empirical measures of particles coupled with a smoother field (concentration of chemo-attractant) variable computed by the spectral method. Instead of using heat kernels causing history dependence and high memory cost, we leverage the implicit Euler discretization to derive a one-step recursion in time for stochastic particle positions and the field variable based on the explicit Green's function of an elliptic operator of the form Laplacian minus a positive constant. In numerical experiments, we observe that the resulting SIPF algorithm is convergent and self-adaptive to the high gradient part of solutions. Despite the lack of analytical knowledge (e.g. a self-similar ansatz) of the blowup, the SIPF algorithm provides a low-cost approach to study the emergence of finite time blowup in 3D by only dozens of Fourier modes and through varying the amount of initial mass and tracking the evolution of the field variable. Notably, the algorithm can handle at ease multi-modal initial data and the subsequent complex evolution involving the merging of particle clusters and formation of a finite time singularity

Linear statistics and pushed Coulomb gas at the edge of beta random matrices: four paths to large deviations

The Airy$_\beta$ point process, $a_i \equiv N^{2/3} (\lambda_i-2)$, describes the eigenvalues $\lambda_i$ at the edge of the Gaussian $\beta$ ensembles of random matrices for large matrix size $N \to \infty$. We study the probability distribution function (PDF) of linear statistics ${\sf L}= \sum_i t \varphi(t^{-2/3} a_i)$ for large parameter $t$. We show the large deviation forms $\mathbb{E}_{{\rm Airy},\beta}[\exp(-{\sf L})] \sim \exp(- t^2 \Sigma[\varphi])$ and $P({\sf L}) \sim \exp(- t^2 G(L/t^2))$ for the cumulant generating function and the PDF. We obtain the exact rate function $\Sigma[\varphi]$ using four apparently different methods (i) the electrostatics of a Coulomb gas (ii) a random Schr\"odinger problem, i.e. the stochastic Airy operator (iii) a cumulant expansion (iv) a non-local non-linear differential Painlev\'e type equation. Each method was independently introduced to obtain the lower tail of the KPZ equation. Here we show their equivalence in a more general framework. Our results are obtained for a class of functions $\varphi$, the monotonous soft walls, containing the monomials $\varphi(x)=(u+x)_+^\gamma$ and the exponential $\varphi(x)=e^{u+x}$ and equivalently describe the response of a Coulomb gas pushed at its edge. The small $u$ behavior of the excess energy $\Sigma[\varphi]$ exhibits a change at $\gamma=3/2$ between a non-perturbative hard wall like regime for $\gamma<3/2$ (third order free-to-pushed transition) and a perturbative deformation of the edge for $\gamma>3/2$ (higher order transition). Applications are given, among them: (i) truncated linear statistics such as $\sum_{i=1}^{N_1} a_i$, leading to a formula for the PDF of the ground state energy of $N_1 \gg 1$ noninteracting fermions in a linear plus random potential (ii) $(\beta-2)/r^2$ interacting spinless fermions in a trap at the edge of a Fermi gas (iii) traces of large powers of random matrices.Comment: Main text : 8 pages. Supp mat : 49 page

Optimization of Spinning Reserve in Stand-alone Wind-Diesel Power Systems

Spinning reserve carried on synchronized units is the most effective resource available to the system operators for managing unforeseen power unbalances, such as demand fluctuations and the sudden loss of generation equipment. The amount of reserve and the speed that it can effectively be deployed determine the supply reliability that the generation system can achieve. Carrying more spinning reserve reduces the probability that the generation system become unable to preserve the momentary power balance and costly remedial actions, such as involuntary load shedding, turns unavoidable to prevent a system collapse. Nevertheless, providing spinning reserve on a continuous basis is expensive. Indeed, the provision of spinning reserve entails incurring in startup costs to commit generating units in excess of the forecasted load, which consequently have to be dispatched at less efficient operating points. The problem of keeping the power balance is still more difficult in stand-alone wind-diesel power systems, since these systems are additionally subjected to random power fluctuations originated in the uncertain and intermittent nature of the wind resource. Furthermore, autonomous power system cannot rely on power imported from interconnections for preserving the power balance. The inherent characteristics of these systems require scheduling more reserve on synchronized units for ensuring adequate security and reliability levels. The higher reserve requirements may substantially deteriorate the economy of these supply systems. The costs of keeping spinning reserve must be compared with the benefits that it provides in terms of lower expected costs of interruptions. In essence, the optimal reserve level can be set so that the marginal cost of carrying an additional MW equals the marginal reduction of the expected load curtailment costs. Despite the apparent simplicity of this optimality condition, determining the optimal amount of spinning reserve in a practical setting presents substantial modelling complexities and computational challenges. Given the random nature of the disturbances and contingencies that may face a generation system, assessing the benefits of carrying a certain amount of spinning reserve involves quantifying the occurrence probability, Wind Power 2 duration, extent and costs of load loss events. Such evaluation entails modelling the stochastic behaviour of system operation by considering the random failure of system components and the stochastic fluctuations of load and wind generation. The problem is probabilistic in its very nature and thus it may be appropriately treated by applying stochastic modelling techniques. Only with the advent of more powerful computing hardware, the problem of optimizing the spinning reserve has attracted the interest of researchers and its solution is currently deemed practicable. This work proposes a novel method for determining the optimal amount of spinning reserve that should be carried in autonomous hybrid wind-diesel generation systems. The optimal spinning reserve is determined by comparing the cost of its provision with the economic benefits it delivers in terms of supply reliability. The proposed approach is still general and can be applied in straightforward manner to establish the optimal reserve level in large interconnected systems. The presented methodology considers with accuracy the probabilistic features of the load and the wind generation, as well as the random outages of the conventional generating units. By applying high-resolution chronological simulation techniques, the stochastic features of the integrated operation of the diesel units and the wind turbine can be detailed replicated. The mathematical model appropriately considers all relevant characteristics and operational constraints of the generating units, e.g. non-linear heat rate curve, maximum and minimum output, startup and synchronization time, minimum down and uptime, ramping, etc. Massive stochastic simulation methods allow assessing the system reliability and valuing the economic costs of loss load events. Global search methods like particle swarm optimization (PSO) are proposed for finding the optimal scheduling policy and spinning reserve requirement that minimizes the sum of the expected operation costs and the expected costs of the energy not served.Fil: Olsina, Fernando Gabriel. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - San Juan; Argentina. Universidad Nacional de San Juan. Facultad de Ingeniería. Instituto de Energía Eléctrica; ArgentinaFil: Larisson, Carlos Hugo. No especifíca