Data Assimilation Based on Sequential Monte Carlo Methods for Dynamic Data Driven Simulation

Simulation models are widely used for studying and predicting dynamic behaviors of complex systems. Inaccurate simulation results are often inevitable due to imperfect model and inaccurate inputs. With the advances of sensor technology, it is possible to collect large amount of real time observation data from real systems during simulations. This gives rise to a new paradigm of Dynamic Data Driven Simulation (DDDS) where a simulation system dynamically assimilates real time observation data into a running model to improve simulation results. Data assimilation for DDDS is a challenging task because sophisticated simulation models often have: 1) nonlinear non-Gaussian behavior 2) non-analytical expressions of involved probability density functions 3) high dimensional state space 4) high computation cost. Due to these properties, most existing data assimilation methods fail to effectively support data assimilation for DDDS in one way or another. This work develops algorithms and software to perform data assimilation for dynamic data driven simulation through non-parametric statistic inference based on sequential Monte Carlo (SMC) methods (also called particle filters). A bootstrap particle filter based data assimilation framework is firstly developed, where the proposal distribution is constructed from simulation models and statistical cores of noises. The bootstrap particle filter-based framework is relatively easy to implement. However, it is ineffective when the uncertainty of simulation models is much larger than the observation model (i.e. peaked likelihood) or when rare events happen. To improve the effectiveness of data assimilation, a new data assimilation framework, named as the SenSim framework, is then proposed, which has a more advanced proposal distribution that uses knowledge from both simulation models and sensor readings. Both the bootstrap particle filter-based framework and the SenSim framework are applied and evaluated in two case studies: wildfire spread simulation, and lane-based traffic simulation. Experimental results demonstrate the effectiveness of the proposed data assimilation methods. A software package is also created to encapsulate the different components of SMC methods for supporting data assimilation of general simulation models

Sequential Monte Carlo Based Data Assimilation Framework and Toolkit for Dynamic System Simulations

Assimilating real-time sensor data into simulations is an effective approach for improving predictive abilities. However, integrating complex simulation models, e.g., discrete event simulation models and agent-based simulation models, is a challenging task. That is because classical data assimilation techniques, such as Kalman Filter, rely on the analytical forms of system transition distribution, which these models do not have. Sequential Monte Carlo methods are a class of most extensively used data assimilation algorithms which recursively estimate system states using Bayesian inference and sampling technique. They are non-parametric filters and thus can work effectively with complex simulation models. Despite of the advantages of Sequential Monte Carlo methods, simulation systems do not automatically fit in data assimilation framework. In most cases, it is a difficult and tedious task to carry out data assimilation for complex simulation models. In addition, Sequential Monte Carlo methods are statistical methods developed by mathematicians while simulation systems are developed by researchers in particular research fields other than math. There is a need to bridge the gap of theory and application and to make it easy to apply SMC methods to simulation applications. This dissertation presents a general framework integrating simulation models and data assimilation, and provides guidance of how to carry out data assimilation for dynamic system simulations. The developed framework formalizes the data assimilation process by defining specifications for both simulation models and data assimilation algorithms. It implements the standard Bootstrap Particle Filtering algorithm and a new \emph{Sensor Informed Particle Filter}, (SenSim) to support effective data assimilation. The developed framework is evaluated based on the application of wildfire spread simulation, and experiment results show the effectiveness of data assimilation. Besides the framework, we also developed an open source software toolkit named as Data Assimilation Framework Toolkit to make it easy for researchers to carry out data assimilation for their own simulation applications. A tutorial example is provided to demonstrate the data assimilation process using this data assimilation toolkit

Distributed Particle Filters for Data Assimilation in Simulation of Large Scale Spatial Temporal Systems

Assimilating real time sensor into a running simulation model can improve simulation results for simulating large-scale spatial temporal systems such as wildfire, road traffic and flood. Particle filters are important methods to support data assimilation. While particle filters can work effectively with sophisticated simulation models, they have high computation cost due to the large number of particles needed in order to converge to the true system state. This is especially true for large-scale spatial temporal simulation systems that have high dimensional state space and high computation cost by themselves. To address the performance issue of particle filter-based data assimilation, this dissertation developed distributed particle filters and applied them to large-scale spatial temporal systems. We first implemented a particle filter-based data assimilation framework and carried out data assimilation to estimate system state and model parameters based on an application of wildfire spread simulation. We then developed advanced particle routing methods in distributed particle filters to route particles among the Processing Units (PUs) after resampling in effective and efficient manners. In particular, for distributed particle filters with centralized resampling, we developed two routing policies named minimal transfer particle routing policy and maximal balance particle routing policy. For distributed PF with decentralized resampling, we developed a hybrid particle routing approach that combines the global routing with the local routing to take advantage of both. The developed routing policies are evaluated from the aspects of communication cost and data assimilation accuracy based on the application of data assimilation for large-scale wildfire spread simulations. Moreover, as cloud computing is gaining more and more popularity; we developed a parallel and distributed particle filter based on Hadoop & MapReduce to support large-scale data assimilation

Data Assimilation for Spatial Temporal Simulations Using Localized Particle Filtering

As sensor data becomes more and more available, there is an increasing interest in assimilating real time sensor data into spatial temporal simulations to achieve more accurate simulation or prediction results. Particle Filters (PFs), also known as Sequential Monte Carlo methods, hold great promise in this area as they use Bayesian inference and stochastic sampling techniques to recursively estimate the states of dynamic systems from some given observations. However, PFs face major challenges to work effectively for complex spatial temporal simulations due to the high dimensional state space of the simulation models, which typically cover large areas and have a large number of spatially dependent state variables. As the state space dimension increases, the number of particles must increase exponentially in order to converge to the true system state. The purpose of this dissertation work is to develop localized particle filtering to support PFs-based data assimilation for large-scale spatial temporal simulations. We develop a spatially dependent particle-filtering framework that breaks the system state and observation data into sub-regions and then carries out localized particle filtering based on these spatial regions. The developed framework exploits the spatial locality property of system state and observation data, and employs the divide-and-conquer principle to reduce state dimension and data complexity. Within this framework, we propose a two-level automated spatial partitioning method to provide optimized and balanced spatial partitions with less boundary sensors. We also consider different types of data to effectively support data assimilation for spatial temporal simulations. These data include both hard data, which are measurements from physical devices, and soft data, which are information from messages, reports, and social network. The developed framework and methods are applied to large-scale wildfire spread simulations and achieved improved results. Furthermore, we compare the proposed framework to existing particle filtering based data assimilation frameworks and evaluate the performance for each of them

Building Occupancy Simulation and Data Assimilation Using a Graph Based Agent Oriented Model

Building occupancy simulation and estimation simulates the dynamics of occupants and estimates the real time spatial distribution of occupants in a building. It can benefit various applications like conserving energy, smart assist, building construction, crowd management, and emergency evacuation. Building occupancy simulation and estimation needs a simulation model and a data assimilation algorithm that assimilates real-time sensor data into the simulation model. Existing build occupancy simulation models include agent-based models and graph-based models. The agent-based models suffer high computation cost for simulating a large number occupants, and graph-based models overlook the heterogeneity and detailed behaviors of individuals. Recognizing the limitations of the existing models, in this dissertation, we combine the benefits of agent and graph based modeling and develop a new graph based agent oriented model which can efficiently simulate a large number of occupants in various building structures. To support real-time occupancy dynamics estimation, we developed a data assimilation framework based on Sequential Monte Carol Methods, and apply it to the graph-based agent oriented model to assimilate real time sensor data. Experimental results show the effectiveness of the developed model and the data assimilation framework. The major contributions of this dissertation work include 1) it provides an efficient model for building occupancy simulation which can accommodate thousands of occupants; 2) it provides an effective data assimilation framework for real-time estimation of building occupancy

Random Finite Set Based Data Assimilation for Dynamic Data Driven Simulation of Maritime Pirate Activity

Maritime piracy is posing a genuine threat to maritime transport. The main purpose of simulation is to predict the behaviors of many actual systems, and it has been successfully applied in many fields. But the application of simulation in the maritime domain is still scarce. The rapid development of network and measurement technologies brings about higher accuracy and better availability of online measurements. This makes the simulation paradigm named as dynamic data driven simulation increasingly popular. It can assimilate the online measurements into the running simulation models and ensure much more accurate prediction of the complex systems under study. In this paper, we study how to utilize the online measurements in the agent based simulation of the maritime pirate activity. A new random finite set based data assimilation algorithm is proposed to overcome the limitations of the conventional vectors based data assimilation algorithms. The random finite set based general data model, measurement model, and simulation model are introduced to support the proposed algorithm. The details of the proposed algorithm are presented in the context of agent based simulation of maritime pirate activity. Two groups of experiments are used to practically prove the effectiveness and superiority of the proposed algorithm

Data Assimilation for Agent-Based Simulation of Smart Environment

Agent-based simulation of smart environment finds its application in studying people’s movement to help the design of a variety of applications such as energy utilization, HAVC control and egress strategy in emergency situation. Traditionally, agent-based simulation is not dynamic data driven, they run offline and do not assimilate real sensor data about the environment. As more and more buildings are equipped with various sensors, it is possible to utilize real time sensor data to inform the simulation. To incorporate the real sensor data into the simulation, we introduce the method of data assimilation. The goal of data assimilation is to provide inference about system state based on the incomplete, ambiguous and uncertain sensor data using a computer model. A typical data assimilation framework consists of a computer model, a series of sensors and a melding scheme. The purpose of this dissertation is to develop a data assimilation framework for agent-based simulation of smart environment. With the developed data assimilation framework, we demonstrate an application of building occupancy estimation which focuses on position estimation using the framework. We build an agent based model to simulate the occupants’ movement s in the building and use this model in the data assimilation framework. The melding scheme we use to incorporate sensor data into the built model is particle filter algorithm. It is a set of statistical method aiming at compute the posterior distribution of the underlying system using a set of samples. It has the benefit that it does not have any assumption about the target distribution and does not require the target system to be written in analytic form .To overcome the high dimensional state space problem as the number of agents increases, we develop a new resampling method named as the component set resampling and evaluate its effectiveness in data assimilation. We also developed a graph-based model for simulating building occupancy. The developed model will be used for carrying out building occupancy estimation with extremely large number of agents in the future

Advanced Processing of Multispectral Satellite Data for Detecting and Learning Knowledge-based Features of Planetary Surface Anomalies

abstract: The marked increase in the inflow of remotely sensed data from satellites have trans- formed the Earth and Space Sciences to a data rich domain creating a rich repository for domain experts to analyze. These observations shed light on a diverse array of disciplines ranging from monitoring Earth system components to planetary explo- ration by highlighting the expected trend and patterns in the data. However, the complexity of these patterns from local to global scales, coupled with the volume of this ever-growing repository necessitates advanced techniques to sequentially process the datasets to determine the underlying trends. Such techniques essentially model the observations to learn characteristic parameters of data-generating processes and highlight anomalous planetary surface observations to help domain scientists for making informed decisions. The primary challenge in defining such models arises due to the spatio-temporal variability of these processes. This dissertation introduces models of multispectral satellite observations that sequentially learn the expected trend from the data by extracting salient features of planetary surface observations. The main objectives are to learn the temporal variability for modeling dynamic processes and to build representations of features of interest that is learned over the lifespan of an instrument. The estimated model parameters are then exploited in detecting anomalies due to changes in land surface reflectance as well as novelties in planetary surface landforms. A model switching approach is proposed that allows the selection of the best matched representation given the observations that is designed to account for rate of time-variability in land surface. The estimated parameters are exploited to design a change detector, analyze the separability of change events, and form an expert-guided representation of planetary landforms for prioritizing the retrieval of scientifically relevant observations with both onboard and post-downlink applications.Dissertation/ThesisDoctoral Dissertation Computer Engineering 201

A Bayesian spatio‐temporal model for short‐term forecasting of precipitation fields

With extreme weather events becoming more common, the risk posed by surface water flooding is ever increasing. In this work we propose a model, and associated Bayesian inference scheme, for generating short-term, probabilistic forecasts of localised precipitation on a spatial grid. Our generative hierarchical dynamic model is formulated in discrete space and time with a lattice-Markov spatio-temporal auto-regressive structure, inspired by continuous models of advection and diffusion. Observations from both weather radar and ground based rain gauges provide information from which we can learn the precipitation field through a latent process in addition to unknown model parameters. Working in the Bayesian paradigm provides a coherent framework for capturing uncertainty, both in the underlying model parameters and in our forecasts. Further, appealing to simulation based sampling using MCMC yields a straightforward solution to handling zeros, treated as censored observations, via data augmentation. Both the underlying state and the observations are of moderately large dimension ( [] and [] respectively) and this renders standard inference approaches computationally infeasible. Our solution is to embed the ensemble Kalman smoother within a Gibbs sampling scheme to facilitate approximate Bayesian inference in reasonable time. Both the methodology and the effectiveness of our posterior sampling scheme are demonstrated via simulation studies and by a case study of real data from the Urban Observatory project based in Newcastle upon Tyne, UK