Data Assimilation for Spatial Temporal Simulations Using Localized Particle Filtering

As sensor data becomes more and more available, there is an increasing interest in assimilating real time sensor data into spatial temporal simulations to achieve more accurate simulation or prediction results. Particle Filters (PFs), also known as Sequential Monte Carlo methods, hold great promise in this area as they use Bayesian inference and stochastic sampling techniques to recursively estimate the states of dynamic systems from some given observations. However, PFs face major challenges to work effectively for complex spatial temporal simulations due to the high dimensional state space of the simulation models, which typically cover large areas and have a large number of spatially dependent state variables. As the state space dimension increases, the number of particles must increase exponentially in order to converge to the true system state. The purpose of this dissertation work is to develop localized particle filtering to support PFs-based data assimilation for large-scale spatial temporal simulations. We develop a spatially dependent particle-filtering framework that breaks the system state and observation data into sub-regions and then carries out localized particle filtering based on these spatial regions. The developed framework exploits the spatial locality property of system state and observation data, and employs the divide-and-conquer principle to reduce state dimension and data complexity. Within this framework, we propose a two-level automated spatial partitioning method to provide optimized and balanced spatial partitions with less boundary sensors. We also consider different types of data to effectively support data assimilation for spatial temporal simulations. These data include both hard data, which are measurements from physical devices, and soft data, which are information from messages, reports, and social network. The developed framework and methods are applied to large-scale wildfire spread simulations and achieved improved results. Furthermore, we compare the proposed framework to existing particle filtering based data assimilation frameworks and evaluate the performance for each of them

Data Assimilation Based on Sequential Monte Carlo Methods for Dynamic Data Driven Simulation

Simulation models are widely used for studying and predicting dynamic behaviors of complex systems. Inaccurate simulation results are often inevitable due to imperfect model and inaccurate inputs. With the advances of sensor technology, it is possible to collect large amount of real time observation data from real systems during simulations. This gives rise to a new paradigm of Dynamic Data Driven Simulation (DDDS) where a simulation system dynamically assimilates real time observation data into a running model to improve simulation results. Data assimilation for DDDS is a challenging task because sophisticated simulation models often have: 1) nonlinear non-Gaussian behavior 2) non-analytical expressions of involved probability density functions 3) high dimensional state space 4) high computation cost. Due to these properties, most existing data assimilation methods fail to effectively support data assimilation for DDDS in one way or another. This work develops algorithms and software to perform data assimilation for dynamic data driven simulation through non-parametric statistic inference based on sequential Monte Carlo (SMC) methods (also called particle filters). A bootstrap particle filter based data assimilation framework is firstly developed, where the proposal distribution is constructed from simulation models and statistical cores of noises. The bootstrap particle filter-based framework is relatively easy to implement. However, it is ineffective when the uncertainty of simulation models is much larger than the observation model (i.e. peaked likelihood) or when rare events happen. To improve the effectiveness of data assimilation, a new data assimilation framework, named as the SenSim framework, is then proposed, which has a more advanced proposal distribution that uses knowledge from both simulation models and sensor readings. Both the bootstrap particle filter-based framework and the SenSim framework are applied and evaluated in two case studies: wildfire spread simulation, and lane-based traffic simulation. Experimental results demonstrate the effectiveness of the proposed data assimilation methods. A software package is also created to encapsulate the different components of SMC methods for supporting data assimilation of general simulation models

Sequential Monte Carlo Based Data Assimilation Framework and Toolkit for Dynamic System Simulations

Assimilating real-time sensor data into simulations is an effective approach for improving predictive abilities. However, integrating complex simulation models, e.g., discrete event simulation models and agent-based simulation models, is a challenging task. That is because classical data assimilation techniques, such as Kalman Filter, rely on the analytical forms of system transition distribution, which these models do not have. Sequential Monte Carlo methods are a class of most extensively used data assimilation algorithms which recursively estimate system states using Bayesian inference and sampling technique. They are non-parametric filters and thus can work effectively with complex simulation models. Despite of the advantages of Sequential Monte Carlo methods, simulation systems do not automatically fit in data assimilation framework. In most cases, it is a difficult and tedious task to carry out data assimilation for complex simulation models. In addition, Sequential Monte Carlo methods are statistical methods developed by mathematicians while simulation systems are developed by researchers in particular research fields other than math. There is a need to bridge the gap of theory and application and to make it easy to apply SMC methods to simulation applications. This dissertation presents a general framework integrating simulation models and data assimilation, and provides guidance of how to carry out data assimilation for dynamic system simulations. The developed framework formalizes the data assimilation process by defining specifications for both simulation models and data assimilation algorithms. It implements the standard Bootstrap Particle Filtering algorithm and a new \emph{Sensor Informed Particle Filter}, (SenSim) to support effective data assimilation. The developed framework is evaluated based on the application of wildfire spread simulation, and experiment results show the effectiveness of data assimilation. Besides the framework, we also developed an open source software toolkit named as Data Assimilation Framework Toolkit to make it easy for researchers to carry out data assimilation for their own simulation applications. A tutorial example is provided to demonstrate the data assimilation process using this data assimilation toolkit

Distributed Particle Filters for Data Assimilation in Simulation of Large Scale Spatial Temporal Systems

Assimilating real time sensor into a running simulation model can improve simulation results for simulating large-scale spatial temporal systems such as wildfire, road traffic and flood. Particle filters are important methods to support data assimilation. While particle filters can work effectively with sophisticated simulation models, they have high computation cost due to the large number of particles needed in order to converge to the true system state. This is especially true for large-scale spatial temporal simulation systems that have high dimensional state space and high computation cost by themselves. To address the performance issue of particle filter-based data assimilation, this dissertation developed distributed particle filters and applied them to large-scale spatial temporal systems. We first implemented a particle filter-based data assimilation framework and carried out data assimilation to estimate system state and model parameters based on an application of wildfire spread simulation. We then developed advanced particle routing methods in distributed particle filters to route particles among the Processing Units (PUs) after resampling in effective and efficient manners. In particular, for distributed particle filters with centralized resampling, we developed two routing policies named minimal transfer particle routing policy and maximal balance particle routing policy. For distributed PF with decentralized resampling, we developed a hybrid particle routing approach that combines the global routing with the local routing to take advantage of both. The developed routing policies are evaluated from the aspects of communication cost and data assimilation accuracy based on the application of data assimilation for large-scale wildfire spread simulations. Moreover, as cloud computing is gaining more and more popularity; we developed a parallel and distributed particle filter based on Hadoop & MapReduce to support large-scale data assimilation

Langevin and Hamiltonian based Sequential MCMC for Efficient Bayesian Filtering in High-dimensional Spaces

Nonlinear non-Gaussian state-space models arise in numerous applications in statistics and signal processing. In this context, one of the most successful and popular approximation techniques is the Sequential Monte Carlo (SMC) algorithm, also known as particle filtering. Nevertheless, this method tends to be inefficient when applied to high dimensional problems. In this paper, we focus on another class of sequential inference methods, namely the Sequential Markov Chain Monte Carlo (SMCMC) techniques, which represent a promising alternative to SMC methods. After providing a unifying framework for the class of SMCMC approaches, we propose novel efficient strategies based on the principle of Langevin diffusion and Hamiltonian dynamics in order to cope with the increasing number of high-dimensional applications. Simulation results show that the proposed algorithms achieve significantly better performance compared to existing algorithms

Statistical Filtering for Multimodal Mobility Modeling in Cyber Physical Systems

A Cyber-Physical System integrates computations and dynamics of physical processes. It is an engineering discipline focused on technology with a strong foundation in mathematical abstractions. It shares many of these abstractions with engineering and computer science, but still requires adaptation to suit the dynamics of the physical world. In such a dynamic system, mobility management is one of the key issues against developing a new service. For example, in the study of a new mobile network, it is necessary to simulate and evaluate a protocol before deployment in the system. Mobility models characterize mobile agent movement patterns. On the other hand, they describe the conditions of the mobile services. The focus of this thesis is on mobility modeling in cyber-physical systems. A macroscopic model that captures the mobility of individuals (people and vehicles) can facilitate an unlimited number of applications. One fundamental and obvious example is traffic profiling. Mobility in most systems is a dynamic process and small non-linearities can lead to substantial errors in the model. Extensive research activities on statistical inference and filtering methods for data modeling in cyber-physical systems exist. In this thesis, several methods are employed for multimodal data fusion, localization and traffic modeling. A novel energy-aware sparse signal processing method is presented to process massive sensory data. At baseline, this research examines the application of statistical filters for mobility modeling and assessing the difficulties faced in fusing massive multi-modal sensory data. A statistical framework is developed to apply proposed methods on available measurements in cyber-physical systems. The proposed methods have employed various statistical filtering schemes (i.e., compressive sensing, particle filtering and kernel-based optimization) and applied them to multimodal data sets, acquired from intelligent transportation systems, wireless local area networks, cellular networks and air quality monitoring systems. Experimental results show the capability of these proposed methods in processing multimodal sensory data. It provides a macroscopic mobility model of mobile agents in an energy efficient way using inconsistent measurements

Proceedings of the 4th Symposium on Management of Future Motorway and Urban Traffic Systems 2022

The 4th Symposium on Management of Future Motorway and Urban Traffic Systems (MFTS) was held in Dresden, Germany, from November 30th to December 2nd, 2022. Organized by the Chair of Traffic Process Automation (VPA) at the “Friedrich List” Faculty of Transport and Traffic Sciences of the TU Dresden, the proceedings of this conference are published as volume 9 in the Chair’s publication series “Verkehrstelematik” and contain a large part of the presented conference extended abstracts. The focus of the MFTS conference 2022 was cooperative management of multimodal transport and reflected the vision of the professorship to be an internationally recognized group in ITS research and education with the goal of optimizing the operation of multimodal transport systems. In 14 MFTS sessions, current topics in demand and traffic management, traffic control in conventional, connected and automated transport, connected and autonomous vehicles, traffic flow modeling and simulation, new and shared mobility systems, digitization, and user behavior and safety were discussed. In addition, special sessions were organized, for example on “Human aspects in traffic modeling and simulation” and “Lesson learned from Covid19 pandemic”, whose descriptions and analyses are also included in these proceedings.:1 Connected and Automated Vehicles 1.1 Traffic-based Control of Truck Platoons on Freeways 1.2 A Lateral Positioning Strategy for Connected and Automated Vehicles in Lane-free Traffic 1.3 Simulation Methods for Mixed Legacy-Autonomous Mainline Train Operations 1.4 Can Dedicated Lanes for Automated Vehicles on Urban Roads Improve Traffic Efficiency? 1.5 GLOSA System with Uncertain Green and Red Signal Phases 2 New Mobility Systems 2.1 A New Model for Electric Vehicle Mobility and Energy Consumption in Urban Traffic Networks 2.2 Shared Autonomous Vehicles Implementation for a Disrupted Public Transport Network 3 Traffic Flow and Simulation 3.1 Multi-vehicle Stochastic Fundamental Diagram Consistent with Transportations Systems Theory 3.2 A RoundD-like Roundabout Scenario in CARLA Simulator 3.3 Multimodal Performance Evaluation of Urban Traffic Control: A Microscopic Simulation Study 3.4 A MILP Framework to Solve the Sustainable System Optimum with Link MFD Functions 3.5 On How Traffic Signals Impact the Fundamental Diagrams of Urban Roads 4 Traffic Control in Conventional Traffic 4.1 Data-driven Methods for Identifying Travel Conditions Based on Traffic and Weather Characteristics 4.2 AI-based Multi-class Traffic Model Oriented to Freeway Traffic Control 4.3 Exploiting Deep Learning and Traffic Models for Freeway Traffic Estimation 4.4 Automatic Design of Optimal Actuated Traffic Signal Control with Transit Signal Priority 4.5 A Deep Reinforcement Learning Approach for Dynamic Traffic Light Control with Transit Signal Priority 4.6 Towards Efficient Incident Detection in Real-time Traffic Management 4.7 Dynamic Cycle Time in Traffic Signal of Cyclic Max-Pressure Control 5 Traffic Control with Autonomous Vehicles 5.1 Distributed Ordering and Optimization for Intersection Management with Connected and Automated Vehicles 5.2 Prioritization of an Automated Shuttle for V2X Public Transport at a Signalized Intersection – a Real-life Demonstration 6 User Behaviour and Safety 6.1 Local Traffic Safety Analyzer (LTSA) - Improved Road Safety and Optimized Signal Control for Future Urban Intersections 7 Demand and Traffic Management 7.1 A Stochastic Programming Method for OD Estimation Using LBSN Check-in Data 7.2 Delineation of Traffic Analysis Zone for Public Transportation OD Matrix Estimation Based on Socio-spatial Practices 8 Workshops 8.1 How to Integrate Human Aspects Into Engineering Science of Transport and Traffic? - a Workshop Report about Discussions on Social Contextualization of Mobility 8.2 Learning from Covid: How Can we Predict Mobility Behaviour in the Face of Disruptive Events? – How to Investigate the Mobility of the FutureDas 4. Symposium zum Management zukünftiger Autobahn- und Stadtverkehrssysteme (MFTS) fand vom 30. November bis 2. Dezember 2022 in Dresden statt und wurde vom Lehrstuhl für Verkehrsprozessautomatisierung (VPA) an der Fakultät Verkehrswissenschaften„Friedrich List“ der TU Dresden organisiert. Der Tagungsband erscheint als Band 9 in der Schriftenreihe „Verkehrstelematik“ des Lehrstuhls und enthält einen Großteil der vorgestellten Extended-Abstracts des Symposiums. Der Schwerpunkt des MFTS-Symposiums 2022 lag auf dem kooperativen Management multimodalen Verkehrs und spiegelte die Vision der Professur wider, eine international anerkannte Gruppe in der ITS-Forschung und -Ausbildung mit dem Ziel der Optimierung des Betriebs multimodaler Transportsysteme zu sein. In 14 MFTS-Sitzungen wurden aktuelle Themen aus den Bereichen Nachfrage- und Verkehrsmanagement, Verkehrssteuerung im konventionellen, vernetzten und automatisierten Verkehr, vernetzte und autonome Fahrzeuge, Verkehrsflussmodellierung und -simulation, neue und geteilte Mobilitätssysteme, Digitalisierung sowie Nutzerverhalten und Sicherheit diskutiert. Darüber hinaus wurden Sondersitzungen organisiert, beispielsweise zu „Menschlichen Aspekten bei der Verkehrsmodellierung und -simulation“ und „Lektionen aus der Covid-19-Pandemie“, deren Beschreibungen und Analysen ebenfalls in diesen Tagungsband einfließen.:1 Connected and Automated Vehicles 1.1 Traffic-based Control of Truck Platoons on Freeways 1.2 A Lateral Positioning Strategy for Connected and Automated Vehicles in Lane-free Traffic 1.3 Simulation Methods for Mixed Legacy-Autonomous Mainline Train Operations 1.4 Can Dedicated Lanes for Automated Vehicles on Urban Roads Improve Traffic Efficiency? 1.5 GLOSA System with Uncertain Green and Red Signal Phases 2 New Mobility Systems 2.1 A New Model for Electric Vehicle Mobility and Energy Consumption in Urban Traffic Networks 2.2 Shared Autonomous Vehicles Implementation for a Disrupted Public Transport Network 3 Traffic Flow and Simulation 3.1 Multi-vehicle Stochastic Fundamental Diagram Consistent with Transportations Systems Theory 3.2 A RoundD-like Roundabout Scenario in CARLA Simulator 3.3 Multimodal Performance Evaluation of Urban Traffic Control: A Microscopic Simulation Study 3.4 A MILP Framework to Solve the Sustainable System Optimum with Link MFD Functions 3.5 On How Traffic Signals Impact the Fundamental Diagrams of Urban Roads 4 Traffic Control in Conventional Traffic 4.1 Data-driven Methods for Identifying Travel Conditions Based on Traffic and Weather Characteristics 4.2 AI-based Multi-class Traffic Model Oriented to Freeway Traffic Control 4.3 Exploiting Deep Learning and Traffic Models for Freeway Traffic Estimation 4.4 Automatic Design of Optimal Actuated Traffic Signal Control with Transit Signal Priority 4.5 A Deep Reinforcement Learning Approach for Dynamic Traffic Light Control with Transit Signal Priority 4.6 Towards Efficient Incident Detection in Real-time Traffic Management 4.7 Dynamic Cycle Time in Traffic Signal of Cyclic Max-Pressure Control 5 Traffic Control with Autonomous Vehicles 5.1 Distributed Ordering and Optimization for Intersection Management with Connected and Automated Vehicles 5.2 Prioritization of an Automated Shuttle for V2X Public Transport at a Signalized Intersection – a Real-life Demonstration 6 User Behaviour and Safety 6.1 Local Traffic Safety Analyzer (LTSA) - Improved Road Safety and Optimized Signal Control for Future Urban Intersections 7 Demand and Traffic Management 7.1 A Stochastic Programming Method for OD Estimation Using LBSN Check-in Data 7.2 Delineation of Traffic Analysis Zone for Public Transportation OD Matrix Estimation Based on Socio-spatial Practices 8 Workshops 8.1 How to Integrate Human Aspects Into Engineering Science of Transport and Traffic? - a Workshop Report about Discussions on Social Contextualization of Mobility 8.2 Learning from Covid: How Can we Predict Mobility Behaviour in the Face of Disruptive Events? – How to Investigate the Mobility of the Futur

Scalable online estimation with performance guarantees: Application to traffic network monitoring

This work is motivated by the need for scalable online estimation with provable performance in cyber-physical systems, especially in traffic monitoring applications. While model-based traffic estimation has achieved great success via experimental deployments, there are substantial open questions on the theoretical understanding of the performance of these estimators. This gap is largely due to the nonlinearity of the underlying traffic flow and the nonobservability of the estimation problems. The main contribution of this dissertation is to explicitly address performance guarantees of filtering algorithms on traffic networks, with specific focus on systems that are unobservable, or even switch among observable and unobservable scenarios. We first consider one-dimensional road sections to establish the main proof techniques before considering more general road networks. To tackle the non-linearity issue in traffic models, the Lighthill-Whitham-Richards partial differential equation (LWR PDE) is transformed to a discrete-time switched linear form, i.e., the switching mode model (SMM). We provide a rigorous analysis on the performance of the Kalman filter (KF) on the SMM. Although the error dynamics of the KF is very likely to diverge under general unobservable systems, we show that in the context of traffic estimation, a uniform upper bound for the mean error exists when the system is unobservable. This is done by exploring the interactions between the physical properties of traffic flows, the stability conditions in the discretization scheme, and the information update in the filter. We also derive error bounds for the KF when the system switches among the observable and unobservable modes of the SMM. The above analysis is then extended to traffic networks with junctions. To support the analysis, we develop a switched linear model describing traffic dynamics on a freeway section with a junction inside. The model, namely the switching mode model with junctions (SMM-J), combines the discretized LWR PDE with a junction solver. Based on the SMM-J, the error bounds of the KF are extended to freeway networks. This dissertation also studies two essential problems related to the scalability issue in the estimation of general cyber-physical systems: (i) state space scalability, where the enormous state dimension causes computational burden on estimators, and (ii) data scalability, where massive data transmission incurs considerable energy, bandwith, or monetary costs. First, we design a distributed local Kalman consensus filter (DLKCF) for large-scale estimation, where the entire state is partitioned into local sections, and the computation task is distributed to local agents. In addition, a consensus term is designed to promote agreement on the estimates of neighboring agents. We also derive the error bounds of the DLKCF used for traffic estimation. Next, we study sensor scheduling schemes designed to select the most informative data to transmit to the estimator, thus reducing data transmission while preserving estimation accuracy. In this context, we propose a filtering algorithm that extracts the implicit information in the scheduling policy and update both the state estimate and the error covariance when data transmission is not triggered, which achieves better estimation accuracy compared to existing algorithms that only update the error covariance in the absence of data transmission