Detecting Flow Anomalies in Distributed Systems

Deep within the networks of distributed systems, one often finds anomalies that affect their efficiency and performance. These anomalies are difficult to detect because the distributed systems may not have sufficient sensors to monitor the flow of traffic within the interconnected nodes of the networks. Without early detection and making corrections, these anomalies may aggravate over time and could possibly cause disastrous outcomes in the system in the unforeseeable future. Using only coarse-grained information from the two end points of network flows, we propose a network transmission model and a localization algorithm, to detect the location of anomalies and rank them using a proposed metric within distributed systems. We evaluate our approach on passengers' records of an urbanized city's public transportation system and correlate our findings with passengers' postings on social media microblogs. Our experiments show that the metric derived using our localization algorithm gives a better ranking of anomalies as compared to standard deviation measures from statistical models. Our case studies also demonstrate that transportation events reported in social media microblogs matches the locations of our detect anomalies, suggesting that our algorithm performs well in locating the anomalies within distributed systems

Inferring Unusual Crowd Events From Mobile Phone Call Detail Records

The pervasiveness and availability of mobile phone data offer the opportunity of discovering usable knowledge about crowd behaviors in urban environments. Cities can leverage such knowledge in order to provide better services (e.g., public transport planning, optimized resource allocation) and safer cities. Call Detail Record (CDR) data represents a practical data source to detect and monitor unusual events considering the high level of mobile phone penetration, compared with GPS equipped and open devices. In this paper, we provide a methodology that is able to detect unusual events from CDR data that typically has low accuracy in terms of space and time resolution. Moreover, we introduce a concept of unusual event that involves a large amount of people who expose an unusual mobility behavior. Our careful consideration of the issues that come from coarse-grained CDR data ultimately leads to a completely general framework that can detect unusual crowd events from CDR data effectively and efficiently. Through extensive experiments on real-world CDR data for a large city in Africa, we demonstrate that our method can detect unusual events with 16% higher recall and over 10 times higher precision, compared to state-of-the-art methods. We implement a visual analytics prototype system to help end users analyze detected unusual crowd events to best suit different application scenarios. To the best of our knowledge, this is the first work on the detection of unusual events from CDR data with considerations of its temporal and spatial sparseness and distinction between user unusual activities and daily routines.Comment: 18 pages, 6 figure

Can multimodal sensing detect and localize transient events?

National Research Foundation (NRF) Singapore under International Research Centres in Singapore Funding Initiativ

Detection of Anomalous Traffic Patterns and Insight Analysis from Bus Trajectory Data

© 2019, Springer Nature Switzerland AG. Detection of anomalous patterns from traffic data is closely related to analysis of traffic accidents, fault detection, flow management, and new infrastructure planning. Existing methods on traffic anomaly detection are modelled on taxi trajectory data and have shortcoming that the data may lose much information about actual road traffic situation, as taxi drivers can select optimal route for themselves to avoid traffic anomalies. We employ bus trajectory data as it reflects real traffic conditions on the road to detect city-wide anomalous traffic patterns and to provide broader range of insights into these anomalies. Taking these considerations, we first propose a feature visualization method by mapping extracted 3-dimensional hidden features to red-green-blue (RGB) color space with a deep sparse autoencoder (DSAE). A color trajectory (CT) is produced by encoding a trajectory with RGB colors. Then, a novel algorithm is devised to detect spatio-temporal outliers with spatial and temporal properties extracted from the CT. We also integrate the CT with the geographic information system (GIS) map to obtain insights for understanding the traffic anomaly locations, and more importantly the road influence affected by the corresponding anomalies. Our proposed method was tested on three real-world bus trajectory data sets to demonstrate the excellent performance of high detection rates and low false alarm rates

Time delay estimation of traffic congestion propagation due to accidents based on statistical causality

The accurate estimation of time delays is crucial in traffic congestion analysis, as this information can be used to address fundamental questions regarding the origin and propagation of traffic congestion. However, the exact measurement of time delays during congestion remains a challenge owing to the complex propagation process between roads and high uncertainty regarding future behavior. To overcome this challenge, we propose a novel time delay estimation method for the propagation of traffic congestion due to accidents using lag-specific transfer entropy (TE). The proposed method adopts Markov bootstrap techniques to quantify uncertainty in the time delay estimator. To the best of our knowledge, our proposed method is the first to estimate time delays based on causal relationships between adjacent roads. We validated the method's efficacy using simulated data, as well as real user trajectory data obtained from a major GPS navigation system in South Korea.Comment: http://www.aimspress.com/article/doi/10.3934/era.202303

Map Matching Based on Conditional Random Fields and Route Preference Mining for Uncertain Trajectories

In order to improve offline map matching accuracy of uncertain GPS trajectories, a map matching algorithm based on conditional random fields (CRF) and route preference mining is proposed. In this algorithm, road offset distance and the temporal-spatial relationship between the sampling points are used as features of GPS trajectory in a CRF model, which integrates the temporal-spatial context information flexibly. The driver route preference is also used to bolster the temporal-spatial context when a low GPS sampling rate impairs the resolving power of temporal-spatial context in CRF, allowing the map matching accuracy of uncertain GPS trajectories to get improved significantly. The experimental results show that our proposed algorithm is more accurate than existing methods, especially in the case of a low-sampling-rate

TRAVEL TIME ESTIMATION USING SPATIO-TEMPORAL INDEX BASED ON CASSANDRA

Travel time estimation plays an important role in traffic monitoring and route planning. Taxicabs equipped with Global Positioning System (GPS) devices have been frequently used to monitor the traffic state, and GPS trajectories of taxicabs also used to estimate path travel time in an urban area. However, in most cases, it is difficult to find a trajectory that fits perfectly with the query path, as some road segments may be traveled by no taxicab in present time slot. This makes it hard to estimate the travel time of the query path. This paper proposes a framework to estimate the travel time of a path by using the GPS trajectories of taxicabs as well as map data sources. In this framework, the travel time is represented as a series of residence time in cells (one cell is the gird segmentation unit), thus the key issues of the estimation are: finding the local traffic patterns of frequently shared paths from historical data and computing the stay time in cells. There are three major processes in this framework: trajectories preprocessing, establishing the temporal-spatial index and cell-based travel time estimation. Based on the temporal-spatial index, an algorithm is developed that uses similar route patterns, the cell-based travel time over a period of history and road network information to estimate the travel time of a path. This paper uses GPS trajectories of 10,357 taxicabs over a period of one week to evaluate the framework. The results demonstrate that this paper’s method is effective and feasible in city-wide scenarios