Big Data for Traffic Estimation and Prediction: A Survey of Data and Tools

Big data has been used widely in many areas including the transportation industry. Using various data sources, traffic states can be well estimated and further predicted for improving the overall operation efficiency. Combined with this trend, this study presents an up-to-date survey of open data and big data tools used for traffic estimation and prediction. Different data types are categorized and the off-the-shelf tools are introduced. To further promote the use of big data for traffic estimation and prediction tasks, challenges and future directions are given for future studies

Final technical report : leveraging mobile network big data for developmental policy

The research addresses how big data can provide evidence to better inform public policy and allow for greater use of evidence in the policy making process. In addition to more detailed research in the area of transportation and urban planning (commuting patterns), this research articulates and answers questions in other domains such as health (modeling the spread of diseases) and official statistics (mapping poverty for instance). Guidelines were translated into legal language so that mobile operators can responsibly share data. Traditional survey methods that provide enough detail to accurately assess conditions are costly and can rarely reach a representative portion of the population, especially in poorer areas

Machine Learning at the Edge: A Data-Driven Architecture with Applications to 5G Cellular Networks

The fifth generation of cellular networks (5G) will rely on edge cloud deployments to satisfy the ultra-low latency demand of future applications. In this paper, we argue that such deployments can also be used to enable advanced data-driven and Machine Learning (ML) applications in mobile networks. We propose an edge-controller-based architecture for cellular networks and evaluate its performance with real data from hundreds of base stations of a major U.S. operator. In this regard, we will provide insights on how to dynamically cluster and associate base stations and controllers, according to the global mobility patterns of the users. Then, we will describe how the controllers can be used to run ML algorithms to predict the number of users in each base station, and a use case in which these predictions are exploited by a higher-layer application to route vehicular traffic according to network Key Performance Indicators (KPIs). We show that the prediction accuracy improves when based on machine learning algorithms that rely on the controllers' view and, consequently, on the spatial correlation introduced by the user mobility, with respect to when the prediction is based only on the local data of each single base station.Comment: 15 pages, 10 figures, 5 tables. IEEE Transactions on Mobile Computin

National Performance Management Research Dataset (NPMRDS) - Speed Validation for Traffic Performance Measures (FHWA-OK-17-02)

This report presents research detailing the use of the first version of the National Performance Management Research Data Set (NPMRDS v.1) comprised of highway vehicle travel times used for computing performance measurements in the state of Oklahoma. Data extraction, preprocessing, and statistical analysis were performed on the dataset and acomprehensive study of dataset characteristics, influencing variables, outliersand anomalies was carried out. In addition, a study on filtering and removing speed data outliers across multiple road segments is developed, and a comparative analysis of raw baseline speed data and cleansed data is performed. A method for improved congestion detection is investigated and developed. Identification and a computational comparison analysis of travel time reliability performance metrics for both raw and cleansed datasets is shown. An outlier removal framework is formulated, and a cleansed and complete version of NPMRDS v.1 is generated. Finally, a validation analysis on the cleansed dataset is presented. In the end, research affirmsthat understanding domain specific characteristics is vital for filtering data outliers and anomalies of this dataset,which in turn is key for calculating accurate performance measurements. Thus, careful consideration for outlierremoval must be taken into account when computing travel time reliability metrics using the NPMRDS.October 2015-October 2017N

Intelligent Transportation Related Complex Systems and Sensors

Building around innovative services related to different modes of transport and traffic management, intelligent transport systems (ITS) are being widely adopted worldwide to improve the efficiency and safety of the transportation system. They enable users to be better informed and make safer, more coordinated, and smarter decisions on the use of transport networks. Current ITSs are complex systems, made up of several components/sub-systems characterized by time-dependent interactions among themselves. Some examples of these transportation-related complex systems include: road traffic sensors, autonomous/automated cars, smart cities, smart sensors, virtual sensors, traffic control systems, smart roads, logistics systems, smart mobility systems, and many others that are emerging from niche areas. The efficient operation of these complex systems requires: i) efficient solutions to the issues of sensors/actuators used to capture and control the physical parameters of these systems, as well as the quality of data collected from these systems; ii) tackling complexities using simulations and analytical modelling techniques; and iii) applying optimization techniques to improve the performance of these systems. It includes twenty-four papers, which cover scientific concepts, frameworks, architectures and various other ideas on analytics, trends and applications of transportation-related data

Discovering critical traffic anomalies from GPS trajectories for urban traffic dynamics understanding

Traffic anomaly (e.g., traffic jams) detection is essential for the development of intelligent transportation systems in smart cities. In particular, detecting critical traffic anomalies (e.g., rare traffic anomalies, sudden accidents) are far more meaningful than detecting general traffic anomalies and more helpful to understand urban traffic dynamics. For example, emerging traffic jams are more significant than regular traffic jams caused by common road bottlenecks like traffic lights or toll road entrances;  and discovering the original location of traffic chaos in an area is more important than finding roads that are just congested. However, using existing traffic indicators that represent traffic conditions, such as traffic flows and speeds, for critical traffic anomaly detection may be not accurate enough. That is, they usually miss some traffic anomalies while wrongly identifying a normal traffic status as an anomaly. Moreover, most existing detection methods only detect general traffic anomalies but not critical traffic anomalies. In this thesis, we provide two new indicators: frequency of jams (captured by stop-point clusters) and Visible Outlier Indexes (VOIs) (based on the Kolmogorov-Smirnov test of speed) to capture critical traffic anomalies more accurately. The advantage of our proposed indicators is that they help separate critical traffic anomalies from general traffic anomalies. The former can discover rare anomalies with low frequency, and the latter can find unexpected anomalies (i.e., when the difference between the predicted VOI and the real VOI is great). Based on these two indicators, we provide three novel methods for comprehensive traffic anomaly analysis, including traffic anomaly identification, prediction, and root cause discovery. First, we provide a novel analysis of spatial-temporal jam frequencies (ASTJF) method for identifying rare traffic anomalies. In the ASTJF method, spatially close stop-points in a time bin are grouped into stop-point clusters (SPCs) using Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm; an SPC is an instance of a spatiotemporal jam. Then, we develop a new adapted Hausdorff distance to measure the similarity of two SPCs and put SPCs which are relevant to the same spatiotemporal jam into a group. Finally, we calculate the number of SPCs in a group as the frequency of the corresponding traffic jams; traffic anomalies are classified as regular jams with high frequency and emerging jams with low frequency. The ASTJF method can correctly identify critical traffic anomalies (i.e., emerging jams). Second, we propose a novel prediction approach -  Visible Outlier Indices and Meshed Spatiotemporal Neighborhoods (VOIMSN) method. In this method, the trajectory data from the given region's geographic spatial neighbors and its time-series neighbors are both converted to the abnormal scores measured by VOIs and quantified by the matrix grid as the input of the prediction model to improve the accuracy. This method provides a comprehensive analysis using all relevant data for building a reliable prediction model. In particular, the proposed meshed spatiotemporal neighborhoods with arbitrary shape, which comprises all potential anomalies instead of just past anomalies, is theoretically more accurate than a fixed-size neighborhood for anomaly prediction. Third, we provide an innovative and integrated root cause analysis method using VOI as the probabilistic indicator of traffic anomalies. This method proposes automatically learns spatiotemporal causal relationships from historical data to build an uneven diffusion model for detecting the root cause of anomalies (i.e., the origin of traffic chaos). It is demonstrated to be better than the heat diffusion model. Experiments conducted on a real-world massive trajectory dataset demonstrate the accuracy and effectiveness of the proposed methods for discovering critical traffic anomalies for a better understanding of urban traffic dynamics

Modeling Spatio-Temporal Evolution of Urban Crowd Flows

Metropolitan cities are facing many socio-economic problems (e.g., frequent traffic congestion, unexpected emergency events, and even human-made disasters) related to urban crowd flows, which can be described in terms of the gathering process of a flock of moving objects (e.g., vehicles, pedestrians) towards specific destinations during a given time period via different travel routes. Understanding the spatio-temporal characteristics of urban crowd flows is therefore of critical importance to traffic management and public safety, yet it is very challenging as it is affected by many complex factors, including spatial dependencies, temporal dependencies, and environmental conditions. In this research, we propose a novel matrix-computation-based method for modeling the morphological evolutionary patterns of urban crowd flows. The proposed methodology consists of four connected steps: (1) defining urban crowd levels, (2) deriving urban crowd regions, (3) quantifying their morphological changes, and (4) delineating the morphological evolution patterns. The proposed methodology integrates urban crowd visualization, identification, and correlation into a unified and efficient analytical framework. We validated the proposed methodology under both synthetic and real-world data scenarios using taxi mobility data in Wuhan, China as an example. Results confirm that the proposed methodology can enable city planners, municipal managers, and other stakeholders to identify and understand the gathering process of urban crowd flows in an informative and intuitive manner. Limitations and further directions with regard to data representativeness, data sparseness, pattern sensitivity, and spatial constraint are also discussed. Document type: Articl