Flow Monitoring Explained: From Packet Capture to Data Analysis With NetFlow and IPFIX

Flow monitoring has become a prevalent method for monitoring traffic in high-speed networks. By focusing on the analysis of flows, rather than individual packets, it is often said to be more scalable than traditional packet-based traffic analysis. Flow monitoring embraces the complete chain of packet observation, flow export using protocols such as NetFlow and IPFIX, data collection, and data analysis. In contrast to what is often assumed, all stages of flow monitoring are closely intertwined. Each of these stages therefore has to be thoroughly understood, before being able to perform sound flow measurements. Otherwise, flow data artifacts and data loss can be the consequence, potentially without being observed. This paper is the first of its kind to provide an integrated tutorial on all stages of a flow monitoring setup. As shown throughout this paper, flow monitoring has evolved from the early 1990s into a powerful tool, and additional functionality will certainly be added in the future. We show, for example, how the previously opposing approaches of deep packet inspection and flow monitoring have been united into novel monitoring approaches

Exploring the Potentials of Using Crowdsourced Waze Data in Traffic Management: Characteristics and Reliability

Real-time traffic information is essential to a variety of practical applications. To obtain traffic data, various traffic monitoring devices, such as loop detectors, infrastructure-mounted sensors, and cameras, have been installed on road networks. However, transportation agencies have sought alternative data sources to monitor traffic, due to the high installation and maintenance cost of conventional data collecting methods. Recently, crowdsourced traffic data has become available and is widely considered to have great potential in intelligent transportation systems. Waze is a crowdsourcing traffic application that enables users to share real-time traffic information. Waze data, including passively collected speed data and actively reported user reports, is valuable for traffic management but has not been explored or evaluated extensively. This dissertation evaluated and explored the potential of Waze data in traffic management from different perspectives. First, this dissertation evaluated and explored Waze traffic speed to understand the characteristics and reliability of Waze traffic speed data. Second, a calibration-free incident detection algorithm with traffic speed data on freeways was proposed, and the results were compared with other commonly used algorithms. Third, a spatial and temporal quality analysis of Waze accident reports to better understand their quality and accuracy was performed. Last, the dissertation proposed a network-based clustering algorithm to identify secondary crashes with Waze user reports, and a case study was performed to demonstrate the applicability of our method and the potential of crowdsourced Waze user reports

A live system for wavelet compression of high speed computer network measurements

Monitoring high-speed networks for a long period of time produces a high volume of data, making the storage of this information practically inefficient. To this end, there is a need to derive an efficient method of data analysis and reduction in order to archive and store the enormous amount of monitored traffic. Satisfying this need is useful not only for administrators but also for researchers who run their experiments on the monitored network. The researchers would like to know how their experiments affect the network's behavior in terms of utilization, delay, packet loss, data rate etc. In this paper a method of compressing computer network measurements while preserving the quality in interesting signal characteristics is presented. Eight different mother wavelets are compared against each other in order to examine which one offers the best results in terms of quality in the reconstructed signal. The proposed wavelet compression algorithm is compared against the lossless compression tool bzip2 in terms of compression ratio (C.R.). Finally, practical results are presented by compressing sampled traffic recorded from a live network

Performance Analysis of a Dynamic Bandwidth Allocation Algorithm in a Circuit-Switched Communications Network

Military communications networks typically employ a gateway multiplexer to aggregate all communications traffic onto a single link. These multiplexers typically use a static bandwidth allocation method via time-division multiplexing (TDM). Inefficiencies occur when a high-bandwidth circuit, e.g., a video teleconferencing circuit, is relatively inactive rendering a considerable portion of the aggregate bandwidth wasted while inactive. Dynamic bandwidth allocation (DBA) reclaims unused bandwidth from circuits with low utilization and reallocates it to circuits with higher utilization without adversely affecting queuing delay. The proposed DBA algorithm developed here measures instantaneous utilization by counting frames arriving during the transmission time of a single frame on the aggregate link. The maximum calculated utilization observed over a monitoring period is then used to calculate the bandwidth available for reallocation. A key advantage of the proposed approach is that it can be applied now and to existing systems supporting heterogeneous permanent virtual circuits. With the inclusion of DBA, military communications networks can bring information to the warfighter more efficiently and in a shorter time even for small bandwidths allocated to deployed sites. The algorithm is general enough to be applied to multiple TDM platforms and robust enough to function at any line speed, making it a viable option for high-speed multiplexers. The proposed DBA algorithm provides a powerful performance boost by optimizing available resources of the communications network. Utilization results indicate the proposed DBA algorithm significantly out-performs the static allocation model in all cases. The best configuration uses a 65536 bps allocation granularity and a 10 second monitoring period. Utilization gains observed with this configuration were almost 17% over the static allocation method. Queuing delays increased by 50% but remained acceptable, even for realtime traffic

Assessing the Impact of Game Day Schedule and Opponents on Travel Patterns and Route Choice using Big Data Analytics

The transportation system is crucial for transferring people and goods from point A to point B. However, its reliability can be decreased by unanticipated congestion resulting from planned special events. For example, sporting events collect large crowds of people at specific venues on game days and disrupt normal traffic patterns. The goal of this study was to understand issues related to road traffic management during major sporting events by using widely available INRIX data to compare travel patterns and behaviors on game days against those on normal days. A comprehensive analysis was conducted on the impact of all Nebraska Cornhuskers football games over five years on traffic congestion on five major routes in Nebraska. We attempted to identify hotspots, the unusually high-risk zones in a spatiotemporal space containing traffic congestion that occur on almost all game days. For hotspot detection, we utilized a method called Multi-EigenSpot, which is able to detect multiple hotspots in a spatiotemporal space. With this algorithm, we were able to detect traffic hotspot clusters on the five chosen routes in Nebraska. After detecting the hotspots, we identified the factors affecting the sizes of hotspots and other parameters. The start time of the game and the Cornhuskers’ opponent for a given game are two important factors affecting the number of people coming to Lincoln, Nebraska, on game days. Finally, the Dynamic Bayesian Networks (DBN) approach was applied to forecast the start times and locations of hotspot clusters in 2018 with a weighted mean absolute percentage error (WMAPE) of 13.8%

Wireless magnetic sensor network for road traffic monitoring and vehicle classification

Efficiency of transportation of people and goods is playing a vital role in economic growth. A key component for enabling effective planning of transportation networks is the deployment and operation of autonomous monitoring and traffic analysis tools. For that reason, such systems have been developed to register and classify road traffic usage. In this paper, we propose a novel system for road traffic monitoring and classification based on highly energy efficient wireless magnetic sensor networks. We develop novel algorithms for vehicle speed and length estimation and vehicle classification that use multiple magnetic sensors. We also demonstrate that, using such a low-cost system with simplified installation and maintenance compared to current solutions, it is possible to achieve highly accurate estimation and a high rate of positive vehicle classification