2,089 research outputs found
An Improved Composite Hypothesis Test for Markov Models with Applications in Network Anomaly Detection
Recent work has proposed the use of a composite hypothesis Hoeffding test for
statistical anomaly detection. Setting an appropriate threshold for the test
given a desired false alarm probability involves approximating the false alarm
probability. To that end, a large deviations asymptotic is typically used
which, however, often results in an inaccurate setting of the threshold,
especially for relatively small sample sizes. This, in turn, results in an
anomaly detection test that does not control well for false alarms. In this
paper, we develop a tighter approximation using the Central Limit Theorem (CLT)
under Markovian assumptions. We apply our result to a network anomaly detection
application and demonstrate its advantages over earlier work.Comment: 6 pages, 6 figures; final version for CDC 201
Botnet Detection using Social Graph Analysis
Signature-based botnet detection methods identify botnets by recognizing
Command and Control (C\&C) traffic and can be ineffective for botnets that use
new and sophisticate mechanisms for such communications. To address these
limitations, we propose a novel botnet detection method that analyzes the
social relationships among nodes. The method consists of two stages: (i)
anomaly detection in an "interaction" graph among nodes using large deviations
results on the degree distribution, and (ii) community detection in a social
"correlation" graph whose edges connect nodes with highly correlated
communications. The latter stage uses a refined modularity measure and
formulates the problem as a non-convex optimization problem for which
appropriate relaxation strategies are developed. We apply our method to
real-world botnet traffic and compare its performance with other community
detection methods. The results show that our approach works effectively and the
refined modularity measure improves the detection accuracy.Comment: 7 pages. Allerton Conferenc
Robust Anomaly Detection in Dynamic Networks
We propose two robust methods for anomaly detection in dynamic networks in
which the properties of normal traffic are time-varying. We formulate the
robust anomaly detection problem as a binary composite hypothesis testing
problem and propose two methods: a model-free and a model-based one, leveraging
techniques from the theory of large deviations. Both methods require a family
of Probability Laws (PLs) that represent normal properties of traffic. We
devise a two-step procedure to estimate this family of PLs. We compare the
performance of our robust methods and their vanilla counterparts, which assume
that normal traffic is stationary, on a network with a diurnal normal pattern
and a common anomaly related to data exfiltration. Simulation results show that
our robust methods perform better than their vanilla counterparts in dynamic
networks.Comment: 6 pages. MED conferenc
Data-driven Estimation of Origin-Destination Demand and User Cost Functions for the Optimization of Transportation Networks
In earlier work (Zhang et al., 2016) we used actual traffic data from the
Eastern Massachusetts transportation network in the form of spatial average
speeds and road segment flow capacities in order to estimate Origin-Destination
(OD) flow demand matrices for the network. Based on a Traffic Assignment
Problem (TAP) formulation (termed "forward problem"), in this paper we use a
scheme similar to our earlier work to estimate initial OD demand matrices and
then propose a new inverse problem formulation in order to estimate user cost
functions. This new formulation allows us to efficiently overcome numerical
difficulties that limited our prior work to relatively small subnetworks and,
assuming the travel latency cost functions are available, to adjust the values
of the OD demands accordingly so that the flow observations are as close as
possible to the solutions of the forward problem. We also derive sensitivity
analysis results for the total user latency cost with respect to important
parameters such as road capacities and minimum travel times. Finally, using the
same actual traffic data from the Eastern Massachusetts transportation network,
we quantify the Price of Anarchy (POA) for a much larger network than that in
Zhang et al. (2016).Comment: Preprint submitted to The 20th World Congress of the International
Federation of Automatic Control, July 9-14, 2017, Toulouse, Franc
The price of anarchy in transportation networks by estimating user cost functions from actual traffic data
We have considered a large-scale road network in Eastern Massachusetts. Using real traffic data in the form of spatial average speeds and the flow capacity for each road segment of the network, we converted the speed data to flow data and estimated the origin-destination flow demand matrices for the network. Assuming that the observed traffic data correspond to user (Wardrop) equilibria for different times-of-the-day and days-of-the-week, we formulated appropriate inverse problems to recover the per-road cost (congestion) functions determining user route selection for each month and time-of-day period. In addition, we analyzed the sensitivity of the total user latency cost to important parameters such as road capacities and minimum travel times. Finally, we formulated a system-optimum problem in order to find socially optimal flows for the network. We investigated the network performance, in terms of the total latency, under a user-optimal policy versus a system-optimal policy. The ratio of these two quantities is defined as the Price of Anarchy (POA) and quantifies the efficiency loss of selfish actions compared to socially optimal ones. Our findings contribute to efforts for a smarter and more efficient city
Distributed MPC for coordinated energy efficiency utilization in microgrid systems
To improve the renewable energy utilization of distributed microgrid systems, this paper presents an optimal distributed model predictive control strategy to coordinate energy management among microgrid systems. In particular, through information exchange among systems, each microgrid in the network, which includes renewable generation, storage systems, and some controllable loads, can maintain its own systemwide supply and demand balance. With our mechanism, the closed-loop stability of the distributed microgrid systems can be guaranteed. In addition, we provide evaluation criteria of renewable energy utilization to validate our proposed method. Simulations show that the supply demand balance in each microgrid is achieved while, at the same time, the system operation cost is reduced, which demonstrates the effectiveness and efficiency of our proposed policy.Accepted manuscrip
- …