998 research outputs found
MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams
Given a stream of graph edges from a dynamic graph, how can we assign anomaly
scores to edges in an online manner, for the purpose of detecting unusual
behavior, using constant time and memory? Existing approaches aim to detect
individually surprising edges. In this work, we propose MIDAS, which focuses on
detecting microcluster anomalies, or suddenly arriving groups of suspiciously
similar edges, such as lockstep behavior, including denial of service attacks
in network traffic data. MIDAS has the following properties: (a) it detects
microcluster anomalies while providing theoretical guarantees about its false
positive probability; (b) it is online, thus processing each edge in constant
time and constant memory, and also processes the data 162-644 times faster than
state-of-the-art approaches; (c) it provides 42%-48% higher accuracy (in terms
of AUC) than state-of-the-art approaches.Comment: 8 pages, Accepted at AAAI Conference on Artificial Intelligence
(AAAI), 2020 [oral paper]; minor fixes, updated experiment
The Ubiquity of Large Graphs and Surprising Challenges of Graph Processing: Extended Survey
Graph processing is becoming increasingly prevalent across many application
domains. In spite of this prevalence, there is little research about how graphs
are actually used in practice. We performed an extensive study that consisted
of an online survey of 89 users, a review of the mailing lists, source
repositories, and whitepapers of a large suite of graph software products, and
in-person interviews with 6 users and 2 developers of these products. Our
online survey aimed at understanding: (i) the types of graphs users have; (ii)
the graph computations users run; (iii) the types of graph software users use;
and (iv) the major challenges users face when processing their graphs. We
describe the participants' responses to our questions highlighting common
patterns and challenges. Based on our interviews and survey of the rest of our
sources, we were able to answer some new questions that were raised by
participants' responses to our online survey and understand the specific
applications that use graph data and software. Our study revealed surprising
facts about graph processing in practice. In particular, real-world graphs
represent a very diverse range of entities and are often very large,
scalability and visualization are undeniably the most pressing challenges faced
by participants, and data integration, recommendations, and fraud detection are
very popular applications supported by existing graph software. We hope these
findings can guide future research
A Survey of Graph-based Deep Learning for Anomaly Detection in Distributed Systems
Anomaly detection is a crucial task in complex distributed systems. A
thorough understanding of the requirements and challenges of anomaly detection
is pivotal to the security of such systems, especially for real-world
deployment. While there are many works and application domains that deal with
this problem, few have attempted to provide an in-depth look at such systems.
In this survey, we explore the potentials of graph-based algorithms to identify
anomalies in distributed systems. These systems can be heterogeneous or
homogeneous, which can result in distinct requirements. One of our objectives
is to provide an in-depth look at graph-based approaches to conceptually
analyze their capability to handle real-world challenges such as heterogeneity
and dynamic structure. This study gives an overview of the State-of-the-Art
(SotA) research articles in the field and compare and contrast their
characteristics. To facilitate a more comprehensive understanding, we present
three systems with varying abstractions as use cases. We examine the specific
challenges involved in anomaly detection within such systems. Subsequently, we
elucidate the efficacy of graphs in such systems and explicate their
advantages. We then delve into the SotA methods and highlight their strength
and weaknesses, pointing out the areas for possible improvements and future
works.Comment: The first two authors (A. Danesh Pazho and G. Alinezhad Noghre) have
equal contribution. The article is accepted by IEEE Transactions on Knowledge
and Data Engineerin
- …