2 research outputs found
Learning Latent Events from Network Message Logs
We consider the problem of separating error messages generated in large
distributed data center networks into error events. In such networks, each
error event leads to a stream of messages generated by hardware and software
components affected by the event. These messages are stored in a giant message
log. We consider the unsupervised learning problem of identifying the
signatures of events that generated these messages; here, the signature of an
error event refers to the mixture of messages generated by the event. One of
the main contributions of the paper is a novel mapping of our problem which
transforms it into a problem of topic discovery in documents. Events in our
problem correspond to topics and messages in our problem correspond to words in
the topic discovery problem. However, there is no direct analog of documents.
Therefore, we use a non-parametric change-point detection algorithm, which has
linear computational complexity in the number of messages, to divide the
message log into smaller subsets called episodes, which serve as the
equivalents of documents. After this mapping has been done, we use a well-known
algorithm for topic discovery, called LDA, to solve our problem. We
theoretically analyze the change-point detection algorithm, and show that it is
consistent and has low sample complexity. We also demonstrate the scalability
of our algorithm on a real data set consisting of million messages
collected over a period of days, from a distributed data center network
which supports the operations of a large wireless service provider.Comment: To Appear in IEEE Transactions on Networking, Appeared in Workshop on
MiLeTS, SIGKDD 201
Industrial and Government Track Short Paper ABSTRACT Event Summarization for System Management β
In system management applications, an overwhelming amount of data are generated and collected in the form of temporal events. While mining temporal event data to discover interesting and frequent patterns has obtained rapidly increasing research efforts, users of the applications are overwhelmed by the mining results. The extracted patterns are generally of large volume and hard to interpret, they may be of no emphasis, intricate and meaningless to non-experts, even to domain experts. While traditional research efforts focus on finding interesting patterns, in this paper, we take a novel approach called event summarization towards the understanding of the seemingly chaotic temporal data. Event summarization aims at providing a concise interpretation of the seemingly chaotic data, so that domain experts may take actions upon the summarized models. Event summarization decomposes the temporal information into many independent subsets and finds well fitted models to describe each subset