970 research outputs found
Inferring functional connectivity from time-series of events in large scale network deployments
To respond rapidly and accurately to network and service outages, network operators must deal with a large number of events resulting from the interaction of various services operating on complex, heterogeneous and evolving networks. In this paper, we introduce the concept of functional connectivity as an alternative approach to monitoring those events. Commonly used in the study of brain dynamics, functional connectivity is defined in terms of the presence of statistical dependencies between nodes. Although a number of techniques exist to infer functional connectivity in brain networks, their straightforward application to commercial network deployments is severely challenged by: (a) non-stationarity of the functional connectivity, (b) sparsity of the time-series of events, and (c) absence of an explicit model describing how events propagate through the network or indeed whether they propagate. Thus, in this paper, we present a novel inference approach whereby two nodes are defined as forming a functional edge if they emit substantially more coincident or short-lagged events than would be expected if they were statistically independent. The output of the method is an undirected weighted graph, where the weight of an edge between two nodes denotes the strength of the statistical dependence between them. We develop a model of time-varying functional connectivity whose parameters are determined by maximising the model's predictive power from one time window to the next. We assess the accuracy, efficiency and scalability of our method on two real datasets of network events spanning multiple months and on synthetic data for which ground truth is available. We compare our method against both a general-purpose time-varying network inference method and network management specific causal inference technique and discuss its merits in terms of sensitivity, accuracy and, importantly, scalability
Recommended from our members
Functional connectivity inference from time-series of events and application to computer networks
Today’s commercial and Internet Service Provider’s networks are large, heterogeneous, fast-evolving and commonly emit millions of events per second. Monitoring, responding to, and predicting incidents is a key challenge for network management operators. This thesis argues that the concept of functional connectivity, widely used in neuroscience and defined in terms of the presence of statistical dependencies between nodes, can unlock informational value about event logs and assist operators in identifying (and even predicting) service outages. However, existing functional connectivity inference methods are not adapted to event data from computer networks. These methods may either require unavailable models of event propagation, be computationally too costly for large networks and/or long recordings, be not adapted to sparse and discrete activity, and/or assume a static network topology. We thus first describe in depth a major commercial network to highlight the challenges faced by network operators and the opportunities offered by thinking in terms of functional connectivities. Next, using a pair of independent Bernoulli processes as reference, we develop a new statistic aimed at measuring coupling strength by quantifying deviation from independence. However, because many statistics will be large by chance, identifying functional edges from the distribution of statistics over every pair of nodes is challenging. Hence, we then develop a method that infers, in specific contexts, the function that associates each statistic to the probability it accounts for the presence of a functional edge. Next, using the previously described statistic, we propose a novel framework to infer a time-varying functional topology from the time-series of emitted events. Applying this paradigm to two major commercial networks, we show that it can reveal a priori unknown groups of devices providing particular services. Finally, we argue that this thesis has implications beyond assisting network monitoring, such as enabling more robust inference of functional connectivity in neuroscience
Recommended from our members
Functional topology inference from network events
In this paper we present a novel approach for inferring functional connectivity within a large-scale network from time series of emitted node events. We do so under the following constraints: (a) non-stationarity of the underlying connectivity, (b) sparsity of the time-series of events, and (c) absence of an explicit model describing how events propagate through the network. We develop an inference method whose output is an undirected weighted network, where the weight of an edge between two nodes denotes the probability of these nodes being functionally connected. Two nodes are assumed to be functionally connected if they show significantly more coincident or short-lagged events than randomly picked pairs of nodes with similar levels of activity. We develop a model of time-varying connectivity whose parameters are determined by maximising the model’s predictive power from one time window to the next. We assess the accuracy, efficiency and scalability of our method on a real dataset of network events spanning multiple months
Network Inference via the Time-Varying Graphical Lasso
Many important problems can be modeled as a system of interconnected
entities, where each entity is recording time-dependent observations or
measurements. In order to spot trends, detect anomalies, and interpret the
temporal dynamics of such data, it is essential to understand the relationships
between the different entities and how these relationships evolve over time. In
this paper, we introduce the time-varying graphical lasso (TVGL), a method of
inferring time-varying networks from raw time series data. We cast the problem
in terms of estimating a sparse time-varying inverse covariance matrix, which
reveals a dynamic network of interdependencies between the entities. Since
dynamic network inference is a computationally expensive task, we derive a
scalable message-passing algorithm based on the Alternating Direction Method of
Multipliers (ADMM) to solve this problem in an efficient way. We also discuss
several extensions, including a streaming algorithm to update the model and
incorporate new observations in real time. Finally, we evaluate our TVGL
algorithm on both real and synthetic datasets, obtaining interpretable results
and outperforming state-of-the-art baselines in terms of both accuracy and
scalability
A smartwater metering deployment based on the fog computing paradigm
In this paper, we look into smart water metering infrastructures that enable continuous, on-demand and bidirectional data exchange between metering devices, water flow equipment, utilities and end-users. We focus on the design, development and deployment of such infrastructures as part of larger, smart city, infrastructures. Until now, such critical smart city infrastructures have been developed following a cloud-centric paradigm where all the data are collected and processed centrally using cloud services to create real business value. Cloud-centric approaches need to address several performance issues at all levels of the network, as massive metering datasets are transferred to distant machine clouds while respecting issues like security and data privacy. Our solution uses the fog computing paradigm to provide a system where the computational resources already available throughout the network infrastructure are utilized to facilitate greatly the analysis of fine-grained water consumption data collected by the smart meters, thus significantly reducing the overall load to network and cloud resources. Details of the system's design are presented along with a pilot deployment in a real-world environment. The performance of the system is evaluated in terms of network utilization and computational performance. Our findings indicate that the fog computing paradigm can be applied to a smart grid deployment to reduce effectively the data volume exchanged between the different layers of the architecture and provide better overall computational, security and privacy capabilities to the system
Recommended from our members
Exploiting functional connectivity inference for efficient root cause analysis
A crucial step in remedying faults within network infrastructure is to determine their root cause. However, the large-scale, complex and dynamic nature of modern architecture makes root cause analysis challenging.
Statistical approaches for causal inference are promising, however, their deployment has been historically limited due to their high time complexity. In this paper we propose a general framework for leveraging the concept of functional connectivity to reduce the computational overhead of causal inference algorithms. We demonstrate on synthetic data that our approach can achieve substantial speedups when combined with state-of-the-art causal discovery algorithms, with only a small cost in terms of loss of causal information in some cases
Big data analyses reveal patterns and drivers of the movements of southern elephant seals
The growing number of large databases of animal tracking provides an
opportunity for analyses of movement patterns at the scales of populations and
even species. We used analytical approaches, developed to cope with big data,
that require no a priori assumptions about the behaviour of the target agents,
to analyse a pooled tracking dataset of 272 elephant seals (Mirounga leonina)
in the Southern Ocean, that was comprised of >500,000 location estimates
collected over more than a decade. Our analyses showed that the displacements
of these seals were described by a truncated power law distribution across
several spatial and temporal scales, with a clear signature of directed
movement. This pattern was evident when analysing the aggregated tracks despite
a wide diversity of individual trajectories. We also identified marine
provinces that described the migratory and foraging habitats of these seals.
Our analysis provides evidence for the presence of intrinsic drivers of
movement, such as memory, that cannot be detected using common models of
movement behaviour. These results highlight the potential for big data
techniques to provide new insights into movement behaviour when applied to
large datasets of animal tracking.Comment: 18 pages, 5 figures, 6 supplementary figure
Recommended from our members
Accelerating causal inference based RCA using prior knowledge from functional connectivity inference
A crucial step in remedying faults within network infrastructures is to determine their root cause. However, the large-scale, complex and dynamic nature of modern networks makes causal inference-based root cause analysis challenging in terms of scalability and knowledge drift over time. In this paper, we propose a framework that utilises the neuroscientific concept of functional connectivity– a graph representation of statistical dependencies between events– as a scalable approach to acquire and maintain prior knowledge for causal inferencebased RCA approaches in dynamic networks. We demonstrate on both synthetic and real-world data that our proposed approach can provide significant speedups to existing causal inference approaches without significant loss of accuracy. We show that, in some cases, such prior knowledge can even improve the accuracy of causal inference. Finally, we discuss the impact of the choice of user-defined parameters on causal inference accuracy and conclude that the framework can safely be deployed in the real world
- …