Interpretable Learning in Multivariate Big Data Analysis for Network Monitoring

There is an increasing interest in the development of new data-driven models useful to assess the performance of communication networks. For many applications, like network monitoring and troubleshooting, a data model is of little use if it cannot be interpreted by a human operator. In this paper, we present an extension of the Multivariate Big Data Analysis (MBDA) methodology, a recently proposed interpretable data analysis tool. In this extension, we propose a solution to the automatic derivation of features, a cornerstone step for the application of MBDA when the amount of data is massive. The resulting network monitoring approach allows us to detect and diagnose disparate network anomalies, with a data-analysis workflow that combines the advantages of interpretable and interactive models with the power of parallel processing. We apply the extended MBDA to two case studies: UGR\u2716, a benchmark flow-based real-traffic dataset for anomaly detection, and Dartmouth\u2718, the longest and largest Wi-Fi trace known to date

Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks

Multivariate time series forecasting is an important machine learning problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. Temporal data arise in these real-world applications often involves a mixture of long-term and short-term patterns, for which traditional approaches such as Autoregressive models and Gaussian Process may fail. In this paper, we proposed a novel deep learning framework, namely Long- and Short-term Time-series network (LSTNet), to address this open challenge. LSTNet uses the Convolution Neural Network (CNN) and the Recurrent Neural Network (RNN) to extract short-term local dependency patterns among variables and to discover long-term patterns for time series trends. Furthermore, we leverage traditional autoregressive model to tackle the scale insensitive problem of the neural network model. In our evaluation on real-world data with complex mixtures of repetitive patterns, LSTNet achieved significant performance improvements over that of several state-of-the-art baseline methods. All the data and experiment codes are available online.Comment: Accepted by SIGIR 201

Modelling network travel time reliability under stochastic demand

A technique is proposed for estimating the probability distribution of total network travel time, in the light of normal day-to-day variations in the travel demand matrix over a road traffic network. A solution method is proposed, based on a single run of a standard traffic assignment model, which operates in two stages. In stage one, moments of the total travel time distribution are computed by an analytic method, based on the multivariate moments of the link flow vector. In stage two, a flexible family of density functions is fitted to these moments. It is discussed how the resulting distribution may in practice be used to characterise unreliability. Illustrative numerical tests are reported on a simple network, where the method is seen to provide a means for identifying sensitive or vulnerable links, and for examining the impact on network reliability of changes to link capacities. Computational considerations for large networks, and directions for further research, are discussed

Towards a Reliable Comparison and Evaluation of Network Intrusion Detection Systems Based on Machine Learning Approaches

Presently, we are living in a hyper-connected world where millions of heterogeneous devices are continuously sharing information in different application contexts for wellness, improving communications, digital businesses, etc. However, the bigger the number of devices and connections are, the higher the risk of security threats in this scenario. To counteract against malicious behaviours and preserve essential security services, Network Intrusion Detection Systems (NIDSs) are the most widely used defence line in communications networks. Nevertheless, there is no standard methodology to evaluate and fairly compare NIDSs. Most of the proposals elude mentioning crucial steps regarding NIDSs validation that make their comparison hard or even impossible. This work firstly includes a comprehensive study of recent NIDSs based on machine learning approaches, concluding that almost all of them do not accomplish with what authors of this paper consider mandatory steps for a reliable comparison and evaluation of NIDSs. Secondly, a structured methodology is proposed and assessed on the UGR'16 dataset to test its suitability for addressing network attack detection problems. The guideline and steps recommended will definitively help the research community to fairly assess NIDSs, although the definitive framework is not a trivial task and, therefore, some extra effort should still be made to improve its understandability and usability further

User equilibrium traffic network assignment with stochastic travel times and late arrival penalty

The classical Wardrop user equilibrium (UE) assignment model assumes traveller choices are based on fixed, known travel times, yet these times are known to be rather variable between trips, both within and between days; typically, then, only mean travel times are represented. Classical stochastic user equilibrium (SUE) methods allow the mean travel times to be differentially perceived across the population, yet in a conventional application neither the UE or SUE approach recognises the travel times to be inherently variable. That is to say, there is no recognition that drivers risk arriving late at their destinations, and that this risk may vary across different paths of the network and according to the arrival time flexibility of the traveller. Recent work on incorporating risky elements into the choice process is seen either to neglect the link to the arrival constraints of the traveller, or to apply only to restricted problems with parallel alternatives and inflexible travel time distributions. In the paper, an alternative approach is described based on the ‘schedule delay’ paradigm, penalising late arrival under fixed departure times. The approach allows flexible travel time densities, which can be fitted to actual surveillance data, to be incorporated. A generalised formulation of UE is proposed, termed a Late Arrival Penalised UE (LAPUE). Conditions for the existence and uniqueness of LAPUE solutions are considered, as well as methods for their computation. Two specific travel time models are then considered, one based on multivariate Normal arc travel times, and an extended model to represent arc incidents, based on mixture distributions of multivariate Normals. Several illustrative examples are used to examine the sensitivity of LAPUE solutions to various input parameters, and in particular its comparison with UE predictions. Finally, paths for further research are discussed, including the extension of the model to include elements such as distributed arrival time constraints and penalties

Sensitivity analysis of the variable demand probit stochastic user equilibrium with multiple user classes

This paper presents a formulation of the multiple user class, variable demand, probit stochastic user equilibrium model. Sufficient conditions are stated for differentiability of the equilibrium flows of this model. This justifies the derivation of sensitivity expressions for the equilibrium flows, which are presented in a format that can be implemented in commercially available software. A numerical example verifies the sensitivity expressions, and that this formulation is applicable to large networks