WAVE-CUSUM: Improving CUSUM performance in network anomaly detection by means of wavelet analysis

n/

Structure-Aware Sampling: Flexible and Accurate Summarization

In processing large quantities of data, a fundamental problem is to obtain a summary which supports approximate query answering. Random sampling yields flexible summaries which naturally support subset-sum queries with unbiased estimators and well-understood confidence bounds. Classic sample-based summaries, however, are designed for arbitrary subset queries and are oblivious to the structure in the set of keys. The particular structure, such as hierarchy, order, or product space (multi-dimensional), makes range queries much more relevant for most analysis of the data. Dedicated summarization algorithms for range-sum queries have also been extensively studied. They can outperform existing sampling schemes in terms of accuracy on range queries per summary size. Their accuracy, however, rapidly degrades when, as is often the case, the query spans multiple ranges. They are also less flexible - being targeted for range sum queries alone - and are often quite costly to build and use. In this paper we propose and evaluate variance optimal sampling schemes that are structure-aware. These summaries improve over the accuracy of existing structure-oblivious sampling schemes on range queries while retaining the benefits of sample-based summaries: flexible summaries, with high accuracy on both range queries and arbitrary subset queries

Advanced Techniques for Detecting Anomalies in Backbone Networks

Con il rapido sviluppo e la crescente complessita' delle reti di computer, i meccanismi tradizionali di network security non riescono a fornire soluzioni dinamiche e integrate adatte a garantire la completa sicurezza di un sistema. In questo contesto, l’uso di sistemi per la rilevazione delle intrusioni (Intrusion Detection System - IDS) e' diventato un elemento chiave nell’ambito della sicurezza delle reti. In questo lavoro di tesi affrontiamo tale problematica, proponendo soluzioni innovative per l’intrusion detection, basate sull’uso di tecniche statistiche (Wavelet Aanalysis, Principal Component Analysis, etc.) la cui applicazione per la rilevazione delle anomalie nel traffico di rete, risulta del tutto originale. L’analisi dei risultati presentata, in questo lavoro di tesi, evidenzia l’efficacia dei metodi proposti

Empirical assessment of VoIP overload detection tests

The control of communication networks critically relies on procedures capable of detecting unanticipated load changes. In this paper we explore such techniques, in a setting in which each connection consumes roughly the same amount of bandwidth (with VoIP as a leading example). We focus on large-deviations based techniques developed earlier in that monitor the number of connections present, and that issue an alarm when this number abruptly changes. The procedures proposed in are demonstrated by using real traces from an operational environment. Our experiments show that our detection procedure is capable of adequately identifying load changes

Performance Evaluation of Network Anomaly Detection Systems

Nowadays, there is a huge and growing concern about security in information and communication technology (ICT) among the scientific community because any attack or anomaly in the network can greatly affect many domains such as national security, private data storage, social welfare, economic issues, and so on. Therefore, the anomaly detection domain is a broad research area, and many different techniques and approaches for this purpose have emerged through the years. Attacks, problems, and internal failures when not detected early may badly harm an entire Network system. Thus, this thesis presents an autonomous profile-based anomaly detection system based on the statistical method Principal Component Analysis (PCADS-AD). This approach creates a network profile called Digital Signature of Network Segment using Flow Analysis (DSNSF) that denotes the predicted normal behavior of a network traffic activity through historical data analysis. That digital signature is used as a threshold for volume anomaly detection to detect disparities in the normal traffic trend. The proposed system uses seven traffic flow attributes: Bits, Packets and Number of Flows to detect problems, and Source and Destination IP addresses and Ports, to provides the network administrator necessary information to solve them. Via evaluation techniques, addition of a different anomaly detection approach, and comparisons to other methods performed in this thesis using real network traffic data, results showed good traffic prediction by the DSNSF and encouraging false alarm generation and detection accuracy on the detection schema. The observed results seek to contribute to the advance of the state of the art in methods and strategies for anomaly detection that aim to surpass some challenges that emerge from the constant growth in complexity, speed and size of today’s large scale networks, also providing high-value results for a better detection in real time.Atualmente, existe uma enorme e crescente preocupação com segurança em tecnologia da informação e comunicação (TIC) entre a comunidade científica. Isto porque qualquer ataque ou anomalia na rede pode afetar a qualidade, interoperabilidade, disponibilidade, e integridade em muitos domínios, como segurança nacional, armazenamento de dados privados, bem-estar social, questões econômicas, e assim por diante. Portanto, a deteção de anomalias é uma ampla área de pesquisa, e muitas técnicas e abordagens diferentes para esse propósito surgiram ao longo dos anos. Ataques, problemas e falhas internas quando não detetados precocemente podem prejudicar gravemente todo um sistema de rede. Assim, esta Tese apresenta um sistema autônomo de deteção de anomalias baseado em perfil utilizando o método estatístico Análise de Componentes Principais (PCADS-AD). Essa abordagem cria um perfil de rede chamado Assinatura Digital do Segmento de Rede usando Análise de Fluxos (DSNSF) que denota o comportamento normal previsto de uma atividade de tráfego de rede por meio da análise de dados históricos. Essa assinatura digital é utilizada como um limiar para deteção de anomalia de volume e identificar disparidades na tendência de tráfego normal. O sistema proposto utiliza sete atributos de fluxo de tráfego: bits, pacotes e número de fluxos para detetar problemas, além de endereços IP e portas de origem e destino para fornecer ao administrador de rede as informações necessárias para resolvê-los. Por meio da utilização de métricas de avaliação, do acrescimento de uma abordagem de deteção distinta da proposta principal e comparações com outros métodos realizados nesta tese usando dados reais de tráfego de rede, os resultados mostraram boas previsões de tráfego pelo DSNSF e resultados encorajadores quanto a geração de alarmes falsos e precisão de deteção. Com os resultados observados nesta tese, este trabalho de doutoramento busca contribuir para o avanço do estado da arte em métodos e estratégias de deteção de anomalias, visando superar alguns desafios que emergem do constante crescimento em complexidade, velocidade e tamanho das redes de grande porte da atualidade, proporcionando também alta performance. Ainda, a baixa complexidade e agilidade do sistema proposto contribuem para que possa ser aplicado a deteção em tempo real

Terrestrial Laser Scanning-Based Bridge Structural Condition Assessment

Objective, accurate, and fast assessment of a bridge’s structural condition is critical to the timely assessment of safety risks. Current practices for bridge condition assessment rely on visual observations and manual interpretation of reports and sketches prepared by inspectors in the field. Visual observation, manual reporting, and interpretation have several drawbacks, such as being labor intensive, subject to personal judgment and experience, and prone to error. Terrestrial laser scanners (TLS) are promising sensors for automatically identifying structural condition indicators, such as cracks, displacements, and deflected shapes, because they are able to provide high coverage and accuracy at long ranges. However, limited research has been conducted on employing laser scanners to detect cracks for bridge condition assessment, and the research has mainly focused on manual detection and measurement of cracks, displacements, or shape deflections from the laser scan point clouds. This research project proposed to measure the performance of TLS for the automatic detection of cracks for bridge structural condition assessment. Laser scanning is an advanced imaging technology that is used to rapidly measure the three-dimensional (3D) coordinates of densely scanned points within a scene. The data gathered by a laser scanner are provided in the form of point clouds, with color and intensity data often associated with each point within the cloud. Point cloud data can be analyzed using computer vision algorithms to detect cracks for the condition assessment of reinforced concrete structures. In this research project, adaptive wavelet neural network (WNN) algorithms for detecting cracks from laser scan point clouds were developed based on the state-of-the-art condition assessment codes and standards. Using the proposed method for crack detection would enable automatic and remote assessment of a bridge’s condition. This would, in turn, result in reducing the costs associated with infrastructure management and improving the overall quality of our infrastructure by enhancing maintenance operations

Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin