Emergent Deep Learning for Anomaly Detection in Internet of Everything

This research presents a new generic deep learning framework for anomaly detection in the Internet of Everything (IoE). It combines decomposition methods, deep neural networks, and evolutionary computation to better detect outliers in IoE environments. The dataset is first decomposed into clusters, while similar observations in the same cluster are grouped. Five clustering algorithms were used for this purpose. The generated clusters are then trained using Deep Learning architectures. In this context, we propose a new recurrent neural network for training time series data. Two evolutionary computational algorithms are also proposed: the genetic and the bee swarm to fine-tune the training step. These algorithms consider the hyper-parameters of the trained models and try to find the optimal values. The proposed solutions have been experimentally evaluated for two use cases: 1) road traffic outlier detection and 2) network intrusion detection. The results show the advantages of the proposed solutions and a clear superiority compared to state-of-the-art approaches.acceptedVersio

Classification and Prediction of Business Incidents Using Deep Learning for Anomaly Detection

Tarkvarasüsteemid omavad tänapäeva äriettevõtetes elutähtsaid funktsioone ja nad on tihti ka äritegevuseks primaarse tähtsusega. Taolised süsteemid võivad koosneda väga suurest hulgast komponentidest, mis on arendatud erinevate meeskondade või ettevõtete poolt ning enamasti ka kasutades erinevaid tehnoloogiaid. Keerukate süsteemide korral võivad olla vead nii rakendustes kui ka võrgus. Probleemid võivad ilmneda konfigureerimisel, mis võib põhjustada ootamatuid pöördeid ärivoos, samuti võivad versiooniuuendused tekitada kooskõlaprobleeme. See kõik võib põhjustada ärile maine- ja finantsilist kahju. Seetõttu on ärile vajalikud proaktiivsed sammud, et tulla toime äriintsidentidega enne nende ebasoodsat mõju teistele komponentidele. See toob kaasa vajaduse analüütilise platvormi järele, kus oleks võimalik eristada süsteemi normaalset käitumist anomaalsest meetrika alusel. Playtech plc kasutab taoliseks automaatseks tuvastamiseks ja häirete tõstatamiseks tüüpilist anomaaliate tuvastamise lähenemist: reeglitel põhinevat tuvastamist. Playtech plc, tarkvarasüsteemides jälgitakse tuhandeid meetrikuid, alustades infrastruktuuri ja süsteemitarkvara ning lõpetades rakenduste ja ärimeetrikutega. Samas on tarkvara paigaldatud ja opereerib rohkem kui 40-s asukohas, igas neist erinevate lõppkasutajate ning ärimudelitest tulenevate erinevustega. Lisaks sellele, on tarkvara pidevas muutumises, nädalaste arendustsüklite tulemustena uuendatakse igal teisel nädalal komponente üle kõigi asukohtade ja paigalduste. Reeglitel põhinev lähenemine on piisavalt efektiivne tuvastamise kiiruse ja täpsuse osas, kuid nõuab palju inimressursse reeglite haldamise ja täppisseadistamise tõttu sellises muutuvas keskkonnas. Seetõttu nähti vajadust leida lahendus mis suutaks automaatselt kohaneda muutuvas keskkonnas ning erinevates tarkvara seadistustes ilma inimese pideva sekkumiseta.Antud töö eesmärk ongi masinõppel põhineva mudeli väljatöötamine ja treenimine, mis tuvastaks ja kategoriseeriks taolisi intsidente. Töö kirjeldab detailselt, kuidas kasutatakse anomaaliate tuvastamise ja süvaõppe tehnikaid täiendamaks olemasolevat lahendust intsidentide tuvastamisel ja klassifitseerimisel.Companies today use a number of software systems to carry out various business activities. Such enterprise standard software solutions consist of a large number of components usually developed by different teams and/or different software vendors using various technologies. In such complex software systems, there can be various issues ranging from problems in the software itself to issues in network.In order to measure the operational performance of applications and infrastructure as well as key performance indicators (KPIs) that evaluate the success of the organization, a lot of business metrics is collected. These metrics have certain data patterns which represent normal business behaviour. Anomalies are some unexpected changes within these data patterns such as degradation or a sudden surge in business metrics values. Additionally, small changes in software system configuration can cause unexpected behaviour in business flows. Version upgrades of different components can introduce compatibility problems. These problems could lead to a change in the normal behaviour of business metrics and cause anomalies. These anomalies if not resolved quickly results in business and financial losses. Therefore, it is necessary for businesses to take proactive steps to manage such business incidents before they can adversely affect it. This brings us to the need for an analytics platform which can analyze patterns of data streams, identify and differentiate normal behaviour of a business metric from anomalous behaviour and could generate a notification.The current anomaly detection and alert system in Playtech plc uses a simple anomaly detection technique that follows a rule based approach and it is observed that it is not efficient. Thus, a more robust, modular and efficient business incident/anomaly detection solution based on advanced machine learning techniques is needed that could work in conjugation with the current system. This thesis proposes, describes and evaluates a business incident/anomaly detection system based on deep learning approach that categorises and predicts the business incidents/anomalies using the available business metrics information

Deep learning for anomaly detection in computer vision

Doctor of PhilosophyDepartment of Computer ScienceWilliam H HsuRecently, deep learning (DL) inspired algorithms have performed remarkably well on many tasks such as machine translation, speech recognition and image classification etc. However, existing state-of-the-art algorithms struggle to learn the discriminative signals between normal and abnormal classes in an anomaly setting. This is due to the fact that these signals are subtle which often comes in form of little deviations in color, shape, structure etc. To tackle this problem, I propose a more efficient approach named AdeNet that requires lower computation and storage, making it more practical for use on edge devices. Anomaly detection use cases often involves the problem of class imbalance, a case where there are overwhelming samples (majority class) of one or more classes as compared to the others (minority class). This problem also contributes to the inability of DL algorithms to learn distinguishing signals. To address this, I propose an Encoder-based Generative Adversarial Network (eGAN) that leverages on pre-trained model to learn a separable distribution of these classes. Another associated problem to detecting anomalies is zero-shot learning (ZSL). This occurs as a result of the fact that it is practically infeasible to present to our model all possible instances of anomalies during training. Yet, we want models that are robust to new unseen out-of-distribution (OOD) samples during inference. Here, I employ the concept of contrastive learning (CL) to tackle this problem by using pretext task that learns to push embeddings of dissimilar classes far apart, and pull embeddings of similar classes. This seemingly simple concept forces the network to learn salient visual signals that are generalizable to identifying zero-shot instances

A Survey of Graph-based Deep Learning for Anomaly Detection in Distributed Systems

Anomaly detection is a crucial task in complex distributed systems. A thorough understanding of the requirements and challenges of anomaly detection is pivotal to the security of such systems, especially for real-world deployment. While there are many works and application domains that deal with this problem, few have attempted to provide an in-depth look at such systems. In this survey, we explore the potentials of graph-based algorithms to identify anomalies in distributed systems. These systems can be heterogeneous or homogeneous, which can result in distinct requirements. One of our objectives is to provide an in-depth look at graph-based approaches to conceptually analyze their capability to handle real-world challenges such as heterogeneity and dynamic structure. This study gives an overview of the State-of-the-Art (SotA) research articles in the field and compare and contrast their characteristics. To facilitate a more comprehensive understanding, we present three systems with varying abstractions as use cases. We examine the specific challenges involved in anomaly detection within such systems. Subsequently, we elucidate the efficacy of graphs in such systems and explicate their advantages. We then delve into the SotA methods and highlight their strength and weaknesses, pointing out the areas for possible improvements and future works.Comment: The first two authors (A. Danesh Pazho and G. Alinezhad Noghre) have equal contribution. The article is accepted by IEEE Transactions on Knowledge and Data Engineerin

Emergent deep learning for anomaly detection in internet of everything

This research presents a new generic deep learning framework for anomaly detection in the Internet of Everything (IoE). It combines decomposition methods, deep neural networks, and evolutionary computation to better detect outliers in IoE environments. The dataset is first decomposed into clusters, while similar observations in the same cluster are grouped. Five clustering algorithms were used for this purpose. The generated clusters are then trained using Deep Learning architectures. In this context, we propose a new recurrent neural network for training time series data. Two evolutionary computational algorithms are also proposed: the genetic and the bee swarm to fine-tune the training step. These algorithms consider the hyper-parameters of the trained models and try to find the optimal values. The proposed solutions have been experimentally evaluated for two use cases: 1) road traffic outlier detection and 2) network intrusion detection. The results show the advantages of the proposed solutions and a clear superiority compared to state-of-the-art approaches

Adaptable and Explainable Predictive Maintenance: Semi-Supervised Deep Learning for Anomaly Detection and Diagnosis in Press Machine Data

Predictive maintenance (PdM) has the potential to reduce industrial costs by anticipating failures and extending the work life of components. Nowadays, factories are monitoring their assets and most collected data belong to correct working conditions. Thereby, semi-supervised data-driven models are relevant to enable PdM application by learning from assets’ data. However, their main challenges for application in industry are achieving high accuracy on anomaly detection, diagnosis of novel failures, and adaptability to changing environmental and operational conditions (EOC). This article aims to tackle these challenges, experimenting with algorithms in press machine data of a production line. Initially, state-of-the-art and classic data-driven anomaly detection model performance is compared, including 2D autoencoder, null-space, principal component analysis (PCA), one-class support vector machines (OC-SVM), and extreme learning machine (ELM) algorithms. Then, diagnosis tools are developed supported on autoencoder’s latent space feature vector, including clustering and projection algorithms to cluster data of synthetic failure types semi-supervised. In addition, explainable artificial intelligence techniques have enabled to track the autoencoder’s loss with input data to detect anomalous signals. Finally, transfer learning is applied to adapt autoencoders to changing EOC data of the same process. The data-driven techniques used in this work can be adapted to address other industrial use cases, helping stakeholders gain trust and thus promote the adoption of data-driven PdM systems in smart factories

Analytics and Artificial Intelligence: Deep Learning for Anomaly Detection - A case study from the financial sector with application to process safety

PresentationThis presentation reports on a case study from the financial sector with application to challenges in the field of process safety. Banks collect massive amounts of data from routine financial transactions. Some of the data is anomalous, is corrupt, or represents a signal that warrants follow-up attention from subject matter experts. Currently, review of such data is often a manual inspection which is time consuming, expensive, and limited to representative data sets. In this case study, we employ machine learning, deep learning to rapidly review historical data and tag anomalous data that represents a signal for follow-up attention. With this methodology, we are able to automate the manual process, quickly finding anomalies, dramatically reducing assessment time, expanding the range and volume of data that can be reviewed, and finding signals that previously would likely be missed. Benefits to the financial institution include major cost reductions and improvements in detection of fraud. Application of the machine learning/data assessment approach to process safety challenges may provide safety and cost-reduction benefits as well

A scalable machine learning system for anomaly detection in manufacturing

Berichte über Rückrufaktionen in der Automobilindustrie gehören inzwischen zum medialen Alltag. Tatsächlich hat deren Häufigkeit und die Anzahl der betroffenen Fahrzeuge in den letzten Jahren weiter zugenommen. Die meisten Aktionen sind auf Fehler in der Produktion zurückzuführen. Für die Hersteller stellt neben Verbesserungen im Qualitätsmanagement die intelligente und automatisierte Analyse von Produktionsprozessdaten ein bislang kaum ausgeschöpftes Potential dar. Die technischen Herausforderungen sind jedoch enorm: die Datenmengen sind gewaltig und die für einen Fehler charakteristischen Datenmuster zwangsläufig unbekannt. Der Einsatz maschineller Lernverfahren (ML) ist ein vielversprechender Ansatz um diese Suche nach der sinnbildlichen Nadel im Häuhaufen zu ermöglichen. Algorithmen sollen anhand der Daten selbständig lernen zwischen normalem und auffälligem Prozessverhalten zu unterscheiden um Prozessexperten frühzeitig zu warnen. Industrie und Forschung versuchen bereits seit Jahren solche ML-Systeme im Produktionsumfeld zu etablieren. Die meisten ML-Projekte scheitern jedoch bereits vor der Produktivphase bzw. verschlingen enorme Ressourcen im Betrieb und liefern keinen wirtschaftlichen Mehrwert. Ziel der Arbeit ist die Entwicklung eines technischen Frameworks zur Implementierung eines skalierbares ML-System für die Anomalieerkennung in Prozessdaten. Die Trainingsprozesse zum Initialisieren und Adaptieren der Modelle sollen hochautomatisierbar sein um einen strukturierten Skalierungsprozess zu ermöglichen. Das entwickelt DM/ML-Verfahren ermöglicht den langfristigen Aufwand für den Systembetrieb durch initialen Mehraufwand für den Modelltrainingsprozess zu senken und hat sich in der Praxis als sowohl relativ als auch absolut Skalierbar bewährt. Dadurch kann die Komplexität auf Systemebene auf ein beherrschbares Maß reduziert werden um einen späteren Systembetrieb zu ermöglichen

Resilient VAE: Unsupervised Anomaly Detection at the SLAC Linac Coherent Light Source

Significant advances in utilizing deep learning for anomaly detection have been made in recent years. However, these methods largely assume the existence of a normal training set (i.e., uncontaminated by anomalies) or even a completely labeled training set. In many complex engineering systems, such as particle accelerators, labels are sparse and expensive; in order to perform anomaly detection in these cases, we must drop these assumptions and utilize a completely unsupervised method. This paper introduces the Resilient Variational Autoencoder (ResVAE), a deep generative model specifically designed for anomaly detection. ResVAE exhibits resilience to anomalies present in the training data and provides feature-level anomaly attribution. During the training process, ResVAE learns the anomaly probability for each sample as well as each individual feature, utilizing these probabilities to effectively disregard anomalous examples in the training data. We apply our proposed method to detect anomalies in the accelerator status at the SLAC Linac Coherent Light Source (LCLS). By utilizing shot-to-shot data from the beam position monitoring system, we demonstrate the exceptional capability of ResVAE in identifying various types of anomalies that are visible in the accelerator