Insider Threat Detection in PRODIGAL

This paper reports on insider threat detection research, during which a prototype system (PRODIGAL) was developed and operated as a testbed for exploring a range of detection and analysis methods. The data and test environment, system components, and the core method of unsupervised detection \ of insider threat leads are presented to document this work and benefit others working in the insider threat domain. \ \ We also discuss a core set of experiments evaluating the prototype’s ability to detect both known and unknown malicious insider behaviors. The experimental results show the ability to detect a large variety of insider threat scenario instances imbedded in real data with no prior knowledge of what scenarios \ are present or when they occur. \ \ We report on an ensemble-based, unsupervised technique for detecting potential insider threat instances. When run over 16 months of real monitored computer usage activity augmented with independently developed and unknown but realistic, insider threat scenarios, this technique robustly achieves results within five percent of the best individual detectors identified after the fact. We discuss factors that contribute to the success of the ensemble method, such as the number and variety of unsupervised detectors and the use of prior knowledge encoded in detectors designed for specific activity patterns. \ \ Finally, the paper describes the architecture of the prototype system, the environment in which we conducted these experiments and that is in the process of being transitioned to operational users

Intrusion Detection System using Bayesian Network Modeling

Computer Network Security has become a critical and important issue due to ever increasing cyber-crimes. Cybercrimes are spanning from simple piracy crimes to information theft in international terrorism. Defence security agencies and other militarily related organizations are highly concerned about the confidentiality and access control of the stored data. Therefore, it is really important to investigate on Intrusion Detection System (IDS) to detect and prevent cybercrimes to protect these systems. This research proposes a novel distributed IDS to detect and prevent attacks such as denial service, probes, user to root and remote to user attacks. In this work, we propose an IDS based on Bayesian network classification modelling technique. Bayesian networks are popular for adaptive learning, modelling diversity network traffic data for meaningful classification details. The proposed model has an anomaly based IDS with an adaptive learning process. Therefore, Bayesian networks have been applied to build a robust and accurate IDS. The proposed IDS has been evaluated against the KDD DAPRA dataset which was designed for network IDS evaluation. The research methodology consists of four different Bayesian networks as classification models, where each of these classifier models are interconnected and communicated to predict on incoming network traffic data. Each designed Bayesian network model is capable of detecting a major category of attack such as denial of service (DoS). However, all four Bayesian networks work together to pass the information of the classification model to calibrate the IDS system. The proposed IDS shows the ability of detecting novel attacks by continuing learning with different datasets. The testing dataset constructed by sampling the original KDD dataset to contain balance number of attacks and normal connections. The experiments show that the proposed system is effective in detecting attacks in the test dataset and is highly accurate in detecting all major attacks recorded in DARPA dataset. The proposed IDS consists with a promising approach for anomaly based intrusion detection in distributed systems. Furthermore, the practical implementation of the proposed IDS system can be utilized to train and detect attacks in live network traffi

Insider Threat Detection using Virtual Machine Introspection

This paper presents a methodology for signaling potentially malicious insider behavior using virtual machine introspection (VMI). VMI provides a novel means to detect potential malicious insiders because the introspection tools remain transparent and inaccessible to the guest and are extremely difficult to subvert. This research develops a four step methodology for development and validation of malicious insider threat alerting using VMI. A malicious attacker taxonomy is used to decompose each scenario to aid identification of observables for monitoring for potentially malicious actions. The effectiveness of the identified observables is validated using two data sets. Results of the research show the developed methodology is effective in detecting the malicious insider scenarios on Windows guests

Insider Threat Detection on the Windows Operating System using Virtual Machine Introspection

Existing insider threat defensive technologies focus on monitoring network traffic or events generated by activities on a user\u27s workstation. This research develops a methodology for signaling potentially malicious insider behavior using virtual machine introspection (VMI). VMI provides a novel means to detect potential malicious insiders because the introspection tools remain transparent and inaccessible to the guest and are extremely difficult to subvert. This research develops a four step methodology for development and validation of malicious insider threat alerting using VMI. Six core use cases are developed along with eighteen supporting scenarios. A malicious attacker taxonomy is used to decompose each scenario to aid identification of observables for monitoring for potentially malicious actions. The effectiveness of the identified observables is validated through the use of two data sets, one containing simulated normal and malicious insider user behavior and the second from a computer network operations exercise. Compiled Memory Analysis Tool - Virtual (CMAT-V) and Xen hypervisor capabilities are leveraged to perform VMI and insider threat detection. Results of the research show the developed methodology is effective in detecting all defined malicious insider scenarios used in this research on Windows guests

Internal hacking detection using machine learning

Tese de mestrado, Ciência de Dados, Universidade de Lisboa, Faculdade de Ciências, 2020Being able to prevent and early detect insider threats through an automated forewarning system has been a massive challenge for large companies. In recent years, to fill this gap several anomaly user behavior algorithms based on machine learning have been proposed, experimentally evaluated and analyzed in numerous surveys. The present work was conducted in the cybersecurity department (DCY) of Altice Portugal (MEO) and aims to address this problem identifying the families of unsupervised anomaly detection techniques that are more effective for insider threats detection based on a large dataset corresponding to a collection of users’ access log records. To this end, multi-domain attributes related to possible insider threats are interactively extracted and processed, creating a summary of user account’s daily activity. A clusteringbased algorithm that groups and characterizes similar accounts was applied. Without any example anomalies required in the training set, anomaly detection techniques were computed over those profiles, identifying unusual changes in user account behavior on a current day. Finally, to make it easier for analysts and managers to understand the anomaly, anomaly metrics and a visualization dashboard were created. To evaluate the efficiency of this project ten insider threat scenarios were injected and was found that the system can successfully detect anomalous behavior that may be an insider threat event

Anomaly-based insider threat detection with expert feedback and descriptions

Abstract. Insider threat is one of the most significant security risks for organizations, hence insider threat detection is an important task. Anomaly detection is a one approach to insider threat detection. Anomaly detection techniques can be categorized into three categories with respect to how much labelled data is needed: unsupervised, semi-supervised and supervised. Obtaining accurate labels of all kinds of incidents for supervised learning is often expensive and impractical. Unsupervised methods do not require labelled data, but they have a high false positive rate because they operate on the assumption that anomalies are rarer than nominals. This can be mitigated by introducing feedback, known as expert-feedback or active learning. This allows the analyst to label a subset of the data. Another problem is the fact that models often are not interpretable, thus it is unclear why the model decided that a data instance is an anomaly. This thesis presents a literature review of insider threat detection, unsupervised and semi-supervised anomaly detection. The performance of various unsupervised anomaly detectors are evaluated. Knowledge is introduced into the system by using state-of-the-art feedback technique for ensembles, known as active anomaly discovery, which is incorporated into the anomaly detector, known as isolation forest. Additionally, to improve interpretability techniques of creating rule-based descriptions for the isolation forest are evaluated. Experiments were performed on CMU-CERT dataset, which is the only publicly available insider threat dataset with logon, removable device and HTTP log data. Models use usage count and session-based features that are computed for users on every day. The results show that active anomaly discovery helps in ranking true positives higher on the list, lowering the amount of data analysts have to analyse. Results also show that both compact description and Bayesian rulesets have the potential to be used in generating decision-rules that aid in analysing incidents; however, these rules are not correct in every instance.Poikkeamapohjainen sisäpiiriuhkien havainta palautteen ja kuvauksien avulla. Tiivistelmä. Sisäpiirinuhat ovat yksi vakavimmista riskeistä organisaatioille. Tästä syystä sisäpiiriuhkien havaitseminen on tärkeää. Sisäpiiriuhkia voidaan havaita poikkeamien havaitsemismenetelmillä. Nämä menetelmät voidaan luokitella kolmeen oppimisluokkaan saatavilla olevan tietomäärän perusteella: ohjaamaton, puoli-ohjattu ja ohjattu. Täysin oikein merkatun tiedon saaminen ohjattua oppimista varten voi olla hyvin kallista ja epäkäytännöllistä. Ohjaamattomat oppimismenetelmät eivät vaadi merkattua tietoa, mutta väärien positiivisten osuus on suurempi, koska nämä menetelmät perustuvat oletukseen että poikkeamat ovat harvinaisempia kuin normaalit tapaukset. Väärien positiivisten osuutta voidaan pienentää ottamalla käyttöön palaute, jolloin analyytikko voi merkata osan datasta. Tässä opinnäytetyössä tutustutaan ensin sisäpiiriuhkien havaitsemiseen, mitä tutkimuksia on tehty ja ohjaamattomaan ja puoli-ohjattuun poikkeamien havaitsemiseen. Muutamien lupaavien ohjaamattomien poikkeamatunnistimien toimintakyky arvioidaan. Järjestelmään lisätään tietoisuutta havaitsemisongelmasta käyttämällä urauurtavaa active anomaly discovery -palautemetelmää, joka on tehty havaitsinjoukoille (engl. ensembles). Tätä arvioidaan Isolation Forest -havaitsimen kanssa. Lisäksi, jotta analytiikko pystyisi paremmin käsittelemään havainnot, tässä työssä myös arvioidaan sääntöpohjaisten kuvausten luontimenetelmä Isolation Forest -havaitsimelle. Kokeilut suoritettiin käyttäen julkista CMU-CERT:in aineistoa, joka on ainoa julkinen aineisto, missä on muun muuassa kirjautumis-, USB-laite- ja HTTP-tapahtumia. Mallit käyttävät käyttöluku- ja istuntopohjaisia piirteitä, jotka luodaan jokaista käyttäjää ja päivää kohti. Tuloksien perusteella Active Anomaly Discovery auttaa epäilyttävämpien tapahtumien sijoittamisessa listan kärkeen vähentäen tiedon määrä, jonka analyytikon tarvitsee tutkia. Kompaktikuvakset (engl. compact descriptions)- ja Bayesian sääntöjoukko -menetelmät pystyvät luomaan sääntöjä, jotka kuvaavat minkä takia tapahtuma on epäilyttävä, mutta nämä säännöt eivät aina ole oikein