Detection of Early-Stage Enterprise Infection by Mining Large-Scale Log Data

Recent years have seen the rise of more sophisticated attacks including advanced persistent threats (APTs) which pose severe risks to organizations and governments by targeting confidential proprietary information. Additionally, new malware strains are appearing at a higher rate than ever before. Since many of these malware are designed to evade existing security products, traditional defenses deployed by most enterprises today, e.g., anti-virus, firewalls, intrusion detection systems, often fail at detecting infections at an early stage. We address the problem of detecting early-stage infection in an enterprise setting by proposing a new framework based on belief propagation inspired from graph theory. Belief propagation can be used either with "seeds" of compromised hosts or malicious domains (provided by the enterprise security operation center -- SOC) or without any seeds. In the latter case we develop a detector of C&C communication particularly tailored to enterprises which can detect a stealthy compromise of only a single host communicating with the C&C server. We demonstrate that our techniques perform well on detecting enterprise infections. We achieve high accuracy with low false detection and false negative rates on two months of anonymized DNS logs released by Los Alamos National Lab (LANL), which include APT infection attacks simulated by LANL domain experts. We also apply our algorithms to 38TB of real-world web proxy logs collected at the border of a large enterprise. Through careful manual investigation in collaboration with the enterprise SOC, we show that our techniques identified hundreds of malicious domains overlooked by state-of-the-art security products

Behavior-based anomaly detection on big data

Recently, cyber-targeted attacks such as APT (Advanced Persistent Threat) are rapidly growing as a social and national threat. It is an intelligent cyber-attack that infiltrates the target organization and enterprise clandestinely using various methods and causes considerable damage by making a final attack after long-term and through preparations. These attacks are threatening cyber worlds such as Internet by infecting and attacking the devices on this environment with the malicious code, and by destroying them or gaining their authorities. Detecting these attacks requires collecting and analysing data from various sources (network, host, security equipment, and devices) over the long haul. Therefore, we propose the method that can recognize the cyber-targeted attack and detect the abnormal behavior based on Big Data. The proposed approach analyses faster and precisely various logs and monitoring data using Big Data storage and processing technology. In particular, we evaluated that the suspicious behavior analysis using MapReduce is effective in analysing large-scale behavior monitoring and log data from various sources

Towards more effective consumer steering via network analysis

Increased data gathering capacity, together with the spread of data analytics techniques, has prompted an unprecedented concentration of information related to the individuals' preferences in the hands of a few gatekeepers. In the present paper, we show how platforms' performances still appear astonishing in relation to some unexplored data and networks properties, capable to enhance the platforms' capacity to implement steering practices by means of an increased ability to estimate individuals' preferences. To this end, we rely on network science whose analytical tools allow data representations capable of highlighting relationships between subjects and/or items, extracting a great amount of information. We therefore propose a measure called Network Information Patrimony, considering the amount of information available within the system and we look into how platforms could exploit data stemming from connected profiles within a network, with a view to obtaining competitive advantages. Our measure takes into account the quality of the connections among nodes as the one of a hypothetical user in relation to its neighbourhood, detecting how users with a good neighbourhood -- hence of a superior connections set -- obtain better information. We tested our measures on Amazons' instances, obtaining evidence which confirm the relevance of information extracted from nodes' neighbourhood in order to steer targeted users

MULTI-DIMENSIONAL PROFILING OF CYBER THREATS FOR LARGE-SCALE NETWORKS

Current multi-domain command and control computer networks require significant oversight to ensure acceptable levels of security. Firewalls are the proactive security management tool at the network’s edge to determine malicious and benign traffic classes. This work aims to develop machine learning algorithms through deep learning and semi-supervised clustering, to enable the profiling of potential threats through network traffic analysis within large-scale networks. This research accomplishes these objectives by analyzing enterprise network data at the packet level using deep learning to classify traffic patterns. In addition, this work examines the efficacy of several machine learning model types and multiple imbalanced data handling techniques. This work also incorporates packet streams for identifying and classifying user behaviors. Tests of the packet classification models demonstrated that deep learning is sensitive to malicious traffic but underperforms in identifying allowed traffic compared to traditional algorithms. However, imbalanced data handling techniques provide performance benefits to some deep learning models. Conversely, semi-supervised clustering accurately identified and classified multiple user behaviors. These models provide an automated tool to learn and predict future traffic patterns. Applying these techniques within large-scale networks detect abnormalities faster and gives network operators greater awareness of user traffic.Outstanding ThesisCaptain, United States Marine CorpsApproved for public release. Distribution is unlimited