Machine Learning and Security of Non-Executable Files

Computer malware is a well-known threat in security which, despite the enormous time and effort invested in fighting it, is today more prevalent than ever. Recent years have brought a surge in one particular type: malware embedded in non-executable file formats, e.g., PDF, SWF and various office file formats. The result has been a massive number of infections, owed primarily to the trust that ordinary computer users have in these file formats. In addition, their feature-richness and implementation complexity have created enormous attack surfaces in widely deployed client software, resulting in regular discoveries of new vulnerabilities. The traditional approach to malware detection – signature matching, heuristics and behavioral profiling – has from its inception been a labor-intensive manual task, always lagging one step behind the attacker. With the exponential growth of computers and networks, malware has become more diverse, wide-spread and adaptive than ever, scaling much faster than the available talent pool of human malware analysts. An automated and scalable approach is needed to fill the gap between automated malware adaptation and manual malware detection, and machine learning is emerging as a viable solution. Its branch called adversarial machine learning studies the security of machine learning algorithms and the special conditions that arise when machine learning is applied for security. This thesis is a study of adversarial machine learning in the context of static detection of malware in non-executable file formats. It evaluates the effectiveness, efficiency and security of machine learning applications in this context. To this end, it introduces 3 data-driven detection methods developed using very large, high quality datasets. PJScan detects malicious PDF files based on lexical properties of embedded JavaScript code and is the fastest method published to date. SL2013 extends its coverage to all PDF files, regardless of JavaScript presence, by analyzing the hierarchical structure of PDF logical building blocks and demonstrates excellent performance in a novel long-term realistic experiment. Finally, Hidost generalizes the hierarchical-structure-based feature set to become the first machine-learning-based malware detector operating on multiple file formats. In a comprehensive experimental evaluation on PDF and SWF, it outperforms other academic methods and commercial antivirus systems in detection effectiveness. Furthermore, the thesis presents a framework for security evaluation of machine learning classifiers in a case study performed on an independent PDF malware detector. The results show that the ability to manipulate a part of the classifier’s feature set allows a malicious adversary to disguise malware so that it appears benign to the classifier with a high success rate. The presented methods are released as open-source software.Schadsoftware ist eine gut bekannte Sicherheitsbedrohung. Trotz der enormen Zeit und des Aufwands die investiert werden, um sie zu beseitigen, ist sie heute weiter verbreitet als je zuvor. In den letzten Jahren kam es zu einem starken Anstieg von Schadsoftware, welche in nicht-ausführbaren Dateiformaten, wie PDF, SWF und diversen Office-Formaten, eingebettet ist. Die Folge war eine massive Anzahl von Infektionen, ermöglicht durch das Vertrauen, das normale Rechnerbenutzer in diese Dateiformate haben. Außerdem hat die Komplexität und Vielseitigkeit dieser Dateiformate große Angriffsflächen in weitverbreiteter Klient-Software verursacht, und neue Sicherheitslücken werden regelmäßig entdeckt. Der traditionelle Ansatz zur Erkennung von Schadsoftware – Mustererkennung, Heuristiken und Verhaltensanalyse – war vom Anfang an eine äußerst mühevolle Handarbeit, immer einen Schritt hinter den Angreifern zurück. Mit dem exponentiellen Wachstum von Rechenleistung und Netzwerkgeschwindigkeit ist Schadsoftware diverser, zahlreicher und schneller-anpassend geworden als je zuvor, doch die Verfügbarkeit von menschlichen Schadsoftware-Analysten kann nicht so schnell skalieren. Ein automatischer und skalierbarer Ansatz ist gefragt, und maschinelles Lernen tritt als eine brauchbare Lösung hervor. Ein Bereich davon, Adversarial Machine Learning, untersucht die Sicherheit von maschinellen Lernverfahren und die besonderen Verhältnisse, die bei der Anwendung von machinellem Lernen für Sicherheit entstehen. Diese Arbeit ist eine Studie von Adversarial Machine Learning im Kontext statischer Schadsoftware-Erkennung in nicht-ausführbaren Dateiformaten. Sie evaluiert die Wirksamkeit, Leistungsfähigkeit und Sicherheit von maschinellem Lernen in diesem Kontext. Zu diesem Zweck stellt sie 3 datengesteuerte Erkennungsmethoden vor, die alle auf sehr großen und diversen Datensätzen entwickelt wurden. PJScan erkennt bösartige PDF-Dateien anhand lexikalischer Eigenschaften von eingebettetem JavaScript-Code und ist die schnellste bisher veröffentliche Methode. SL2013 erweitert die Erkennung auf alle PDF-Dateien, unabhängig davon, ob sie JavaScript enthalten, indem es die hierarchische Struktur von logischen PDF-Bausteinen analysiert. Es zeigt hervorragende Leistung in einem neuen, langfristigen und realistischen Experiment. Schließlich generalisiert Hidost den auf hierarchischen Strukturen basierten Merkmalsraum und wurde zum ersten auf maschinellem Lernen basierten Schadsoftware-Erkennungssystem, das auf mehreren Dateiformaten anwendbar ist. In einer umfassenden experimentellen Evaulierung auf PDF- und SWF-Formaten schlägt es andere akademische Methoden und kommerzielle Antiviren-Lösungen bezüglich Erkennungswirksamkeit. Überdies stellt diese Doktorarbeit ein Framework für Sicherheits-Evaluierung von auf machinellem Lernen basierten Klassifikatoren vor und wendet es in einer Fallstudie auf eine unabhängige akademische Schadsoftware-Erkennungsmethode an. Die Ergebnisse zeigen, dass die Fähigkeit, nur einen Teil von Features, die ein Klasifikator verwendet, zu manipulieren, einem Angreifer ermöglicht, Schadsoftware in Dateien so einzubetten, dass sie von der Erkennungsmethode mit hoher Erfolgsrate als gutartig fehlklassifiziert wird. Die vorgestellten Methoden wurden als Open-Source-Software veröffentlicht

An Analysis of Various Methods to Identify Web Based Applications Vulnerabilities

Recently Web based applications takes a noteworthy palace in people’s daily routine furthermore as in progress of nation’s different domains as well. Web based applications have undergone a reasonably express improvement within the last few decades and their appreciation is moving faster than that was predictable few years ago. Presently, huge volumes of transactions are prepared online using various Web based applications. Even though these Webs based applications are utilized by a lot of people, in some cases the defense level is vulnerable, and that compiles them prone to obtain negotiation. In most of the eventualities, a client has to be recognized previous to any contact is set upped with the backend data. A precipitate client shouldn't be permitted entrée to the scheme without legal credentials. However, a crafted injection query provides illegal entrée to illegal clients. This is regularly accomplished via SQL Injection input. In spite of the occurrence of dissimilar procedures to discover and avoid SQL injection, still there stays a shocking threat into Web based applications. While working on this paper, we studied and did analysis on various manners of SQL Injection vulnerabilities, different types of assaults, and their recognition and avoidance techniques. Flanking we present our assessment of this analysis. We also clarified future research direction for probabilities and possible expansion of challenge proceedings against different web application attacks. Keywords: Vulnerability, SQL injection, XSS, Web based application, WWW, ID

Malware detection based on graph attention networks for intelligent transportation systems

Intelligent Transportation Systems (ITS) aim to make transportation smarter, safer, reliable, and environmentally friendly without detrimentally affecting the service quality. ITS can face security issues due to their complex, dynamic, and non-linear properties. One of the most critical security problems is attacks that damage the infrastructure of the entire ITS. Attackers can inject malware code that triggers dangerous actions such as information theft and unwanted system moves. The main objective of this study is to improve the performance of malware detection models using Graph Attention Networks. To detect malware attacks addressing ITS, a Graph Attention Network (GAN)-based framework is proposed in this study. The inputs to this framework are the Application Programming Interface (API)-call graphs obtained from malware and benign Android apk files. During the graph creation, network metrics and the Node2Vec model are utilized to generate the node features. A GAN-based model is combined with different types of node features during the experiments and the performance is compared against Graph Convolutional Network (GCN). Experimental results demonstrated that the integration of the GAN and Node2Vec models provides the best performance in terms of F-measure and accuracy parameters and, also, the use of an attention mechanism in GAN improves the performance. Furthermore, node features generated with Node2Vec resulted in a 3% increase in classification accuracy compared to the features generated with network metrics. 2021 by the authors. Licensee MDPI, Basel, Switzerland.Scopus2-s2.0-8511720183

Transcend:Detecting Concept Drift in Malware Classification Models

Building machine learning models of malware behavior is widely accepted as a panacea towards effective malware classification. A crucial requirement for building sustainable learning models, though, is to train on a wide variety of malware samples. Unfortunately, malware evolves rapidly and it thus becomes hard—if not impossible—to generalize learning models to reflect future, previously-unseen behaviors. Consequently, most malware classifiers become unsustainable in the long run, becoming rapidly antiquated as malware continues to evolve. In this work, we propose Transcend, a framework to identify aging classification models in vivo during deployment, much before the machine learning model’s performance starts to degrade. This is a significant departure from conventional approaches that retrain aging models retrospectively when poor performance is observed. Our approach uses a statistical comparison of samples seen during deployment with those used to train the model, thereby building metrics for prediction quality. We show how Transcend can be used to identify concept drift based on two separate case studies on Android andWindows malware, raising a red flag before the model starts making consistently poor decisions due to out-of-date training

A functional framework to evade network IDS

Proceeding of: 44th Hawaii International Conference on System Science, Kauai, HI, January 4-7, 2011Signature based Network Intrusion Detection Systems (NIDS) apply a set of rules to identify hostile traffic in network segments. Currently they are so effective detecting known attacks that hackers seek new techniques to go unnoticed. Some of these techniques consist of exploiting network protocols ambiguities. Nowadays NIDS are prepared against most of these evasive techniques, as they are recognized and sorted out. The emergence of new evasive forms may cause NIDS to fail. In this paper we present an innovative functional framework to evade NIDS. Primary, NIDS are modeled accurately by means of Genetic Programming (GP). Then, we show that looking for evasions on models is simpler than directly trying to understand the behavior of NIDS. We present a proof of concept showing how to evade a self-built NIDS regarding two publicly available datasets. Our framework can be used to audit NIDS.This work was partially supported by CDTI, Ministerio de Industria, Turismo y Comercio of Spain in collaboration with Telefonica I+D, Project SEGUR@ CENIT-2007 2004.Publicad

Using Malware Analysis to Evaluate Botnet Resilience

Bos, H.J. [Promotor]Steen, M.R. van [Promotor

Understanding Concurrency Vulnerabilities in Linux Kernel

While there is a large body of work on analyzing concurrency related software bugs and developing techniques for detecting and patching them, little attention has been given to concurrency related security vulnerabilities. The two are different in that not all bugs are vulnerabilities: for a bug to be exploitable, there needs be a way for attackers to trigger its execution and cause damage, e.g., by revealing sensitive data or running malicious code. To fill the gap, we conduct the first empirical study of concurrency vulnerabilities reported in the Linux operating system in the past ten years. We focus on analyzing the confirmed vulnerabilities archived in the Common Vulnerabilities and Exposures (CVE) database, which are then categorized into different groups based on bug types, exploit patterns, and patch strategies adopted by developers. We use code snippets to illustrate individual vulnerability types and patch strategies. We also use statistics to illustrate the entire landscape, including the percentage of each vulnerability type. We hope to shed some light on the problem, e.g., concurrency vulnerabilities continue to pose a serious threat to system security, and it is difficult even for kernel developers to analyze and patch them. Therefore, more efforts are needed to develop tools and techniques for analyzing and patching these vulnerabilities.Comment: It was finished in Oct 201