Resilient and Scalable Android Malware Fingerprinting and Detection

Malicious software (Malware) proliferation reaches hundreds of thousands daily. The manual analysis of such a large volume of malware is daunting and time-consuming. The diversity of targeted systems in terms of architecture and platforms compounds the challenges of Android malware detection and malware in general. This highlights the need to design and implement new scalable and robust methods, techniques, and tools to detect Android malware. In this thesis, we develop a malware fingerprinting framework to cover accurate Android malware detection and family attribution. In this context, we emphasize the following: (i) the scalability over a large malware corpus; (ii) the resiliency to common obfuscation techniques; (iii) the portability over different platforms and architectures. In the context of bulk and offline detection on the laboratory/vendor level: First, we propose an approximate fingerprinting technique for Android packaging that captures the underlying static structure of the Android apps. We also propose a malware clustering framework on top of this fingerprinting technique to perform unsupervised malware detection and grouping by building and partitioning a similarity network of malicious apps. Second, we propose an approximate fingerprinting technique for Android malware's behavior reports generated using dynamic analyses leveraging natural language processing techniques. Based on this fingerprinting technique, we propose a portable malware detection and family threat attribution framework employing supervised machine learning techniques. Third, we design an automatic framework to produce intelligence about the underlying malicious cyber-infrastructures of Android malware. We leverage graph analysis techniques to generate relevant, actionable, and granular intelligence that can be used to identify the threat effects induced by malicious Internet activity associated to Android malicious apps. In the context of the single app and online detection on the mobile device level, we further propose the following: Fourth, we design a portable and effective Android malware detection system that is suitable for deployment on mobile and resource constrained devices, using machine learning classification on raw method call sequences. Fifth, we elaborate a framework for Android malware detection that is resilient to common code obfuscation techniques and adaptive to operating systems and malware change overtime, using natural language processing and deep learning techniques. We also evaluate the portability of the proposed techniques and methods beyond Android platform malware, as follows: Sixth, we leverage the previously elaborated techniques to build a framework for cross-platform ransomware fingerprinting relying on raw hybrid features in conjunction with advanced deep learning techniques

Design and Analysis of Mobile Operating System Security Architecture using Formal Methods

The Android operating system (OS) is now used in the majority of mobile devices. Hence, Android security is an important issue to handle. In this work, we tackle the problem using two separate approaches: directly modifying Android OS and developed a framework to provide a guarantee of non-interference. Firstly, we present a design and an implementation of a security policy specifi- cation language based on metric linear-time temporal logic (MTL) to specify timing- dependent security policies. The design of the language is driven by the problem of runtime monitoring of applications in mobile devices. A main case of the study is the privilege escalation attack in the Android OS, where an unprivileged app gains ac- cess to privileged resource or functionalities through indirect flow. To capture these attacks, we extend MTL with recursive definitions to express call chains between apps. We then show how our language design can be used to specify policies to detect privilege escalation under various fine-grained constraints. We present a new algorithm for monitoring safety policies written in our specification language. The monitor does not need to store the entire history of events generated by the apps. We modified the Android OS kernel to allow us to insert our generated monitors mod- ularly. We have tested the modified OS (LogicDroid) on an actual device, and show that it is effective in detecting policy violations. Furthermore, LogicDroid is able to prevent a previously unknown exploit to breach Android security which allows an unprivileged application to access certain critical and privileged functionalities of an Android phone, such as making phone calls, terminating phone calls, and sending SMS, without having to ask any permissions to do so. Subsequently, we provided a framework to ensure non-interference properties of DEX bytecode. Each application in Android runs in an instance of the Dalvik virtual machine, which is a register-based virtual machine (VM). Most applications for Android are developed using Java, compiled to Java bytecode and further into DEX bytecode. Following a methodology that has been developed for Java byte- code certification by Barthe et al., we developed a type-based method for certifying non-interference property of a DEX program. To this end, we develop a formal oper- ational semantics of the Dalvik VM, a type system for DEX bytecode, and prove the soundness of the type system with respect to a notion of non-interference. We have also formalized the proof of a subset of DEX in Coq for an additional guarantee that our proof is correct. We then study the translation process from Java bytecode to DEX bytecode, as implemented in the dx tool in the Android SDK. We show that an abstracted version of the translation from Java bytecode to DEX bytecode preserves the non-interference property. More precisely, we show that if the Java bytecode is typable in Barthe et al.’s type system, then its translation is typable in our type system. This result opens up the possibility to leverage existing bytecode verifiers for Java to certify non-interference properties of Android bytecode