Botnet Detection in Virtual Environments Using NetFlow

For both enterprises and service providers, the exponential growth of cloud and virtual infrastructures brings vast performance and financial benefits but this growth has undoubtedly introduced unforeseen problems in terms of new opportunities for malware and cybercrime to flourish. Botnets could be created entirely within the cloud using virtual resources, for a myriad of purposes including DDoS-as-a-Service. This study has sought to determine whether distributed packet capture utilising mirroring technology or some form of sampling mechanism provides better performance for detecting cybercrime style activities within virtual environments. Recommendations are for a distributed monitoring technique which can provide end-to-end monitoring capabilities while minimising the performance impact on popular adoptions of cloud or virtual infrastructures. Investigations have concentrated on distributed monitoring techniques utilising virtual network switches, looking for a proof of concept demonstrator where sample Command & Control and Peer-to-Peer botnet activities can be detected utilising flow capture technologies such as NetFlow, sFlow or IPFIX. This paper demonstrates how by inserting a monitoring function into a virtual or cloud architecture the capture and analysis of traffic parameters using NetFlow can be used to identify the presence of an HTTP-based Command & Control botnet

On Partitional Clustering of Malware

In this paper we fully describe a novel clustering method for malware, from the transformation of data into a manipulable standardised data matrix, finding the number of clusters until the clustering itself including visualisation of the high-dimensional data. Our clustering method deals well with categorical data and clusters the behavioural data of 17,000 websites, acquired with Capture-HPC, in less than 2 minutesPeer reviewedFinal Accepted Versio

A Survey and Evaluation of Android-Based Malware Evasion Techniques and Detection Frameworks

Android platform security is an active area of research where malware detection techniques continuously evolve to identify novel malware and improve the timely and accurate detection of existing malware. Adversaries are constantly in charge of employing innovative techniques to avoid or prolong malware detection effectively. Past studies have shown that malware detection systems are susceptible to evasion attacks where adversaries can successfully bypass the existing security defenses and deliver the malware to the target system without being detected. The evolution of escape-resistant systems is an open research problem. This paper presents a detailed taxonomy and evaluation of Android-based malware evasion techniques deployed to circumvent malware detection. The study characterizes such evasion techniques into two broad categories, polymorphism and metamorphism, and analyses techniques used for stealth malware detection based on the malware’s unique characteristics. Furthermore, the article also presents a qualitative and systematic comparison of evasion detection frameworks and their detection methodologies for Android-based malware. Finally, the survey discusses open-ended questions and potential future directions for continued research in mobile malware detection

Resurrecting anti-virtualization and anti-debugging: Unhooking your hooks

Dynamic malware analysis involves the debugging of the associated binary files and the monitoring of changes in sandboxed environments. This allows the investigator to manipulate the code execution path and environment to develop an understanding of the malware’s internal workings, aims and modus operandi. However, the latest state of the art malware may incor- porate anti-virtual environment (VM) and anti-debugging countermeasures (i.e. to determine whether the malware is being executed in a VM or us- ing a debugger prior to payload execution). We argue that for the malware to be effective, it will need to support an array of anti-detection and eva- sion mechanisms. In essence, from the malware’s perspective, it needs to adopt a “defence in depth” paradigm to achieve its underlying business logic functionality. Beyond the malicious uses, software vendors to preserve the intellectual property rights of their products often resort to similar methods to deter competitors from gaining intelligence from the binaries or prevent customers from using their products in unauthorised hardware. In this work, we illustrate how Windows architecture impedes the work of debuggers when they analyse with armoured binaries. The debugger and the malware have the same privileges, so the attacker may manipulate theaddress space that the debugger operates and, e.g. bypass detection. We showcase this by presenting a new framework (ANTI), which automates the procedure of integrating anti-debugging and anti-VM in the binary. Specifi- cally, ANTI introduces an anti-hooking method targeting Windows binaries, where hooks applied by state of the art debuggers are removed and injects its code in other processes. This significantly compounds the challenge of binary analysis. Our extensive evaluation also demonstrates that ANTI successfully circumvents detection from state-of-the-art detection methods. Therefore, ANTI illustrates that current tools for dynamic analysis have serious implementation gaps that allow for binaries to bypass them. More alarmingly, ANTI shows how one can use well-known methods to “resurrect” old attacks

Détection d'intrusion sur les objets connectés par analyse comportementale

Alors que la course à l’innovation des fabricants d’objets connectés s’accélère, de plus en plus d’attaques informatiques impliquant de tels objets, ou même les ciblant, sévissent. Ainsi, des campagnes de déni de service distribué comme celle de Mirai ont mis à mal des infrastructures informatiques gigantesques. En outre, le nombre de vulnérabilités découvertes dans l’écosystème des objets connectés ne cesse de grandir. La sécurité physique des utilisateurs d’objet connectés peut également être menacée par l’insécurité de ces plateformes. Dans le même temps, les solutions de sécurité proposées sont souvent spécifiques à un des nombreux protocoles de communication utilisés par les objets intelligents, ou s’appuient sur les caractéristiques matérielles et logicielles d’un type d’objet. De plus, peu de constructeurs permettent des mises à jour des micrologiciels de ces objets, augmentant ainsi les menaces contre les usagers de tels objets. Dans ce contexte, il devient nécessaire de fournir des solutions de sécurité applicables à l’ensemble des objets proposés sur le marché. Cela nécessite de parvenir à collecter de l’information de manière efficace sur l’ensemble de ces systèmes et de réaliser une analyse automatique qui ne dépend pas de l’objet surveillé. Ainsi, différentes solutions ont été proposées, se basant essentiellement sur les informations réseau provenant des objets à surveiller, mais peu d’approches se basent sur le comportement même de l’objet. En conséquence, nous proposons dans ce mémoire une solution de détection d’intrusion se basant sur les anomalies des objets surveillés. Les différents outils que nous avons développés permettent de collecter des informations relatives au comportement des objets surveillés de manière efficace et à différents niveaux, comme le mode usager ou directement dans le noyau du système d’exploitation. Pour ce faire, nous nous appuyons sur des techniques performantes de traçage. L’envoi des traces générées ainsi que leur traitement produira un jeu de données qui sera labellisé automatiquement. Ensuite, différents algorithmes d’apprentissage automatique permettront de détecter les anomalies sur le système de manière automatique et totalement indépendante du type d’objet surveillé. Notre solution introduit très peu de baisse de performance sur les objets connectés surveillés, et montre d’excellents résultats pour détecter divers types d’attaques qui ont été implémentées durant les travaux de recherche. Différents algorithmes ont été étudiés, et les techniques à base d’arbre ont montré des résultats bien plus élevés que des réseaux de neurones profonds. De plus, les outils développés pendant ce projet de recherche permettent d’utiliser les librairies les plus populaires d’apprentissage automatique sur des traces au format CTF, ouvrant ainsi la voie à la prédiction de performance d’un système ou à des analyses de traces plus automatiques et puissantes.----------ABSTRACT: While vendors are creating more and more connected devices, the rate of cyberattacks involving or targeting such devices keeps increasing. For instance, some massive distributed denial of services campaigns such as Mirai used poorly secured devices to shut down popular services on the Internet for many hours or IoT malware like Brickerbot are regularly launched. The insecurity of smart devices create many threats for the users, and vulnerabilities on devices are disclosed every day, while only little vendors let their devices being updated. Meanwhile, proposed security solutions often failed at being compatible with all the devices, because of the numerous protocols used in the Internet of Things or the heterogeneity of firmware used in such devices. This explains why it has become essential to creating a security solution for smart devices that are not specific to the kind of device. The first step to being able to protect a device is being able to detect an intrusion on it. This requires to collect data effectively from the monitored system to launch automated analysis that is not specific to the device. While several solutions matching those criteria have been proposed, most of them studied the network activity of the device, and only little focus on the device behavior. As a consequence, we developed a solution using a device behavior to detect intrusion on it. We obtained very high detection performances with several attacks that have been implemented. Furthermore, we studied various classification algorithm to highlight that tree-based algorithm performed more than the other techniques, including recurrent deep neural network, with the data we collected effectively with tracing techniques. Moreover, we obtained very little overhead on the monitored device because of the architecture we developed. Finally, our tools also enable users to use traces in CTF binary format to feed the most popular Python machine learning libraries thanks to a whole toolchain of processing data

Analysis avoidance techniques of malicious software

Anti Virus (AV) software generally employs signature matching and heuristics to detect the presence of malicious software (malware). The generation of signatures and determination of heuristics is dependent upon an AV analyst having successfully determined the nature of the malware, not only for recognition purposes, but also for the determination of infected files and startup mechanisms that need to be removed as part of the disinfection process. If a specimen of malware has not been previously extensively analyzed, it is unlikely to be detected by AV software. In addition, malware is becoming increasingly profit driven and more likely to incorporate stealth and deception techniques to avoid detection and analysis to remain on infected systems for a myriad of nefarious purposes. Malware extends beyond the commonly thought of virus or worm, to customized malware that has been developed for specific and targeted miscreant purposes. Such customized malware is highly unlikely to be detected by AV software because it will not have been previously analyzed and a signature will not exist. Analysis in such a case will have to be conducted by a digital forensics analyst to determine the functionality of the malware. Malware can employ a plethora of techniques to hinder the analysis process conducted by AV and digital forensics analysts. The purpose of this research has been to answer three research questions directly related to the employment of these techniques as: 1. What techniques can malware use to avoid being analyzed? 2. How can the use of these techniques be detected? 3. How can the use of these techniques be mitigated

REFORM: A framework for malware packer analysis using information theory and statistical methods

Malware (malicious software) is a term used to describe computer viruses, Trojan horses, and other pieces of software that are used to attack computer systems. The increasing outbreak of malware in recent years poses a serious security threat to computer networks. Malware writers often obfuscate malware to hinder malware scanners from malicious code detection, i.e., to hide the fact that the software is actually malicious. Packing is the most common obfuscation method used by malware writers. Recently, there has been a dramatic increase in the number of new packers and variants of existing ones. Moreover, packers are employing increasingly sophisticated anti-unpacker tricks and obfuscation methods. Identifying a packer and obtaining a sample of unpacked malware are important to AV (Anti-virus) researchers who work on updating antivirus software to defend against malware, so that they can perform in-depth analysis. However, packer analysis is a technically intense research task, requiring the AV experts' deep knowledge of hardware, operating systems, compilers and programming languages. The significant growth of packers, in both number and complexity, prevents AV researchers from carrying out their daily AV research work efficiently and effectively. This PhD project has investigated the common features of packers and presented a novel, fast yet effective packer analysis framework called REFORM (Reverse Engineering For Obfuscation ReMoval). The system applies various technologies including reverse engineering, compression algorithms and statistical methods to de-obfuscate packers. REFORM is comprised of three major components that solve the problem of automatic packer analysis at three important stages of the packer analysis life cycle, namely packer detection, packer identification and unpacking, respectively: (1) It incorporates a novel randomness test that preserves local detail in the packer. This makes it easy for an AV researcher to distinguish areas of compressed/encrypted data from other code and data. (2) Using the above randomness test, each packer is seen to exhibit a unique pattern in its randomness distribution. The REFORM framework therefore provides an extremely effective packer classification model based on a set of randomness measurements generated from a packed file. Various statistical classifiers have also been integrated in REFORM to achieve even better classification performance. (3) REFORM enables an efficient generic unpacking strategy which uses an ordered address execution histogram to capture the memory after the unpacking loop has executed. We demonstrate REFORM 's capability on speeding up packer detection, identification and unpacking procedures. Such an automatic system is shown in the thesis to be essential to keeping up with the accelerating growth in packed malware

Techniques for the reverse engineering of banking malware

Malware attacks are a significant and frequently reported problem, adversely affecting the productivity of organisations and governments worldwide. The well-documented consequences of malware attacks include financial loss, data loss, reputation damage, infrastructure damage, theft of intellectual property, compromise of commercial negotiations, and national security risks. Mitiga-tion activities involve a significant amount of manual analysis. Therefore, there is a need for automated techniques for malware analysis to identify malicious behaviours. Research into automated techniques for malware analysis covers a wide range of activities. This thesis consists of a series of studies: an anal-ysis of banking malware families and their common behaviours, an emulated command and control environment for dynamic malware analysis, a technique to identify similar malware functions, and a technique for the detection of ransomware. An analysis of the nature of banking malware, its major malware families, behaviours, variants, and inter-relationships are provided in this thesis. In doing this, this research takes a broad view of malware analysis, starting with the implementation of the malicious behaviours through to detailed analysis using machine learning. The broad approach taken in this thesis differs from some other studies that approach malware research in a more abstract sense. A disadvantage of approaching malware research without domain knowledge, is that important methodology questions may not be considered. Large datasets of historical malware samples are available for countermea-sures research. However, due to the age of these samples, the original malware infrastructure is no longer available, often restricting malware operations to initialisation functions only. To address this absence, an emulated command and control environment is provided. This emulated environment provides full control of the malware, enabling the capabilities of the original in-the-wild operation, while enabling feature extraction for research purposes. A major focus of this thesis has been the development of a machine learn-ing function similarity method with a novel feature encoding that increases feature strength. This research develops techniques to demonstrate that the machine learning model trained on similarity features from one program can find similar functions in another, unrelated program. This finding can lead to the development of generic similar function classifiers that can be packaged and distributed in reverse engineering tools such as IDA Pro and Ghidra. Further, this research examines the use of API call features for the identi-fication of ransomware and shows that a failure to consider malware analysis domain knowledge can lead to weaknesses in experimental design. In this case, we show that existing research has difficulty in discriminating between ransomware and benign cryptographic software. This thesis by publication, has developed techniques to advance the disci-pline of malware reverse engineering, in order to minimize harm due to cyber-attacks on critical infrastructure, government institutions, and industry.Doctor of Philosoph