Kernel Rootkits Detection Method by Monitoring Branches Using Hardware Features

An operating system is an essential piece of software that manages hardware and software resources. Thus, attacks on an operating system kernel using kernel rootkits pose a particularly serious threat. Detecting an attack is difficult when the operating system kernel is infected with a kernel rootkit. For this reason, handling an attack will be delayed causing an increase in the amount of damage done to a computer system. In this paper, we propose Kernel Rootkits Guard (KRGuard), which is a new method to detect kernel rootkits that monitors branch records in the kernel space. Since many kernel rootkits make branches that differ from the usual branches in the kernel space, KRGuard can detect these differences by using the hardware features of commodity processors. Our evaluation shows that KRGuard can detect kernel rootkits that involve new branches in the system call handler processing with small overhead

KRGuard: Kernel Rootkits Detection Method by Monitoring Branches Using Hardware Features

Attacks on an operating system kernel using kernel rootkits pose a particularly serious threat. Detecting an attack is difficult when the operating system kernel is infected with a kernel rootkit. For this reason, handling an attack will be delayed causing an increase in the amount of damage done to a computer system. In this paper, we discuss KRGuard (Kernel Rootkits Guard), which is a new method to detect kernel rootkits that monitors branch records in the kernel space. Since many kernel rootkits make branches that differ from the usual branches in the kernel space, KRGuard can detect these differences by using hardware features of commodity processors. Our evaluation shows that KRGuard can detect kernel rootkits with small overhead

Effectiveness of Linux rootkit detection tools

Abstract. Rootkits — a type of software that specializes in hiding entities in computer systems while enabling continuous control or access to it — are particularly difficult to detect compared to other kinds of software. Various tools exist for detecting rootkits, utilizing a wide variety of detection techniques and mechanisms. However, the effectiveness of such tools is not well established, especially in contemporary academic research and in the context of the Linux operating system. This study carried out an empirical evaluation of the effectiveness of five tools with capabilities to detect Linux rootkits: OSSEC, AIDE, Rootkit Hunter, Chkrootkit and LKRG. The effectiveness of each tool was tested by injecting 15 publicly available rootkits in individual detection tests in virtual machines running Ubuntu 16.04, executing the detection tool and capturing its results for analysis. A total of 75 detection tests were performed. The results showed that only 37.3% of the detection tests provided any indication of a rootkit infection or suspicious system behaviour, with the rest failing to provide any signs of anomalous behaviour. However, combining the findings of multiple detection tools increased the overall detection rate to 93.3%, as all but a single rootkit were discovered by at least one tool. Variation was observed in the effectiveness of the detection tools, with detection rates ranging from 13.3% to 53.3%. Variation in detection effectiveness was also found between categories of rootkits, as the overall detection rate was 46.7% for user mode rootkits and 31.1% for kernel mode rootkits. Overall, the findings showed that while an individual detection tool‘s effectiveness can be lacking, using a combination of tools considerably increased the likelihood of a successful detection

Detecting kernel rootkits

Kernel rootkits are a special category of malware that are deployed directly in the kernel and hence have unmitigated reign over the functionalities of the kernel itself. We seek to detect such rootkits that are deployed in the real world by first observing how the majority of kernel rootkits operate. To this end, comparable to how rootkits function in the real world, we write our own kernel rootkit that manipulates the network driver, thus giving us control over all packets sent into the network. We then implement a mechanism to thwart the attacks of such rootkits by noticing that a large number of the rootkits deployed today rely heavily on the redirection of function pointers within the kernel. By overwriting the desired function pointer to its own function, a rootkit can perform a proverbial man-in-the-middle attack. Our goal is not just the detection of kernel rootkits, but also to levy as little an impact on system performance as possible. Hence our technique is to leverage existing kernel functionalities (in the case of Linux) such as kprobes to identify potential attack scenarios from within the sytem rather than from outside it (such as a VMM). We hope to introduce real-world security in devices where performance and resource constraints are tantamount to security considerations

Análise de malware com suporte de hardware

Orientadores: Paulo Lício de Geus, André Ricardo Abed GrégioDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: O mundo atual é impulsionado pelo uso de sistemas computacionais, estando estes pre- sentes em todos aspectos da vida cotidiana. Portanto, o correto funcionamento destes é essencial para se assegurar a manutenção das possibilidades trazidas pelos desenvolvi- mentos tecnológicos. Contudo, garantir o correto funcionamento destes não é uma tarefa fácil, dado que indivíduos mal-intencionados tentam constantemente subvertê-los visando benefíciar a si próprios ou a terceiros. Os tipos mais comuns de subversão são os ataques por códigos maliciosos (malware), capazes de dar a um atacante controle total sobre uma máquina. O combate à ameaça trazida por malware baseia-se na análise dos artefatos coletados de forma a permitir resposta aos incidentes ocorridos e o desenvolvimento de contramedidas futuras. No entanto, atacantes têm se especializado em burlar sistemas de análise e assim manter suas operações ativas. Para este propósito, faz-se uso de uma série de técnicas denominadas de "anti-análise", capazes de impedir a inspeção direta dos códigos maliciosos. Dentre essas técnicas, destaca-se a evasão do processo de análise, na qual são empregadas exemplares capazes de detectar a presença de um sistema de análise para então esconder seu comportamento malicioso. Exemplares evasivos têm sido cada vez mais utilizados em ataques e seu impacto sobre a segurança de sistemas é considerá- vel, dado que análises antes feitas de forma automática passaram a exigir a supervisão de analistas humanos em busca de sinais de evasão, aumentando assim o custo de se manter um sistema protegido. As formas mais comuns de detecção de um ambiente de análise se dão através da detecção de: (i) código injetado, usado pelo analista para inspecionar a aplicação; (ii) máquinas virtuais, usadas em ambientes de análise por questões de escala; (iii) efeitos colaterais de execução, geralmente causados por emuladores, também usados por analistas. Para lidar com malware evasivo, analistas tem se valido de técnicas ditas transparentes, isto é, que não requerem injeção de código nem causam efeitos colaterais de execução. Um modo de se obter transparência em um processo de análise é contar com suporte do hardware. Desta forma, este trabalho versa sobre a aplicação do suporte de hardware para fins de análise de ameaças evasivas. No decorrer deste texto, apresenta-se uma avaliação das tecnologias existentes de suporte de hardware, dentre as quais máqui- nas virtuais de hardware, suporte de BIOS e monitores de performance. A avaliação crítica de tais tecnologias oferece uma base de comparação entre diferentes casos de uso. Além disso, são enumeradas lacunas de desenvolvimento existentes atualmente. Mais que isso, uma destas lacunas é preenchida neste trabalho pela proposição da expansão do uso dos monitores de performance para fins de monitoração de malware. Mais especificamente, é proposto o uso do monitor BTS para fins de construção de um tracer e um debugger. O framework proposto e desenvolvido neste trabalho é capaz, ainda, de lidar com ataques do tipo ROP, um dos mais utilizados atualmente para exploração de vulnerabilidades. A avaliação da solução demonstra que não há a introdução de efeitos colaterais, o que per- mite análises de forma transparente. Beneficiando-se desta característica, demonstramos a análise de aplicações protegidas e a identificação de técnicas de evasãoAbstract: Today¿s world is driven by the usage of computer systems, which are present in all aspects of everyday life. Therefore, the correct working of these systems is essential to ensure the maintenance of the possibilities brought about by technological developments. However, ensuring the correct working of such systems is not an easy task, as many people attempt to subvert systems working for their own benefit. The most common kind of subversion against computer systems are malware attacks, which can make an attacker to gain com- plete machine control. The fight against this kind of threat is based on analysis procedures of the collected malicious artifacts, allowing the incident response and the development of future countermeasures. However, attackers have specialized in circumventing analysis systems and thus keeping their operations active. For this purpose, they employ a series of techniques called anti-analysis, able to prevent the inspection of their malicious codes. Among these techniques, I highlight the analysis procedure evasion, that is, the usage of samples able to detect the presence of an analysis solution and then hide their malicious behavior. Evasive examples have become popular, and their impact on systems security is considerable, since automatic analysis now requires human supervision in order to find evasion signs, which significantly raises the cost of maintaining a protected system. The most common ways for detecting an analysis environment are: i) Injected code detec- tion, since injection is used by analysts to inspect applications on their way; ii) Virtual machine detection, since they are used in analysis environments due to scalability issues; iii) Execution side effects detection, usually caused by emulators, also used by analysts. To handle evasive malware, analysts have relied on the so-called transparent techniques, that is, those which do not require code injection nor cause execution side effects. A way to achieve transparency in an analysis process is to rely on hardware support. In this way, this work covers the application of the hardware support for the evasive threats analysis purpose. In the course of this text, I present an assessment of existing hardware support technologies, including hardware virtual machines, BIOS support, performance monitors and PCI cards. My critical evaluation of such technologies provides basis for comparing different usage cases. In addition, I pinpoint development gaps that currently exists. More than that, I fill one of these gaps by proposing to expand the usage of performance monitors for malware monitoring purposes. More specifically, I propose the usage of the BTS monitor for the purpose of developing a tracer and a debugger. The proposed framework is also able of dealing with ROP attacks, one of the most common used technique for remote vulnerability exploitation. The framework evaluation shows no side-effect is introduced, thus allowing transparent analysis. Making use of this capability, I demonstrate how protected applications can be inspected and how evasion techniques can be identifiedMestradoCiência da ComputaçãoMestre em Ciência da ComputaçãoCAPE

Automated Virtual Machine Introspection for Host-Based Intrusion Detection

This thesis examines techniques to automate configuration of an intrusion detection system utilizing hardware-assisted virtualization. These techniques are used to detect the version of a running guest operating system, automatically configure version-specific operating system information needed by the introspection library, and to locate and monitor important operating system data structures. This research simplifies introspection library configuration and is a step toward operating system independent introspection. An operating system detection algorithm and Windows virtual machine system service dispatch table monitor are implemented using the Xen hypervisor and a modified version of the XenAccess library. All detection and monitoring is implemented from the Xen management domain. Results of the operating system detection are used to initialize the XenAccess library. Library initialization time and kernel symbol retrieval are compared to the standard library. The algorithm is evaluated using nine versions of the Windows operating system. The system service dispatch table monitor is evaluated using the Agony and ProAgent rootkits. The automation techniques successfully detect the operating system and system service dispatch table hooks for the nine Windows versions tested. The modified XenAccess library exhibits an average initialization speedup of 1.9. Kernel symbol lookup is 10 times faster, on average. The hook detector is able to detect all hooks used by both rookits

Survey of Return-Oriented Programming Defense Mechanisms

A prominent software security violation-buffer overflow attack has taken various forms and poses serious threats until today. One such vulnerability is return-oriented programming attack. An return-oriented programming attack circumvents the dynamic execution prevention, which is employed in modern operating systems to prevent execution of data segments, and attempts to execute unintended instructions by overwriting the stack exploiting the buffer overflow vulnerability. Numerous defense mechanisms have been proposed in the past few years to mitigate/prevent the attack – compile time methods that add checking logic to the program code before compilation, dynamic methods that monitor the control-flow integrity during execution and randomization methods that aim at randomizing instruction locations. This paper discusses (i) these different static, dynamic, and randomization techniques proposed recently and (ii) compares the techniques based on their effectiveness and performances

Acceleration of Statistical Detection of Zero-day Malware in the Memory Dump Using CUDA-enabled GPU Hardware

This paper focuses on the anticipatory enhancement of methods of detecting stealth software. Cyber security detection tools are insufficiently powerful to reveal the most recent cyber-attacks which use malware. In this paper, we will present first an idea of the highest stealth malware, as this is the most complicated scenario for detection because it combines both existing anti-forensic techniques together with their potential improvements. Second, we present new detection methods, which are resilient to this hidden prototype. To help solve this detection challenge, we have analyzed Windows memory content using a new method of Shannon Entropy calculation; methods of digital photogrammetry; the Zipf Mandelbrot law, as well as by disassembling the memory content and analyzing the output. Finally, we present an idea and architecture of the software tool, which uses CUDA enabled GPU hardware to speed-up memory forensics. All three ideas are currently a work in progress

Acceleration of Statistical Detection of Zero-day Malware in the Memory Dump Using CUDA-enabled GPU Hardware

This paper focuses on the anticipatory enhancement of methods of detecting stealth software. Cyber security detection tools are insufficiently powerful to reveal the most recent cyber-attacks which use malware. In this paper, we will present first an idea of the highest stealth malware, as this is the most complicated scenario for detection because it combines both existing anti-forensic techniques together with their potential improvements. Second, we will present new detection methods which are resilient to this hidden prototype. To help solve this detection challenge, we have analyzed Windows’ memory content using a new method of Shannon Entropy calculation; methods of digital photogrammetry; the Zipf–Mandelbrot law, as well as by disassembling the memory content and analyzing the output. Finally, we present an idea and architecture of the software tool, which uses CUDA-enabled GPU hardware, to speed-up memory forensics. All three ideas are currently a work in progress. Keywords: rootkit detection, anti-forensics, memory analysis, scattered fragments, anticipatory enhancement, CUDA