Methods for detecting kernel rootkits.

Rootkits are stealthy, malicious software that allow an attacker to gain and maintain control of a system, attack other systems, destroy evidence, and decrease the chance of detection. Existing detection methods typically rely on a priori knowledge and operate by either (a) saving the system state before infection and comparing this information post infection, or (b) installing a detection program before infection. This dissertation focuses on detection using reduced a priori knowledge in the form of general knowledge of the statistical properties of broad classes of operating system/architecture pairs. Four new approaches to rootkit detection were implemented and evaluated. A general distribution model is employed against kernel rootkits utilizing the system call table modification attack. Using approaches from the field of outlier detection, this approach successfully detected four different rootkits, with no false positives. Scalability is, however, an issue with this approach. A second, normality-based approach was investigated for use against rootkits infecting systems via the system call table modification attack. This approach was partially successful, but did generate false positives in 0.35% of cases. The general distribution model was then applied to rootkits infecting systems via the system call target modification attack. This dataset is dramatically larger, including disassembled memory addresses from the entire kernel. Finally, a modified version of the normality based approach proved effective in detecting kernel rootkits infecting the kernel via the system call target modification attack. This approach capitalizes on the discovery that system calls are loaded into memory sequentially, with the higher level calls, which are more likely to be infected by kernel rootkits loaded first, and the lower level calls loaded later. In the single case evaluated, the enyelkm rootkit, neither false positives nor false positives were indicated. As a final evaluation, these techniques were applied to the Microsoft Windows operating systems. The Windows equivalent of the system call table, the system service descriptor table (SSDT), appears to be almost perfectly normally distributed. A Windows rootkit employing the system call table modification attack was detected using the general distribution and \u27assumption of normality\u27 models

Rootkit Detection Using A Cross-View Clean Boot Method

In cyberspace, attackers commonly infect computer systems with malware to gain capabilities such as remote access, keylogging, and stealth. Many malware samples include rootkit functionality to hide attacker activities on the target system. After detection, users can remove the rootkit and associated malware from the system with commercial tools. This research describes, implements, and evaluates a clean boot method using two partitions to detect rootkits on a system. One partition is potentially infected with a rootkit while the other is clean. The method obtains directory listings of the potentially infected operating system from each partition and compares the lists to find hidden files. While the clean boot method is similar to other cross-view detection techniques, this method is unique because it uses a clean partition of the same system as the clean operating system, rather than external media. The method produces a 0% false positive rate and a 40.625% true positive rate. In operation, the true positive rate should increase because the experiment produces limitations that prevent many rootkits from working properly. Limitations such as incorrect rootkit setup and rootkits that detect VMware prevent the method from detecting rootkit behavior in this experiment. Vulnerabilities of the method include the assumption that the system restore folder is clean and the assumption that the clean partition is clean. This thesis provides recommendations for more effective rootkit detection

Effectiveness of Linux rootkit detection tools

Abstract. Rootkits — a type of software that specializes in hiding entities in computer systems while enabling continuous control or access to it — are particularly difficult to detect compared to other kinds of software. Various tools exist for detecting rootkits, utilizing a wide variety of detection techniques and mechanisms. However, the effectiveness of such tools is not well established, especially in contemporary academic research and in the context of the Linux operating system. This study carried out an empirical evaluation of the effectiveness of five tools with capabilities to detect Linux rootkits: OSSEC, AIDE, Rootkit Hunter, Chkrootkit and LKRG. The effectiveness of each tool was tested by injecting 15 publicly available rootkits in individual detection tests in virtual machines running Ubuntu 16.04, executing the detection tool and capturing its results for analysis. A total of 75 detection tests were performed. The results showed that only 37.3% of the detection tests provided any indication of a rootkit infection or suspicious system behaviour, with the rest failing to provide any signs of anomalous behaviour. However, combining the findings of multiple detection tools increased the overall detection rate to 93.3%, as all but a single rootkit were discovered by at least one tool. Variation was observed in the effectiveness of the detection tools, with detection rates ranging from 13.3% to 53.3%. Variation in detection effectiveness was also found between categories of rootkits, as the overall detection rate was 46.7% for user mode rootkits and 31.1% for kernel mode rootkits. Overall, the findings showed that while an individual detection tool‘s effectiveness can be lacking, using a combination of tools considerably increased the likelihood of a successful detection

A Study of Rootkit Stealth Techniques and Associated Detection Methods

In today\u27s world of advanced computing power at the fingertips of any user, we must constantly think of computer security. Information is power and this power is had within our computer systems. If we cannot trust the information within our computer systems then we cannot properly wield the power that comes from such information. Rootkits are software programs that are designed to develop and maintain an environment in which malware may hide on a computer system after successful compromise of that computer system. Rootkits cut at the very foundation of the trust that we put in our information and subsequent power. This thesis seeks to understand rootkit hiding techniques, rootkit finding techniques and develops attack trees and defense trees in order to help us identify deficiencies in detection to further increase the trust in our information systems

SMM rootkit: a new breed of OS independent malware

The emergence of hardware virtualization technology has led to the development of OS independent malware such as the virtual machine-based rootkits (VMBRs). In this paper, we draw attention to a different but related threat that exists on many commodity systems in operation today: The system management Mode based rootkit (SMBR). System Management mode (SMM) is a relatively obscure mode on Intel processors used for low-level hardware control. It has its own private memory space and execution environment which is generally invisible to code running outside (e.g., the Operating System). Furthermore, SMM code is completely non-preemptible, lacks any concept of privilege level, and is immune to memory protection mechanisms. These features make it a potentially attractive home for stealthy rootkits used for high-profile targeted attacks. In this paper, we present our development of a proof of concept SMM rootkit. In it, we explore the potential of system management mode for malicious use by implementing a chipset level keylogger and a network backdoor capable of directly interacting with the network card to send logged keystrokes to a remote machine via UDP and receive remote command packets stealthily. By modifying and reflashing the BIOS, the SMM rootkit can install itself on a computer even if the computer has originally locked its SMM. The rootkit hides its memory footprint and requires no changes to the existing operating system. It is compared and contrasted with VMBRs. Finally, techniques to defend against these threats are explored. By taking an offensive perspective we hope to help security researchers better understand the depth and scope of the problems posed by an emerging class of OS independent malware