HyBIS: Windows Guest Protection through Advanced Memory Introspection

Effectively protecting the Windows OS is a challenging task, since most implementation details are not publicly known. Windows has always been the main target of malwares that have exploited numerous bugs and vulnerabilities. Recent trusted boot and additional integrity checks have rendered the Windows OS less vulnerable to kernel-level rootkits. Nevertheless, guest Windows Virtual Machines are becoming an increasingly interesting attack target. In this work we introduce and analyze a novel Hypervisor-Based Introspection System (HyBIS) we developed for protecting Windows OSes from malware and rootkits. The HyBIS architecture is motivated and detailed, while targeted experimental results show its effectiveness. Comparison with related work highlights main HyBIS advantages such as: effective semantic introspection, support for 64-bit architectures and for latest Windows (8.x and 10), advanced malware disabling capabilities. We believe the research effort reported here will pave the way to further advances in the security of Windows OSes

Effectiveness of Linux rootkit detection tools

Abstract. Rootkits — a type of software that specializes in hiding entities in computer systems while enabling continuous control or access to it — are particularly difficult to detect compared to other kinds of software. Various tools exist for detecting rootkits, utilizing a wide variety of detection techniques and mechanisms. However, the effectiveness of such tools is not well established, especially in contemporary academic research and in the context of the Linux operating system. This study carried out an empirical evaluation of the effectiveness of five tools with capabilities to detect Linux rootkits: OSSEC, AIDE, Rootkit Hunter, Chkrootkit and LKRG. The effectiveness of each tool was tested by injecting 15 publicly available rootkits in individual detection tests in virtual machines running Ubuntu 16.04, executing the detection tool and capturing its results for analysis. A total of 75 detection tests were performed. The results showed that only 37.3% of the detection tests provided any indication of a rootkit infection or suspicious system behaviour, with the rest failing to provide any signs of anomalous behaviour. However, combining the findings of multiple detection tools increased the overall detection rate to 93.3%, as all but a single rootkit were discovered by at least one tool. Variation was observed in the effectiveness of the detection tools, with detection rates ranging from 13.3% to 53.3%. Variation in detection effectiveness was also found between categories of rootkits, as the overall detection rate was 46.7% for user mode rootkits and 31.1% for kernel mode rootkits. Overall, the findings showed that while an individual detection tool‘s effectiveness can be lacking, using a combination of tools considerably increased the likelihood of a successful detection

A Study of Rootkit Stealth Techniques and Associated Detection Methods

In today\u27s world of advanced computing power at the fingertips of any user, we must constantly think of computer security. Information is power and this power is had within our computer systems. If we cannot trust the information within our computer systems then we cannot properly wield the power that comes from such information. Rootkits are software programs that are designed to develop and maintain an environment in which malware may hide on a computer system after successful compromise of that computer system. Rootkits cut at the very foundation of the trust that we put in our information and subsequent power. This thesis seeks to understand rootkit hiding techniques, rootkit finding techniques and develops attack trees and defense trees in order to help us identify deficiencies in detection to further increase the trust in our information systems

Rootkit Detection Using A Cross-View Clean Boot Method

In cyberspace, attackers commonly infect computer systems with malware to gain capabilities such as remote access, keylogging, and stealth. Many malware samples include rootkit functionality to hide attacker activities on the target system. After detection, users can remove the rootkit and associated malware from the system with commercial tools. This research describes, implements, and evaluates a clean boot method using two partitions to detect rootkits on a system. One partition is potentially infected with a rootkit while the other is clean. The method obtains directory listings of the potentially infected operating system from each partition and compares the lists to find hidden files. While the clean boot method is similar to other cross-view detection techniques, this method is unique because it uses a clean partition of the same system as the clean operating system, rather than external media. The method produces a 0% false positive rate and a 40.625% true positive rate. In operation, the true positive rate should increase because the experiment produces limitations that prevent many rootkits from working properly. Limitations such as incorrect rootkit setup and rootkits that detect VMware prevent the method from detecting rootkit behavior in this experiment. Vulnerabilities of the method include the assumption that the system restore folder is clean and the assumption that the clean partition is clean. This thesis provides recommendations for more effective rootkit detection

Methods for detecting kernel rootkits.

Rootkits are stealthy, malicious software that allow an attacker to gain and maintain control of a system, attack other systems, destroy evidence, and decrease the chance of detection. Existing detection methods typically rely on a priori knowledge and operate by either (a) saving the system state before infection and comparing this information post infection, or (b) installing a detection program before infection. This dissertation focuses on detection using reduced a priori knowledge in the form of general knowledge of the statistical properties of broad classes of operating system/architecture pairs. Four new approaches to rootkit detection were implemented and evaluated. A general distribution model is employed against kernel rootkits utilizing the system call table modification attack. Using approaches from the field of outlier detection, this approach successfully detected four different rootkits, with no false positives. Scalability is, however, an issue with this approach. A second, normality-based approach was investigated for use against rootkits infecting systems via the system call table modification attack. This approach was partially successful, but did generate false positives in 0.35% of cases. The general distribution model was then applied to rootkits infecting systems via the system call target modification attack. This dataset is dramatically larger, including disassembled memory addresses from the entire kernel. Finally, a modified version of the normality based approach proved effective in detecting kernel rootkits infecting the kernel via the system call target modification attack. This approach capitalizes on the discovery that system calls are loaded into memory sequentially, with the higher level calls, which are more likely to be infected by kernel rootkits loaded first, and the lower level calls loaded later. In the single case evaluated, the enyelkm rootkit, neither false positives nor false positives were indicated. As a final evaluation, these techniques were applied to the Microsoft Windows operating systems. The Windows equivalent of the system call table, the system service descriptor table (SSDT), appears to be almost perfectly normally distributed. A Windows rootkit employing the system call table modification attack was detected using the general distribution and \u27assumption of normality\u27 models

Covert Android Rootkit Detection: Evaluating Linux Kernel Level Rootkits on the Android Operating System

This research developed kernel level rootkits for Android mobile devices designed to avoid traditional detection methods. The rootkits use system call hooking to insert new handler functions that remove the presence of infection data. The effectiveness of the rootkit is measured with respect to its stealth against detection methods and behavior performance benchmarks. Detection method testing confirms that while detectable with proven tools, system call hooking detection is not built-in or currently available in the Google Play Android App Store. Performance behavior benchmarking showed that system call hooking affects the completion time of the targeted system calls. However, this delay\u27s magnitude may not be noticeable by users. The rootkits implemented targets Android 4.0 on the emulator available from the Android Open Source Project (AOSP) and the Samsung Galaxy Nexus. The rootkits are compiled against both Linux kernel 2.6 and 3.0, respectively. This research shows the Android\u27s Linux kernel is vulnerable to system call hooking and additional measures should be implemented before handling sensitive data with Android

Rootkit detection on embedded IoT devices

IoT systems are subject to cyber attacks, including infecting embedded IoT devices with rootkits. Rootkits are malicious software that typically run with elevated privileges, which makes their detection challenging. In this paper, we address this challenge: we propose a rootkit detection approach for embedded IoT devices that takes advantage of a trusted execution environment (TEE), which is often supported on popular IoT platforms, such as ARM based embedded boards. The TEE provides an isolated environment for our rootkit detection algorithms, and prevents the rootkit from interfering with their execution even if the rootkit has root privileges on the untrusted part of the IoT device. Our rootkit detection algorithms identify modifications made by the rootkit to the code of the operating system kernel, to system programs, and to data influencing the control flow (e.g., hooking system calls), as well as inconsistencies created by the rootkit in certain kernel data structures (e.g., those responsible to handle process related information). We also propose algorithms to detect rootkit components in the persistent storage of the device. Besides describing our approach and algorithms in details, we also report on a prototype implementation and on the evaluation of our design and implementation, which is based on testing our prototype with rootkits that we developed for this purpose