Covert Android Rootkit Detection: Evaluating Linux Kernel Level Rootkits on the Android Operating System

This research developed kernel level rootkits for Android mobile devices designed to avoid traditional detection methods. The rootkits use system call hooking to insert new handler functions that remove the presence of infection data. The effectiveness of the rootkit is measured with respect to its stealth against detection methods and behavior performance benchmarks. Detection method testing confirms that while detectable with proven tools, system call hooking detection is not built-in or currently available in the Google Play Android App Store. Performance behavior benchmarking showed that system call hooking affects the completion time of the targeted system calls. However, this delay\u27s magnitude may not be noticeable by users. The rootkits implemented targets Android 4.0 on the emulator available from the Android Open Source Project (AOSP) and the Samsung Galaxy Nexus. The rootkits are compiled against both Linux kernel 2.6 and 3.0, respectively. This research shows the Android\u27s Linux kernel is vulnerable to system call hooking and additional measures should be implemented before handling sensitive data with Android

A Study of Rootkit Stealth Techniques and Associated Detection Methods

In today\u27s world of advanced computing power at the fingertips of any user, we must constantly think of computer security. Information is power and this power is had within our computer systems. If we cannot trust the information within our computer systems then we cannot properly wield the power that comes from such information. Rootkits are software programs that are designed to develop and maintain an environment in which malware may hide on a computer system after successful compromise of that computer system. Rootkits cut at the very foundation of the trust that we put in our information and subsequent power. This thesis seeks to understand rootkit hiding techniques, rootkit finding techniques and develops attack trees and defense trees in order to help us identify deficiencies in detection to further increase the trust in our information systems

Rootkit Detection Using A Cross-View Clean Boot Method

In cyberspace, attackers commonly infect computer systems with malware to gain capabilities such as remote access, keylogging, and stealth. Many malware samples include rootkit functionality to hide attacker activities on the target system. After detection, users can remove the rootkit and associated malware from the system with commercial tools. This research describes, implements, and evaluates a clean boot method using two partitions to detect rootkits on a system. One partition is potentially infected with a rootkit while the other is clean. The method obtains directory listings of the potentially infected operating system from each partition and compares the lists to find hidden files. While the clean boot method is similar to other cross-view detection techniques, this method is unique because it uses a clean partition of the same system as the clean operating system, rather than external media. The method produces a 0% false positive rate and a 40.625% true positive rate. In operation, the true positive rate should increase because the experiment produces limitations that prevent many rootkits from working properly. Limitations such as incorrect rootkit setup and rootkits that detect VMware prevent the method from detecting rootkit behavior in this experiment. Vulnerabilities of the method include the assumption that the system restore folder is clean and the assumption that the clean partition is clean. This thesis provides recommendations for more effective rootkit detection

Hardware Virtualization Applied to Rootkit Defense

This research effort examines the idea of applying virtualization hardware to enhance operating system security against rootkits. Rootkits are sets of tools used to hide code and/or functionality from the user and operating system. Rootkits can accomplish this feat through using access to one part of an operating system to change another part that resides at the same privilege level. Hardware assisted virtualization (HAV) provides an opportunity to defeat this tactic through the introduction of a new operating mode. Created to aid operating system virtualization, HAV provides hardware support for managing and saving multiple states of the processor. This hardware support overcomes a problem in pure software virtualization, which is the need to modify guest software to run at a less privileged level. Using HAV, guest software can operate at the pre-HAV most privileged level. This thesis provides a plan to protect data structures targeted by rootkits through unconventional use of HAV technology to secure system resources such as memory. This method of protection will provide true real-time security through OS attack prevention, rather than reaction

Automated Virtual Machine Introspection for Host-Based Intrusion Detection

This thesis examines techniques to automate configuration of an intrusion detection system utilizing hardware-assisted virtualization. These techniques are used to detect the version of a running guest operating system, automatically configure version-specific operating system information needed by the introspection library, and to locate and monitor important operating system data structures. This research simplifies introspection library configuration and is a step toward operating system independent introspection. An operating system detection algorithm and Windows virtual machine system service dispatch table monitor are implemented using the Xen hypervisor and a modified version of the XenAccess library. All detection and monitoring is implemented from the Xen management domain. Results of the operating system detection are used to initialize the XenAccess library. Library initialization time and kernel symbol retrieval are compared to the standard library. The algorithm is evaluated using nine versions of the Windows operating system. The system service dispatch table monitor is evaluated using the Agony and ProAgent rootkits. The automation techniques successfully detect the operating system and system service dispatch table hooks for the nine Windows versions tested. The modified XenAccess library exhibits an average initialization speedup of 1.9. Kernel symbol lookup is 10 times faster, on average. The hook detector is able to detect all hooks used by both rookits

Rootkit Guard (RG) - an architecture for rootkit resistant file-system implementation based on TPM

Recent rootkit-attack mitigation work neglected to address the integrity of the mitigation tool itself. Both detection and prevention arms of current rootkit-attack mitigation solutions can be given credit for the advancement of multiple methodologies for rootkit defense but if the defense system itself is compromised, how is the defense system to be trusted? Another deficiency not addressed is how platform integrity can be preserved without availability of current RIDS or RIPS solutions, which operate only upon the loading of the kernel i.e. without availability of a trusted boot environment. To address these deficiencies, we present our architecture for solving rootkit persistence – Rootkit Guard (RG). RG is a marriage between TrustedGRUB (providing trusted boot), IMA (Integrity Measurement Architecture) (serves as RIDS) and SELinux (serves as RIPS). TPM hardware is utilised to provide total integrity of our platform via storage of the aggregate of the clean snapshot of our platform OS kernel into TPM hardware registers (i.e. the PCR) – of which no software attacks have been demonstrated to date. RG solves rootkit persistence by leveraging on one vital but simple strategy: the mounting of rootkit defense via prevention of the execution of configuration binaries or build initialisation scripts. We adopted the technique of rootkit persistence prevention via thwarting the initialisation of a rootkit’s installation procedure; if the rootkit is successfully installed, proper deployment via thwarting of the rootkit’s configuration is prevented. We had subjected the RG to 8 real world Linux 2.6 rootkits and the RG was successful in solving rootkit persistence in all 8 evaluated rootkits. In terms of performance, the RG introduced a maximum of 11% overhead and an average of 4% overhead, hence permitting deployment in production environments

Effectiveness of Linux rootkit detection tools

Abstract. Rootkits — a type of software that specializes in hiding entities in computer systems while enabling continuous control or access to it — are particularly difficult to detect compared to other kinds of software. Various tools exist for detecting rootkits, utilizing a wide variety of detection techniques and mechanisms. However, the effectiveness of such tools is not well established, especially in contemporary academic research and in the context of the Linux operating system. This study carried out an empirical evaluation of the effectiveness of five tools with capabilities to detect Linux rootkits: OSSEC, AIDE, Rootkit Hunter, Chkrootkit and LKRG. The effectiveness of each tool was tested by injecting 15 publicly available rootkits in individual detection tests in virtual machines running Ubuntu 16.04, executing the detection tool and capturing its results for analysis. A total of 75 detection tests were performed. The results showed that only 37.3% of the detection tests provided any indication of a rootkit infection or suspicious system behaviour, with the rest failing to provide any signs of anomalous behaviour. However, combining the findings of multiple detection tools increased the overall detection rate to 93.3%, as all but a single rootkit were discovered by at least one tool. Variation was observed in the effectiveness of the detection tools, with detection rates ranging from 13.3% to 53.3%. Variation in detection effectiveness was also found between categories of rootkits, as the overall detection rate was 46.7% for user mode rootkits and 31.1% for kernel mode rootkits. Overall, the findings showed that while an individual detection tool‘s effectiveness can be lacking, using a combination of tools considerably increased the likelihood of a successful detection

Autoscopy Jr.: Intrusion Detection for Embedded Control Systems

Securing embedded control systems within the power grid presents a unique challenge: on top of the resource restrictions inherent to these devices, SCADA systems must also accommodate strict timing requirements that are non-negotiable, and their massive scale greatly amplifies costs such as power consumption. These constraints make the conventional approach to host intrusion detection--namely, employing virtualization in some manner--too costly or impractical for embedded control systems within critical infrastructure. Instead, we take an in-kernel approach to system protection, building upon the Autoscopy system developed by Ashwin Ramaswamy that places probes on indirectly-called functions and uses them to monitor its host system for behavior characteristic of control-flow-altering malware, such as rootkits. In this thesis, we attempt to show that such a method would indeed be a viable method of protecting embedded control systems. We first identify several issues with the original prototype, and present a new version of the program (dubbed Autoscopy Jr.) that uses trusted location lists to verify that control is coming from a known, trusted location inside our kernel. Although we encountered additional performance overhead when testing our new design, we developed a kernel profiler that allowed us to identify the probes responsible for this overhead and discard them, leaving us with a final probe list that generated less than 5% overhead on every one of our benchmark tests. Finally, we attempted to run Autoscopy Jr. on two specialized kernels (one with an optimized probing framework, and another with a hardening patch installed), finding that the former did not produce enough performance benefits to preclude using our profiler, and that the latter required a different method of scanning for indirect functions for Autoscopy Jr. to operate. We argue that Autoscopy Jr. is indeed a feasible intrusion detection system for embedded control systems, as it can adapt easily to a variety of system architectures and allows us to intelligently balance security and performance on these critical devices

SecureQEMU: Emulation-based Software Protection Providing Encrypted Code Execution and Page Granularity Code Signing

This research presents an original emulation-based software protection scheme providing protection from reverse code engineering (RCE) and software exploitation using encrypted code execution and page-granularity code signing, respectively. Protection mechanisms execute in trusted emulators while remaining out-of-band of untrusted systems being emulated. This protection scheme is called SecureQEMU and is based on a modified version of Quick Emulator (QEMU) [5]. RCE is a process that uncovers the internal workings of a program. It is used during vulnerability and intellectual property (IP) discovery. To protect from RCE program code may have anti-disassembly, anti-debugging, and obfuscation techniques incorporated. These techniques slow the process of RCE, however, once defeated protected code is still comprehensible. Encryption provides static code protection, but encrypted code must be decrypted before execution. SecureQEMUs\u27 scheme overcomes this limitation by keeping code encrypted during execution. Software exploitation is a process that leverages design and implementation errors to cause unintended behavior which may result in security policy violations. Traditional exploitation protection mechanisms provide a blacklist approach to software protection. Specially crafted exploit payloads bypass these protection mechanisms. SecureQEMU provides a whitelist approach to software protection by executing signed code exclusively. Unsigned malicious code (exploits, backdoors, rootkits, etc.) remain unexecuted, therefore, protecting the system. SecureQEMUs\u27 cache mechanisms increase performance by 0.9% to 1.8% relative to QEMU. Emulation overhead for SecureQEMU varies from 1400% to 2100% with respect to native performance. SecureQEMUs\u27 performance increase is negligible with respect to emulation overhead. Dependent on risk management strategy, SecureQEMU\u27s protection benefits may outweigh emulation overhead