Malware detection and analysis via layered annotative execution

Malicious software (i.e., malware) has become a severe threat to interconnected computer systems for decades and has caused billions of dollars damages each year. A large volume of new malware samples are discovered daily. Even worse, malware is rapidly evolving to be more sophisticated and evasive to strike against current malware analysis and defense systems. This dissertation takes a root-cause oriented approach to the problem of automatic malware detection and analysis. In this approach, we aim to capture the intrinsic natures of malicious behaviors, rather than the external symptoms of existing attacks. We propose a new architecture for binary code analysis, which is called whole-system out-of-the-box fine-grained dynamic binary analysis, to address the common challenges in malware detection and analysis. to realize this architecture, we build a unified and extensible analysis platform, codenamed TEMU. We propose a core technique for fine-grained dynamic binary analysis, called layered annotative execution, and implement this technique in TEMU. Then on the basis of TEMU, we have proposed and built a series of novel techniques for automatic malware detection and analysis. For postmortem malware analysis, we have developed Renovo, Panorama, HookFinder, and MineSweeper, for detecting and analyzing various aspects of malware. For proactive malware detection, we have built HookScout as a proactive hook detection system. These techniques capture intrinsic characteristics of malware and thus are well suited for dealing with new malware samples and attack mechanisms

Aplicación de selección de características, métricas de aprendizaje y reducción de dimensión en sistemas de detección de intrusos /

Las redes de computadores inicialmente fueron diseñadas para una cantidad limitada de usuarios, hoy día se presentan como una necesidad para los hogares, pequeñas, medianas y grandes organizaciones. Los malos diseños de estructura de las redes de computadores han generado brechas de seguridad para mantener la integralidad, confidencialidad y disponibilidad de la información que es transferida por dicho medio, por ello existe la necesidad de proponer nuevas estrategias que permitan la identificación de ingresos no autorizados a las redes de computadores. El desarrollo de esta investigación tiene como propósito la aplicación de técnicas de selección de características, métricas de aprendizaje y reducción de dimensión en sistemas de detección de intrusos, utilizando los datos almacenados en el dataset NSL-KDD, el cual contiene 225.000 registros de conexiones en una red de computadores con 41 características.Incluye bibliografía, anexo

Buffer Overflow Vulnerability Diagnosis For Commodity Software

Buffer overflow attacks have been a computer security threat in software-based systems andapplications for decades. The existence of buffer overflow vulnerabilities makes the systemsusceptible to Internet worms and denial of service (DDoS) attacks which can cause hugesocial and financial impacts. Due to its importance, buffer overflow problem has been intensively studied. Researchershave proposed different techniques to defend against unknown buffer overflow attacks. Theyhave also investigated various solutions, including automatic signature generation, automatic patch generation, etc., to automatically protect computer systems with known vulnerabilities. The effectiveness and efficiency of the automatic signature generation approaches andthe automatic patch generation approaches are all based on the accurate understanding ofthe vulnerabilities, the buffer overflow vulnerability diagnosis (BOVD). Currently, the results of automatic signature generation and automatic patch generation are far from satisfaction due to the insufficient research results from the automatic BOVD. This thesis defines the automatic buffer overflow vulnerability diagnosis (BOVD) problemand provides solutions towards automatic BOVD for commodity software. It targets oncommodity software when source code and symbol table are not available. The solutionscombine both of the dynamic analysis techniques and static analysis techniques to achievethe goal. Based on the observation that buffer overflow attack happens when the size of the destination buffer is smaller than the total number of writes after the data copy process if the buffer overflow attack happens through a data copy procedure, the diagnosis results return the information of the size of destination buffer, the total number of writes of a data copy procedure and how the user inputs are related with them. They are achieved through bound analysis, loop analysis and input analysis respectively. We demonstrate the effectiveness of this thesis approach using real world vulnerable applications including the buffer overflow vulnerabilities attacked by the record-setting Slammer and Blaster worms. This thesis also does the complete case study for buffer overflow vulnerabilities which may have independent interests to researchers. Our buffer overflow case study results can help other researchers to design more effective defense systems and debugging tools against buffer overflow attacks

Using Virtualisation to Protect Against Zero-Day Attacks

Bal, H.E. [Promotor]Bos, H.J. [Copromotor