Control Flow to Detect Malware

International audienceThis study proposes a malware detection strategy based on control flow. It consists in searching in the control flow graph of the analysed program for an induced sub-graph which corresponds to the control flow graphs of a malicious program. The resulting detector is build over a strong theoretical framework.Finally, experiments are carried out in order to evaluates the proposed detection strategy

Morphological Detection of Malware

International audienceIn the field of malware detection, method based on syntactical consideration are usually efficient. However, they are strongly vulnerable to obfuscation techniques. This study proposes an efficient construction of a morphological malware detector based on a syntactic and a semantic analysis, technically on control flow graphs of programs (CFG). Our construction employs tree automata techniques to provide an efficient representation of the CFG database. Next, we deal with classic obfuscation of programs by mutation using a generic graph rewriting engine. Finally, we carry out experiments to evaluate the false-positive ratio of the proposed methods

An enhanced performance model for metamorphic computer virus classification and detectioN

Metamorphic computer virus employs various code mutation techniques to change its code to become new generations. These generations have similar behavior and functionality and yet, they could not be detected by most commercial antivirus because their solutions depend on a signature database and make use of string signature-based detection methods. However, the antivirus detection engine can be avoided by metamorphism techniques. The purpose of this study is to develop a performance model based on computer virus classification and detection. The model would also be able to examine portable executable files that would classify and detect metamorphic computer viruses. A Hidden Markov Model implemented on portable executable files was employed to classify and detect the metamorphic viruses. This proposed model that produce common virus statistical patterns was evaluated by comparing the results with previous related works and famous commercial antiviruses. This was done by investigating the metamorphic computer viruses and their features, and the existing classifications and detection methods. Specifically, this model was applied on binary format of portable executable files and it was able to classify if the files belonged to a virus family. Besides that, the performance of the model, practically implemented and tested, was also evaluated based on detection rate and overall accuracy. The findings indicated that the proposed model is able to classify and detect the metamorphic virus variants in portable executable file format with a high average of 99.7% detection rate. The implementation of the model is proven useful and applicable for antivirus programs

Sub-circuit Selection and Replacement Algorithms Modeled as Term Rewriting Systems

Intent protection is a model of software obfuscation which, among other criteria, prevents an adversary from understanding the program’s function for use with contextual information. Relating this framework for obfuscation to malware detection, if a malware detector can perfectly normalize a program P and any obfuscation (variant) of the program O(P), the program is not intent protected. The problem of intent protection on programs can also be modeled as intent protection on combinational logic circuits. If a malware detector can perfectly normalize a circuit C and any obfuscation (variant) O(C) of the circuit, the circuit is not intent protected. In this effort, the research group set the primary goal as determining if a malware detector based upon the mechanisms of term rewriting theory can perfectly normalize circuits transformed by a sub-circuit selection and replacement algorithm, even when the transformation algorithm is known. The research group set the secondary goal as relating this result on circuit transformations to the realm of software obfuscation. The transformation rules of the sub-circuit selection and replacement algorithm are identified and modeled as rewrite rules in a term rewriting system. These rewrite rules are examined for critical overlaps which cannot be resolved by a widely used completion algorithm known as Knuth-Bendix. The research group performs an analysis of the critical overlaps found within the rewrite rules and successfully relates these results to the instruction-substitution obfuscations of a software obfuscator

A Semantics-Based Approach to Malware Detection

Malware detection is a crucial aspect of software security. Current malware detectors work by checking for signatures, which attempt to capture the syntactic characteristics of the machine-level byte sequence of the malware. This reliance on a syntactic approach makes current detectors vulnerable to code obfuscations, increasingly used by malware writers, that alter the syntactic properties of the malware byte sequence without significantly affecting their execution behavior. This paper takes the position that the key to malware identification lies in their semantics. It proposes a semantics-based framework for reasoning about malware detectors and proving properties such as soundness and completeness of these detectors. Our approach uses a trace semantics to characterize the behavior of malware as well as that of the program being checked for infection, and uses abstract interpretation to ``hide'' irrelevant aspects of these behaviors. As a concrete application of our approach, we show that (1) standard signature matching detection schemes are generally sound but not complete, (2) the semantics-aware malware detector proposed byChristodorescu et al. is complete with respect to a number of common obfuscations used by malware writers and (3) the malware detection scheme proposed by Kinder et al. and based on standard model-checking techniques is sound in general and complete on some, but not all, obfuscations handled by the semantics-aware malware detector

Developing a Qualia-Based Multi-Agent Architecture for Use in Malware Detection

Detecting network intruders and malicious software is a significant problem for network administrators and security experts. New threats are emerging at an increasing rate, and current signature and statistics-based techniques are not keeping pace. Intelligent systems that can adapt to new threats are needed to mitigate these new strains of malware as they are released. This research detects malware based on its qualia, or essence rather than its low-level implementation details. By looking for the underlying concepts that make a piece of software malicious, this research avoids the pitfalls of static solutions that focus on predefined bit sequence signatures or anomaly thresholds. This research develops a novel, hierarchical modeling method to represent a computing system and demonstrates the representation’s effectiveness by modeling the Blaster worm. Using Latent Dirichlet Allocation and Support Vector Machines abstract concepts are automatically generated that can be used in the hierarchical model for malware detection. Finally, the research outlines a novel system that uses multiple levels of individual software agents that sharing contextual relationships and information across different levels of abstraction to make decisions. This qualia-based system provides a framework for developing intelligent classification and decision-making systems for a number of application areas

Deterministic, Efficient Variation of Circuit Components to Improve Resistance to Reverse Engineering

This research proposes two alternative methods for generating semantically equivalent circuit variants which leave the circuit\u27s internal structure pseudo-randomly determined. Component fusion deterministically selects subcircuits using a component identification algorithm and replaces them using a deterministic algorithm that generates canonical logic forms. Component encryption seeks to alter the semantics of individual circuit components using an encoding function, but preserves the overall circuit semantics by decoding signal values later in the circuit. Experiments were conducted to examine the performance of component fusion and component encryption against representative trials of subcircuit selection-and-replacement and Boundary Blurring, two previously defined methods for circuit obfuscation. Overall, results support the conclusion that both component fusion and component encryption generate more secure variants than previous methods and that these variants are more efficient in terms of required circuit delay and the power and area required for their implementation

Evaluation Methodologies in Software Protection Research

Man-at-the-end (MATE) attackers have full control over the system on which the attacked software runs, and try to break the confidentiality or integrity of assets embedded in the software. Both companies and malware authors want to prevent such attacks. This has driven an arms race between attackers and defenders, resulting in a plethora of different protection and analysis methods. However, it remains difficult to measure the strength of protections because MATE attackers can reach their goals in many different ways and a universally accepted evaluation methodology does not exist. This survey systematically reviews the evaluation methodologies of papers on obfuscation, a major class of protections against MATE attacks. For 572 papers, we collected 113 aspects of their evaluation methodologies, ranging from sample set types and sizes, over sample treatment, to performed measurements. We provide detailed insights into how the academic state of the art evaluates both the protections and analyses thereon. In summary, there is a clear need for better evaluation methodologies. We identify nine challenges for software protection evaluations, which represent threats to the validity, reproducibility, and interpretation of research results in the context of MATE attacks

Normalizing metamorphic malware using term rewriting

The paper presents a method for detecting metamorphic malicious programs which replacing sequence of instructions by equivalent instructions and by inserting irrelevant code fragments. The source-to-source transformations of a metamorphic malware are modeled as a term rewriting system and well known concepts from term rewriting are used to construct a malware normalizer: an algorithm that maps the set of all variants of a target malware to a single code segment called the ‘normal form’. If the result of transforming a suspect program by the normalizer is a program that matches the normal form, then the suspect program is a variant of the target malware. When such a normalizer cannot be constructed, such as due to the inability to complete the rewrite rules, it is proposed that a normalizer be approximated by using a priority scheme to resolve conflicts in the application of the rewrite rules. The proposed malware normalization method considerably reduces the space/time cost incurred by static signaturebased scanners attempting to detect variants of a target metamorphic malware. A case study is presented that successfully eliminated the need to store a unique signature for each of the variants of the “W32.Evol ” virus by constructing a normalizer for the variants.