International audienceThis study proposes a malware detection strategy based on control flow graphs. It carries out experiments to evaluates the false-positive ratios of the proposed methods. Moreover, it presents some insight to establish detection methods sound with respect to some obfuscation techniques

Bonfante, Guillaume

Kaczmarek, Matthieu

Marion, Jean-Yves

INRIA a CCSD electronic archive server

HAL Id: inria-00176235https://hal.inria.fr/inria-00176235Submitted on 3 Oct 2007HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Control Flow Graphs as Malware SignaturesGuillaume Bonfante, Matthieu Kaczmarek, Jean-Yves MarionTo cite this version:Guillaume Bonfante, Matthieu Kaczmarek, Jean-Yves Marion. Control Flow Graphs as MalwareSignatures. International Workshop on the Theory of Computer Viruses, Matthieu Kaczmarek; Guil-laume Bonfante, May 2007, Nancy, France. ￿inria-00176235￿Control Flow Graphs as Malware SignaturesGuillaume Bonfante, Matthieu Kaczmarek and Jean-Yves MarionNancy-Université - Loria - INPL - Ecole Nationale Supérieure des Mines de NancyB.P. 239, 54506 Vandœuvre-lès-Nancy Cédex, FranceAbstractThis study proposes a malware detection strategy based on control flowgraphs. It carries out experiments to evaluates the false-positive ratios ofthe proposed methods. Moreover, it presents some insight to establishdetection methods sound with respect to some obfuscation techniques.IntroductionIn this paper, we propose a detection strategy based on control flow graphs(CFGs). More precisely, we show how flow graphs can be used as signatures.This is the first step of a broader semantics based approach to detection. Forthe last twenty years, commercial software has treated this problem with thehelp of detection strategies based on string signatures. This method requires adatabase of signatures continuously updated. To maintain the database, someexperts analyse every new malicious programs and try to reverse engineer acorresponding signature, with the constraint to produce as least as possiblefalse-positives. The complexity of such an analysis is illustrated by [7]. Theamount of time and manpower induced by this strategy is now challenged byadvanced obfuscation techniques [5, 1, 2].In order to enhance this strategy, current detection research interest in de-tecting semantic aspects. Different approaches have been proposed. In [14, 15],assembler instructions are specified in an equational logic and a prover is usedto identify malicious behaviour. The methods exposed in [4, 6] are based on ab-stract interpretation, the considered signatures are built on control flow graphsand on traces of system states. In [13, 10], CFGs and data flow are used todetect malware.This study follows the same vein. It carryies out some experiments usingCFGs as signatures. This detection strategy is motivated by the fact that thedomain of CFGs is at least as complex as the domain of strings. That is for anyinteger n, there are more than 2n differents CFGs composed of n nodes. It isworth to mention that the CFG extraction is fully automatic.This paper goes through the following roadmap. The Sect. 1 defines somenotions about detection and places this work in a sound theoretical context. TheSect. 2 describes the CFGs induced by x86 assembler which is the architecturetarget by our experiment. The experiments are presented in the Sect. 3. Finally,we question some problems opened by this work in the Sect. 4.121 DetectionDetectors. We consider the detection notion defined in [6]. Let P be the setof programs and M ⊂ P the set of malicious programs. We associate M to aset of signatures M. A detector on M is a function D : P × M → {0, 1}. Thenwe say that a program p is detected if there is a signature m ∈ M such thatD(p,m) = 1.Applying this definition on the classical signature scanning method, the setM would consist in the list of all known malware string signatures and theassociated detector would be defined by D(p,m) = 1 if m is a substring of pand D(p,m) = 0 otherwise.Evaluation. A detector aims at the discrimination of sane programs frommalware. As a result a detector can only be evaluated w.r.t a set of saneprograms S ⊂ P and set of malicious programs M ⊂ P. We suppose thatM ∩ S = ∅. We define w.r.t D, M and S the set of false-positives F+ = {p |p ∈ M ∧ ∀m ∈ M : D(p,m) = 0}. We associate to F+ the false-positive ratiodefined by the ratio between the cardinals of F+ and of S.2 CFGs in assembler x86Description. A control flow graph (CFG) is composed of linked nodes. Wedefine six kinds of node jmp, jcc, call, ret, inst and end which correspondto the control flow structure. First, x86 assembler programs have four kinds offlow instruction: non-conditional jumps (jmp), conditional jumps (jcc), func-tion calls (call) and function returns (ret). Then, we abstract any contiguoussequence of instructions without flow modification as a node of kind inst. Fi-nally, the kind end corresponds to the end of the program.This induces the following properties on the graph. Any CFG has a uniquenode of kind end. Nodes of kind jmp, call and inst have only one successor.Nodes of kind jcc have exactly two successors. Nodes of kind call and retare well parenthesised. If a node of kind ret has no corresponding node of kindcall then it is linked to the node of kind end. We present two examples of CFGextraction in Figure 1.We use the following structure to represent nodes.typedef struct node t{add t node add;add t ret add;kind t kind;node t ∗children;} node t;Reduction. While experimenting, we have observed that nodes of kind jmpand inst do not really provide information about the control flow. Indeed, theyhave a unique successor and they do not influence the successing flow – as docall nodes. Then, we propose to also experiment with reduced graph.We propose the following graph reduction. For any node of kind inst orjmp remove the node from the graph and link all its predecessors to its uniquesuccessor. The Figure 2 presents a CFG and its reduction.30x1288 add edi, 0x40x128b mul dword [0x4056c9]0x1291 mov [edi], eax0x1293 inc dword [0x4056c5]0x1299 cmp dword [0x4056c5], 0x2700x12a3 jnz 0x12880x12a5 pop edi0x12d1 call 0x126f0x12d5 mov dword [ebp−0x4], 0x00x126f push ebp0x1270 mov ebp, esp0x1272 push edi0x1273 lea edi , [0x405814]0x1279 mov eax, [ebp+0x8]0x127c mov [edi], eax0x127e mov dword [0x4056c5], 0x10x1288 add edi, 0x40x128b mul dword [0x4056c9]0x1291 mov [edi], eax0x1293 inc dword [0x4056c5]0x1299 cmp dword [0x4056c5], 0x2700x12a3 jnz 0x12880x12a5 pop edi0x12a6 retFigure 1: CFG extraction3 ExperimentsMethodology. These experiments examines CFGs as signatures for malwaredetection. Our experiment target win32 x86 architecture. A collection of 750programs has been gathered and the ClamAV scanner has been used to checkthat they are sane programs. Then, a collection of 2278 has been collected frompublic sources. CFGs have been automatically extracted from malware and con-stitute the signatures. Finally, two algorithm detection have been implemented,we evaluates their false-positive ratios.Detection based on CFGs. The first algorithm computes the detector Ddefined by D(p,m) = 1 if m if m is isomorphic to the CFG of p and D(p,m) = 0otherwise. Informally speaking, the detector D search for an exact malwareCFG. We divide our collections of sane programs and malware in four categorydepending on the size of the CFGs. In the first category the CFGs of theprograms have less that 50 nodes, in the second they have between 50 and 100nodes, in the third they have between 100 and 5000 nodes, and in last categorythey have more that 5000 nodes. We interest in false-positives for each category.The Table 1 presents the results.We observe that the false-positive ratio decrease w.r.t the size of CFGs.Under 100 nodes the CFG is not reliable to discriminate malware from saneprograms, but over 100 nodes the results are much better. This is really encour-4Figure 2: CFG reductionSize of CFGs < 50 50 − 100 100 − 5000 > 5000 OverallSane programs (S) 15 51 383 301 750Malware (M) 440 635 1033 170 2278False-positives (F+) 10 15 7 1 33Ratios 66.7% 29.4% 1.8% 0.3% 4.4%Table 1: Results of the experimentsaging because the detection accuracy increase with the size of programs whereclassical signature scanning has more false-positives when the size increases.As a result, CFG detection could enhance classical detection and it would beinteresting to associate the two methods.Detection based on reduced CFGs. The second method uses the detectorR defined by R(p,m) = 1 if m is the reduction of the CFG of p and R(p,m) = 0otherwise. In other terms, the detector R does not consider flow without branch.We do the same experiments as previously considering reduced CFGs. TheTable 2 presents the results.Size of CFGs < 50 50 − 100 100 − 5000 > 5000 OverallSane programs (S) 74 39 528 109 750Malware (M) 1138 228 850 62 2278False-positives (F+) 26 0 8 0 34Ratios 35.1% 0.0% 1.5% 0.0% 4.5%Table 2: Results of the experiments with reductionThis detection method add one new false-positive, but it has the advantageto group most of false-positives under 50 nodes. Over 50 nodes the accuracyis good. This experiment corroborates the hypothesis that nodes of kind instand jmp do not provide information about control flow.Moreover, using reduced graph, the detection becomes sound w.r.t someobfuscation techniques such as code reordering. By code reordering, we consider5the technique that consists in three steps. Split the program in blocks, permutesthem and link the block with jumps to recover the original flow.Comparison. For comparison, statistical methods used in [9] induce false-positive ratios between 0, 5% and 34%. A detector based on artificial neu-ral networks developed at IBM [12] presents a false-positive ratio lower that1%. The data mining methods surveyed in [11] present false-positive ratios be-tween 2, 2% and 47, 5%. Heuristics methods from antivirus industry tested in [8]present false-positive ratios lower than 0, 2%.4 Further ResearchWe have not considered host infection. In our framework, this detection wouldbe associated to sub-graph search. In [3], such a study has been carried out forthe virus MetaPHOR. The exposed results are promising.It would be interesting to face our methods to metamorphic engines in orderto evaluate the soundness w.r.t program obfuscation. We already know that thedetector R is sound w.r.t code permutation, but it would be of worth to studythe obfuscation techniques described in [11, 1, 2].We should also consider the specific problem of packed and encrypted mal-ware. In our study the signature of packed malware only target the decompres-sion procedure. As a result if the used packer is public, such as UPX, this causefalse-positive.References[1] B. Barak, O. Goldreich, R. Impagliazzo, S. Rudich, A. Sahai, S. Vadhan,and K. Yang. On the (im) possibility of obfuscating programs. LectureNotes in Computer Science, 2139:19–23, 2001.[2] P. Beaucamps and E. Filiol. On the possibility of practically obfuscatingprograms towards a unified perspective of code protection. Journal inComputer Virology, 3(1):3–21, April 2007.[3] D. Bruschi, Martignoni, L., and M. Monga. Detecting self-mutating mal-ware using control-flow graph matching. Technical report, Universit degliStudi di Milano, September 2006.[4] M. Christodorescu, S. Jha, S. Seshia, D. Song, and R. Bryant. Semantics-aware malware detection. IEEE Symposium on Security and Privacy, 2005.[5] C. Collberg, C. Thomborson, and D. Low. A taxonomy of obfuscatingtransformations. University of Auckland Technical Report, 170, 1997.[6] M. Dalla Preda, M. Christodorescu, S. Jha, and S. Debray. A Semantics-Based Approach to Malware Detection. In POPL’07, 2007.[7] P. Ferrie and P. Ször. Hunting for metamorphism. Virus Bulletin, march2001.6[8] D. Gryaznov. Scanners of the Year 2000: Heuristics. Proceedings of the 5thInternational Virus Bulletin, 1999.[9] J. Kephart and W. Arnold. Automatic Extraction of Computer Virus Sig-natures. 4th Virus Bulletin International Conference, pages 178–184, 1994.[10] A. Lakhotia and M. Mohammed. Imposing Order on Program Statementsto Assist Anti-Virus Scanners. Proceedings of Eleventh Working Conferenceon Reverse Engineering, IEEE Computer Society Press, 2004.[11] M. Schultz, E. Eskin, E. Zadok, and S. Stolfo. Data Mining Methods forDetection of New Malicious Executables. Proceedings of the IEEE Sympo-sium on Security and Privacy, page 38, 2001.[12] G. Tesauro, J. Kephart, and G. Sorkin. Neural networks for computervirus recognition. Expert, IEEE [see also IEEE Intelligent Systems andTheir Applications], 11(4):5–6, 1996.[13] M. Venable, M. Chouchane, M. Karim, and A. Lakhotia. Analyzing Mem-ory Accesses in Obfuscated x86 Executables. Conference on Detection ofIntrusions and Malware & Vulnerability Assessment (DIMVA), pages 1–18,2005.[14] M. Webster. Algebraic Specification of Computer Viruses and Their Envi-ronments. Selected Papers from the First Conference on Algebra and Coal-gebra in Computer Science Young Researchers Workshop, pages 99–113,2005.[15] M. Webster and G. Malcolm. Detection of metamorphic computer virusesusing algebraic specification. Journal in Computer Virology, 2(3):149–161,2006.

Control Flow Graphs as Malware Signatures

This study proposes a malware detection strategy based on control flow graphs. It carries out experiments to evaluates the false-positive ratios of the proposed methods. Moreover, it presents some insight to establish detection methods sound with respect to some obfuscation techniques

Control Flow Graphs as Malware Signatures

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server

CiteSeerX