We identify 10 generic pitfalls that can affect the experimental outcome of AI driven solutions in computer security. We find that they are prevalent in the literature and provide recommendations for overcoming them in the future

Arp, Daniel

Cavallaro, Lorenzo

Pendlebury, Feargus

Pierazzi, Fabio

Quiring, Erwin

Rieck, Konrad

Warnecke, Alexander

Wressnegger, Christian

English

UCL Discovery

Taking the Red Pill:Lessons Learned on Machine Learning for Computer SecurityDaniel Arp   Technische Universität Berlin and University College LondonErwin Quiring   ICSI and Ruhr University BochumFeargus Pendlebury   University College LondonAlexander Warnecke   Technische Universität Berlin and BIFOLDFabio Pierazzi   King’s College LondonChristian Wressnegger   KASTEL Security Research Labs and Karlsruher Institute of TechnologyLorenzo Cavallaro   University College LondonKonrad Rieck   Technische Universität Berlin and BIFOLDArti￿cial intelligence and ma-chine learning have enabled remark-able progress in science and industry.This advancement has naturally alsoimpacted computer security, withnearly every major vendor now mar-keting AI-driven solutions for threatanalysis and detection. Similarly, thenumber of research papers applyingmachine learning to solve securitytasks has literally exploded.These works come with the im-plicit promise that learning algo-rithms provide signi￿cant bene￿tscompared to traditional solutions. Inrecent years, however, di￿erent stud-ies have shown that learning-basedapproaches often fail to provide thepromised performance in practicedue to various restrictions ignoredin the original publications [e.g., 1, 2,3, 4]. In this article, we want to ask:Are there generic pitfalls that can af-fect the experimental outcome whenapplying machine learning in secu-rity? And, if so, how can researchersavoid stepping into them?Why Should I Care?As a thorough researcher, one mighttend to think “this can never happento me.” However, as we will discussin this article, pitfalls come in var-ious forms and ￿avors, some obvi-ous but others noticeable only witha very cautious eye. Hence, evenexperienced researchers might stepinto them from time to time withoutnoticing. With this work, we want toraise awareness of these issues in theresearch community to reduce theirprevalence in security research. De-tailed recommendations and guide-lines for each of the pitfalls can befound in the original papaer [5].Entering the MatrixTo illustrate the problem withoutpointing ￿ngers at others, let us con-sider the ￿ctional company Meta-Cortex. The company specializes inthe discovery of security vulnera-bilities in source code. The neces-sary code analysis, however, is time-consuming and tedious. To speed upthe process, the company decides toleverage machine learning to auto-mate the discovery of vulnerabilities,as corresponding methods have al-ready been seemingly successfullyapplied in the literature [e.g., 6, 7].Speci￿cally, MetaCortex choosesto build its solution on a deep neu-ral network due to the popularity ofthis learning model in other ￿elds.And indeed, the developed modelachieves an excellent performanceon a common benchmark dataset, sothat the company decides to inte-grate it into a new product and sell itto its customers, awaiting a big com-mercial success.Taking the Red PillUnfortunately, it turns out that themodel performance cannot keep upwith the promises made, and Meta-Cortex’s customers start to complain.What went wrong? A closer lookat the company’s machine learningpipeline uncovers various pitfallsthe company accidentally stepped inwhile developing their model.In the following, we discusssome of the factors that might haveled to an over-estimation of themethod’s capabilities. Note that,while we focus on this solely ￿ctiveexample, all discussed issues are in-spired by actual mistakes found inprevious works.Spurious Correlations. Eventhough deep neural networks haveled to major breakthroughs in var-ious areas, it is often unclear whythey achieve this impressive per-formance. Fortunately, in recentyears, several methods have beendeveloped that enable the interpreta-tion of these models, shedding somelight on the decision-making processof neural networks [8]. The appli-cation of an explanation method,like Layer-wise Relevance Propaga-This a preprint version of an article published in IEEE Security & Privacy Magazine, September 2023PresentPresent (but discussed)Partly presentPartly present (but discussed)Unclear from textDoes not applyNot presentCommon pitfalls with prevalence in  the machine learning  for security literatureSampling biasLabel inaccuracyData snoopingSpurious correlationsBiased parametersInappropriate baselinesInappropriate measuresBase rate fallacyLab-only evaluationInappropriate threat modelP1P2P3P4P5 P8P7P6 P9P10Data collection  and labeling System design and learningPerformance  evaluationDeployment and operationMachine learning workflowSecurity problem, e.g., novel attacksSecurity solution, e.g., learning-based IDSFigure 1: Common pitfalls of machine learning in computer security and their prevalence in the literature.tion (LRP) [9], allows researchers toinspect the features most relevant tothe prediction.In the case of our ￿ctive com-pany, we ￿nd that the highlightedfeatures are barely connected to secu-rity vulnerabilities. Instead, the mostrelevant features are meaningless to-kens like brackets or commas in thesource code, which have no seman-tic relevance to vulnerable code andthus represent non-causal spuriouscorrelations. It seems as if these arti-facts serve as shortcuts that allow themodel to distinguish between vulner-able and non-vulnerable code. Buthow did this happen?Sampling Bias. A possible andcommon reason for the presenceof spurious correlations is samplingbias. In this case, the distributionof the training data does not su￿-ciently represent the distribution attest time. Consequently, the modelis not able to learn the underlyingconcept of the given task, but ratherrelies on artifacts introduced in thetraining distribution. When compos-ing a dataset for training the model,researchers need to be aware thatthere exist a variety of sources forsampling bias, some of which beingvery subtle. The strategy for collect-ing the data might thus bias the re-sulting dataset towards certain soft-ware versions, code authors, or pro-gramming languages. These factorsare then (maybe indirectly) used bythe machine learning model to dis-tinguish between classes, instead ofsolving the actual task [e.g., 5, 10].Data Snooping. Let us assumethat the collected dataset does notsu￿er from sampling bias. Are thereany other—less known–issues thatmight result in huge di￿erences inthe performance at training and testtime? Indeed, another common pit-fall that can lead to over-optimisticresults is commonly referred to asdata snooping. Here, a learningmodel is trained with informationthat would not be available in prac-tice. While this appears to be a pit-fall that can be avoided very easilyat the ￿rst glance, it turns out to bemuch harder to avoid than expectedin many cases.The reason is that data snoop-ing exists in many di￿erent forms,some of which can be easily over-looked. For example, using incorrecttime splits that ignore time depen-dencies within the data can in￿atethe actual performance [1]. Similarly,this pitfall applies if noisy data isremoved from the test set based onknowledge that would normally notbe available at training time. Evenmore subtle, solely evaluating well-known benchmark data might alsoover-estimate the performance. Es-tablished benchmarks come with ahistory, so that researchers may un-noticeably use knowledge from priorwork—including insights from thetest distribution.The Greater PictureThe previous example illustrates thatthere obviously exists a number ofpitfalls that can potentially harm theexperimental outcome. However, thepreviously discussed issues are justthe tip of the iceberg and the Meta-Cortex company likely faces furtherproblems.In this section, we provide a sys-tematic overview of common pit-falls and their prevalence in a set of30 top publications we selected forour study. We follow the individualstages of a typical machine-learningpipeline that Figure 1 illustrates to-gether with all pitfalls. The coloredbar shows their prevalence in ourstudy, with warmer colors depictingthe presence of a pitfall. Table 1 sum-marizes each pitfall.2Pitfall DescriptionP1 Sampling Bias The composed dataset does not su￿ciently represent the actual distribution.P2 Label Inaccuracy The ground-truth labels are inaccurate, unstable, or erroneous.P3 Data Snooping Information is used at training time that is usually not available in practice.P4 Spurious correlations A learning model relies on false associations caused by artifacts unrelated to the task.P5 Biased Parameter Selection Final parameters of a learning method are indirectly determined on the test set.P6 Inappropriate Baseline No adequate baseline methods are used in the evaluation for comparison.P7 Inappropriate Performance Measures Used performance measures are not suitable for the application scenario.P8 Base Rate Fallacy Large class imbalance is ignored when interpreting the performance.P9 Lab-Only Evaluation The developed system is only tested in a laboratory setting.P10 Inappropriate Threat Model Attacks against the machine-learning component itself are not considered.Table 1: Overview of common pitfalls of machine learning in computer security.(1) Data collection and labelingphase. Before we can start withdeveloping a new learning-basedmethod, we ￿rst have to collect anexpressive dataset that resembles thedata distribution we assume to see inpractice. Moreover, we also often re-quire meaningful label informationif we want to apply supervised learn-ing. Unfortunately, the compositionof a realistic dataset with labels is of-ten challenging, leading to the ￿rsttwo pitfalls: (P1) Sampling bias and(P2) label inaccuracy.We have already discussed howsampling bias can a￿ect the exper-imental outcome. Similarly, in thecase of P2, the labels are erroneousor unstable, which, in turn, can alsoimpact the performance of a learningmodel if we do not correct this noise.(2) System design and learningphase. Once we have composed adataset, we can design and train ourlearning model. This stage includesthe pre-processing of the data, the ex-traction of suitable features, as wellas learning the actual model. In thisstage, we can step in three pitfallswhen not being careful: (P3) Datasnooping, (P4) spurious correlations,and (P5) biased parameter selection.In the case of P3 and P5, theseparation of training and test par-tition is ￿awed, so that the modeluses information that is unavailableat test time, biasing the outcome ofthe experimental setup. For instance,the developers might ignore timedependencies within the collecteddata, such that the machine learningmodel is trained on data comprisingfuture knowledge (which is not avail-able outside the matrix). Another is-sue arises from P4. Here, the featuredesign allows the model to pick upon artifacts unrelated to the securitypattern, thus creating a shortcut forsolving the actual tasks. While thiscan be unproblematic in some cases,it can also lead to serious problemsand let the model fail completely dur-ing its deployment.(3) Evaluation Phase. In the nextstage, we evaluate the previouslytrained model and examine its per-formance on test data. Here, we haveto pay attention to not step into oneof the following three pitfalls: (P6)Inappropriate baseline, (P7) inappro-priate performance measures, or (P8)the base rate fallacy [11].In the case of P6, the learningmodel is not compared against suit-able baseline approaches. For in-stance, a simple, non-learning-basedmethod can sometimes achieve asimilar or even better performancethan a complex deep neural network.However, due the lack of compari-son, this fact remains hidden. A sim-ilar problem arises if the chosen per-formance measures are not appro-priate for the application scenario(P7). As an example, we often have todeal with highly imbalanced datasetsin security, like in malware detec-tion. In these cases, we have to iden-tify malicious objects that representonly a small proportion of the en-tire data distribution. When usingthe wrong metrics in these settings,like the accuracy, one gets an en-tirely false estimate of the true per-formance of a learning-based system.Moreover, even if proper metrics areused, the performance of a systemmight still be overestimated by ignor-ing the base rate of the negative classin reality (P8). Let us assume, for ex-ample, a seemingly e￿cient classi￿erwith 99% true positives at 1% falsepositives. Yet, if we have a class ratioof 1:100, even 1% false positives stillcause 100 false positives for every 99true positives.3(4) Deployment and OperationPhase. Finally, we obtain a learn-ing model whose detection perfor-mance meets our requirements. Wecan now deploy and operate it in thewild. We might already assume thatwe have successfully escaped the ma-trix. Unfortunately, there are stilltwo additional pitfalls that can havea severe impact on the performance.First, we should account for anypractical limitations that we did notconsider throughout the evaluation.Oftentimes, new learning methodsare solely evaluated in lab-only en-vironments (P9), where crucial con-straints of realistic settings are ig-nored, such as run-time or storagerestrictions. As a result, a promis-ing method might turn out to be un-suitable in a production setting. Fur-thermore, we need to consider thesecurity of our learning-based system,as adversaries might run targeted at-tacks against it (P10). For instance,malicious actors could try to circum-vent detection or derive informationabout the underlying learning model.Bending the SpoonNaturally, the question arises howlikely each of the previously dis-cussed pitfalls occurs. To get an intu-ition, we review 30 academic paperspublished at top conferences for se-curity between 2011 and 2020. Whenselecting the papers, we ensure thatthey cover a wide range of security-related topics, ranging from learning-based malware detection to intelli-gent vulnerability discovery. If a pit-fall’s presence is unclear, the review-ers decide conservatively and alwaysgive the authors the bene￿t of thedoubt. A precise description of ourassessment criteria can be found inthe original article [5].Figure 1 highlights the outcomeof the study. We ￿nd that the pitfallsare widespread even in top research.Each paper is a￿ected by at leastthree of the discussed issues. Themost prevalent pitfall is samplingbias (P1), followed by data snooping(P3), which are at least partly presentin 90% and 73% of the consideredpublications, respectively. Similarly,other pitfalls occur frequently, suchas the use of inappropriate perfor-mance measures (P7) or the evalua-tion in a lab-only setting (P9), both ofwhich appear in at least 50% of all thepapers. Interestingly, we ￿nd thatthat the presence of a pitfall is onlyaccompanied by a discussion in 22%of the cases, indicating that there isa lack of awareness regarding thesecommon issues.To get a full picture of the situ-ation, we have also collected feed-back from the authors of the re-viewed papers. The vast majorityof the authors from which we re-ceived a response agreed that thereis a lack of awareness for the iden-ti￿ed pitfalls and con￿rm that theseare widespread in security research.Escaping the MatrixThe discussed pitfalls are more thanjust an academic problem. In fact,they introduce severe biases and hin-der actual progress in research. As aresult, we need to discuss within thecommunity how to overcome theseproblems in the future.First and foremost, it is possi-ble to avoid the identi￿ed pitfalls inmany cases. Therefore, we recom-mend double-checking each stage ofthe machine learning pipeline andlooking out for potential issues whendeveloping a new approach. To thisend, detailed recommendations andguidelines for each of the pitfallscan be found in the original publi-cation [5]. For instance, methods to￿x inaccurate labels or methods ofexplainable AI (XAI) to check for spu-rious correlations are applicable.Unfortunately, there exist casesin which it can be challenging toavoid a pitfall entirely. As an exam-ple, it might be hard to compensatefor sampling bias due to a lack ofdata. In these cases, it is crucial toopenly discuss the problem so thatother researchers can solve it in thefuture. In general, we thus recom-mend to “do your best” by mitigatingpitfalls where possible and acknowl-edge remaining problems openly.Finally, we like to stress that itis not our intention to take the funout of machine learning in securityresearch. In fact, the opposite is true.We strive to promote sound researchand bring the enormous potential ofarti￿cial intelligence into the realityof security, so that red pills are nolonger necessary.References[1] F. Pendlebury, F. Pierazzi, R. Jor-daney, J. Kinder, and L. Cavallaro.TESSERACT: Eliminating exper-imental bias in malware classi￿-cation across space and time. In:Proc. of USENIX Security Sympo-sium. 2019, pp.729–746.[2] M. Juarez, S. Afroz, G. Acar,C. Diaz, and R. Greenstadt. ACritical Evaluation of WebsiteFingerprinting Attacks. In: Proc.of ACM Conference on Com-puter and Communications Secu-rity (CCS). 2014.[3] S. Kapoor and A. Narayanan.Leakage and the ReproducibilityCrisis in ML-based Science. In:arXiv preprint arXiv:2207.07048(2022).4[4] L. Cavallaro, J. Kinder, F. Pendle-bury, and F. Pierazzi. Are Ma-chine Learning Models for Mal-ware Detection Ready for PrimeTime? In: IEEE Security & Privacy21(2) (2023), 53–56.[5] D. Arp, E. Quiring, F. Pendle-bury, A. Warnecke, F. Pierazzi, C.Wressnegger, L. Cavallaro, and K.Rieck. Dos and don’ts of machinelearning in computer security. In:Proc. of USENIX Security Sympo-sium. 2022.[6] F. Yamaguchi, N. Golde, D. Arp,and K. Rieck. Modeling andDiscovering Vulnerabilities withCode Property Graphs. In: IEEESymposium on Security and Pri-vacy (S&P). 2014, pp.590–604.[7] Z. Li, D. Zou, S. Xu, X. Ou, H. Jin,S. Wang, Z. Deng, and Y. Zhong.VulDeePecker: A Deep Learning-Based System for VulnerabilityDetection. In: Proc. of Networkand Distributed System SecuritySymposium (NDSS). 2018.[8] A. Warnecke, D. Arp, C. Wress-negger, and K. Rieck. EvaluatingExplanation Methods for DeepLearning in Security. In: Proc. ofthe IEEE European Symposium onSecurity and Privacy (EuroS&P).2020.[9] S. Bach, A. Binder, G. Montavon,F. Klauschen, K.-R. Müller, andW. Samek. On Pixel-Wise Expla-nations for Non-Linear Classi￿erDecisions by Layer-Wise Rele-vance Propagation. In: PLOS ONE10(7) (July 2015), 1–46.[10] S. Chakraborty, R. Krishna, Y.Ding, and B. Ray. Deep Learn-ing Based Vulnerability Detec-tion: Are We There Yet? In: IEEETransactions on Software Engi-neering 48(9) (2022), 3280–3296.[11] S. Axelsson. The base-rate fallacyand the di￿culty of intrusion de-tection. In: ACM Transactions onInformation and System Security(TISSEC) 3(3) (2000), 186–205.5

Lessons Learned on Machine Learning for Computer Security

https://discovery.ucl.ac.uk/10176802/1/2023-spmag.pdf

Lessons Learned on Machine Learning for Computer Security

Abstract

Similar works

Full text

Available Versions

UCL Discovery