Improving Attack Detection in Host-Based IDS by Learning Properties of System Call Arguments

Abstract

Beginning with the work of Forrest et al, several researchers have developed intrusion detection techniques based on modeling program behaviors in terms of system calls. A weakness of these techniques is that they focus primarily on system call names, and not the arguments. This weakness makes them susceptible to several classes of attacks, including mimicry attacks, attacks on securitycritical data, and race-condition attacks. To address this weakness, we present an approach for capturing data-flow behaviors of programs. We provide a formal definition of data-flow behaviors on system call traces, and describe efficient algorithms for building such models. Our algorithm can be layered on top of most existing control-flow models. This layering increases the precision of models by exploting control-flow context to refine data-flow properties. We present a detailed experimental evaluation to establish the effectiveness of the approach, paying particular attention to detection of sophisticated attacks. A unique benefit of our approach is that the models contain sufficient information regarding resources (such as files) accessed by a program to make it feasible to formally reason about the security assurances provided by the model.