Skip to main content
Article thumbnail
Location of Repository

Network based buffer overflow detection by exploit code analysis

By Stig Andersson, Andrew Clark and George Mohay


Buffer overflow attacks continue to be a major security problem and detecting attacks of this nature is therefore crucial to network security. Signature based network based intrusion detection systems (NIDS) compare network traffic to signatures modelling suspicious or attack traffic to detect network attacks. Since detection is based on pattern matching, a signature modelling the attack must exist for the NIDS to detect it, and it is therefore only capable of detecting known attacks. This paper proposes a method to detect buffer overflow attacks by parsing the payload of network packets in search of shellcode which is the remotely executable component of a buffer overflow attack. By analysing the shellcode it is possible to determine which system calls the exploit uses, and hence the operation of the exploit. Current NIDS-based buffer overflow detection techniques mainly rely upon specific signatures for each new attack. Our approach is able to detect previously unseen buffer overflow attacks, in addition to existing ones, without the need for specific signatures for each new attack. The method has been implemented and tested for buffer overflow attacks on Linux on the Intel x86 architecture using the Snort NIDS

Topics: 080303 Computer System Security, intrusion detection, buffer overflow detection, network monitoring, network security
Publisher: University of Queensland
Year: 2004
OAI identifier:

Suggested articles

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.