Analyzing the Social Structure and Dynamics of E-mail and Spam in Massive Backbone Internet Traffic

E-mail is probably the most popular application on the Internet, with everyday business and personal communications dependent on it. Spam or unsolicited e-mail has been estimated to cost businesses significant amounts of money. However, our understanding of the network-level behavior of legitimate e-mail traffic and how it differs from spam traffic is limited. In this study, we have passively captured SMTP packets from a 10 Gbit/s Internet backbone link to construct a social network of e-mail users based on their exchanged e-mails. The focus of this paper is on the graph metrics indicating various structural properties of e-mail networks and how they evolve over time. This study also looks into the differences in the structural and temporal characteristics of spam and non-spam networks. Our analysis on the collected data allows us to show several differences between the behavior of spam and legitimate e-mail traffic, which can help us to understand the behavior of spammers and give us the knowledge to statistically model spam traffic on the network-level in order to complement current spam detection techniques.Comment: 15 pages, 20 figures, technical repor

Minimizing the Time of Spam Mail Detection by Relocating Filtering System to the Sender Mail Server

Unsolicited Bulk Emails (also known as Spam) are undesirable emails sent to massive number of users. Spam emails consume the network resources and cause lots of security uncertainties. As we studied, the location where the spam filter operates in is an important parameter to preserve network resources. Although there are many different methods to block spam emails, most of program developers only intend to block spam emails from being delivered to their clients. In this paper, we will introduce a new and efficient approach to prevent spam emails from being transferred. The result shows that if we focus on developing a filtering method for spams emails in the sender mail server rather than the receiver mail server, we can detect the spam emails in the shortest time consequently to avoid wasting network resources.Comment: 10 pages, 7 figure

Detecting spam relays by SMTP traffic characteristics using an autonomous detection system

Spam emails are flooding the Internet. Research to prevent spam is an ongoing concern. SMTP traffic was collected from different sources in real networks and analyzed to determine the difference regarding SMTP traffic characteristics of legitimate email clients, legitimate email servers and spam relays. It is found that SMTP traffic from legitimate sites and non-legitimate sites are different and could be distinguished from each other. Some methods, which are based on analyzing SMTP traffic characteristics, were purposed to identify spam relays in the network in this thesis. An autonomous combination system, in which machine learning technologies were employed, was developed to identify spam relays in this thesis. This system identifies spam relays in real time before spam emails get to an end user by using SMTP traffic characteristics never involving email real content. A series of tests were conducted to evaluate the performance of this system. And results show that the system can identify spam relays with a high spam relay detection rate and an acceptable ratio of false positive errors

Modelling email traffic workloads with RNN and LSTM models

Analysis of time series data has been a challenging research subject for decades. Email traffic has recently been modelled as a time series function using a Recurrent Neural Network (RNN) and RNNs were shown to provide higher prediction accuracy than previous probabilistic models from the literature. Given the exponential rise of email workloads which need to be handled by email servers, in this paper we first present and discuss the literature on modelling email traffic. We then explain the advantages and limitations of different approaches as well as their points of agreement and disagreement. Finally, we present a comprehensive comparison between the performance of RNN and Long Short Term Memory (LSTM) models. Our experimental results demonstrate that both approaches can achieve high accuracy over four large datasets acquired from different universities’ servers, outperforming existing work, and show that the use of LSTM and RNN is very promising for modelling email traffic

CallRank: Combating SPIT using call duration, social networks and global reputation

The growing popularity of IP telephony systems has made them attractive targets for spammers. Voice call spam, also known as Spam over Internet Telephony (SPIT), is potentially a more serious problem than email spam because of the real time processing requirements of voice packets. We explore a novel mechanism that uses duration of calls between users to combat SPIT. CallRank, the scheme proposed by us, uses call duration to establish social network linkages and global reputations for callers, based on which call recipients can decide whether the caller is legitimate or not. CallRank has been implemented within a VoIP system simulation and our results show that we are able to achieve a false negative rate of 10 % and a false positive rate of 3% even in the presence of a significant fraction of spammers

Context-Aware Network Security.

The rapid growth in malicious Internet activity, due to the rise of threats like automated worms, viruses, and botnets, has driven the development of tools designed to protect host and network resources. One approach that has gained significant popularity is the use of network based security systems. These systems are deployed on the network to detect, characterize and mitigate both new and existing threats. Unfortunately, these systems are developed and deployed in production networks as generic systems and little thought has been paid to customization. Even when it is possible to customize these devices, the approaches for customization are largely manual or ad hoc. Our observation of the production networks suggest that these networks have significant diversity in end-host characteristics, threat landscape, and traffic behavior -- a collection of features that we call the security context of a network. The scale and diversity in security context of production networks make manual or ad hoc customization of security systems difficult. Our thesis is that automated adaptation to the security context can be used to significantly improve the performance and accuracy of network-based security systems. In order to evaluate our thesis, we explore a system from three broad categories of network-based security systems: known threat detection, new threat detection, and reputation-based mitigation. For known threat detection, we examine a signature-based intrusion detection system and show that the system performance improves significantly if it is aware of the signature set and the traffic characteristics of the network. Second, we explore a large collection of honeypots (or honeynet) that are used to detect new threats. We show that operating system and application configurations in the network impact honeynet accuracy and adapting to the surrounding network provides a significantly better view of the network threats. Last, we apply our context-aware approach to a reputation-based system for spam blacklist generation and show how traffic characteristics on the network can be used to significantly improve its accuracy. We conclude with the lessons learned from our experiences adapting to network security context and the future directions for adapting network-based security systems to the security context.Ph.D.Computer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/64745/1/sushant_1.pd