Measurement and Evaluation of a Real World Deployment of a Challenge-Response Spam Filter

Abstract

Despite the numberof existing solutions, spam still accounts for a large percentage of the email traffic on the Internet. Both the effectiveness andthe impact of manycommon antispam techniques have already been largely studied and evaluated against multiple datasets. However, some of the less known solutions still lack a proper experimental validation. For example, Challenge-Response (CR) systems have been largely discussed, and often criticized, because they shift the effort to protect the user’s mailbox from the recipient to the sender of the messages. In addition, these systems are believed to produce a lot of backscattered emails that further deteriorate the global Internet situation. In this paper we present the first comprehensive measurement study of a real anti-spam system based on a challengeresponse technique. In our work we analyze a large amount of data, collected for a period of six months from over forty companiesprotectedbyacommercialchallenge-responseproduct. We designed our experiments from three different point of views: the end user, the system administrator, and the entire Internet community. Our results cover many different aspects such as the amount of challenges sent, the delay on the message delivery, and the likelihood of getting the challenge server blacklisted. Our aim is neither to attack nor to defend CR-based solutions. Instead, we hope that our findings will shed some light on some of the myths about these kind of systems, and will help both users and companies to take an informed decision on the topic