2 research outputs found
Personal Email Spam Filtering with Minimal User Interaction
This thesis investigates ways to reduce or eliminate the necessity of user input to
learning-based personal email spam filters. Personal spam filters have been shown in
previous studies to yield superior effectiveness, at the cost of requiring extensive user training which may be burdensome or impossible.
This work describes new approaches to solve the problem of building a personal
spam filter that requires minimal user feedback. An initial study investigates how well a personal filter can learn from different sources of data, as opposed to user’s messages. Our initial studies show that inter-user training yields substantially inferior results to
intra-user training using the best known methods. Moreover, contrary to previous
literature, it is found that transfer learning degrades the performance of spam filters when the source of training and test sets belong to two different users or different times.
We also adapt and modify a graph-based semi-supervising learning algorithm to
build a filter that can classify an entire inbox trained on twenty or fewer user judgments.
Our experiments show that this approach compares well with previous techniques when
trained on as few as two training examples.
We also present the toolkit we developed to perform privacy-preserving user studies
on spam filters. This toolkit allows researchers to evaluate any spam filter that conforms to a standard interface defined by TREC, on real users’ email boxes. Researchers have access only to the TREC-style result file, and not to any content of a user’s email
stream.
To eliminate the necessity of feedback from the user, we build a personal autonomous filter that learns exclusively on the result of a global spam filter. Our laboratory experiments show that learning filters with no user input can substantially
improve the results of open-source and industry-leading commercial filters that employ no user-specific training. We use our toolkit to validate the performance of the
autonomous filter in a user study
Spam Filter Improvement Through Measurement
This work supports the thesis that sound quantitative evaluation for
spam filters leads to substantial improvement in the classification
of email. To this end, new laboratory testing methods and datasets
are introduced, and evidence is presented that their adoption at Text
REtrieval Conference (TREC)and elsewhere has led to an improvement in state of the art
spam filtering. While many of these improvements have been discovered
by others, the best-performing method known at this time -- spam filter
fusion -- was demonstrated by the author.
This work describes four principal dimensions of spam filter evaluation
methodology and spam filter improvement. An initial study investigates
the application of twelve open-source filter configurations in a laboratory
environment, using a stream of 50,000 messages captured from a single
recipient over eight months. The study measures the impact of user
feedback and on-line learning on filter performance using methodology
and measures which were released to the research community as the
TREC Spam Filter Evaluation Toolkit.
The toolkit was used as the basis of the TREC Spam Track, which the
author co-founded with Cormack. The Spam Track, in addition to evaluating
a new application (email spam), addressed the issue of testing systems
on both private and public data. While streams of private messages
are most realistic, they are not easy to come by and cannot be shared
with the research community as archival benchmarks. Using the toolkit,
participant filters were evaluated on both, and the differences found
not to substantially confound evaluation; as a result, public corpora
were validated as research tools. Over the course of TREC and similar
evaluation efforts, a dozen or more archival benchmarks --
some private and some public -- have become available.
The toolkit and methodology have spawned improvements in the state
of the art every year since its deployment in 2005. In 2005, 2006,
and 2007, the spam track yielded new best-performing systems based
on sequential compression models, orthogonal sparse bigram features,
logistic regression and support vector machines. Using the TREC participant
filters, we develop and demonstrate methods for on-line filter fusion
that outperform all other reported on-line personal spam filters