A machine learning approach to server-side anti-spam e-mail filtering

Spam-detection systems based on traditional methods have several obvious disadvantages like low detection rate, necessity of regular knowledge bases’ updates, impersonal filtering rules. New intelligent methods for spam detection, which use statistical and machine learning algorithms, solve these problems successfully. But these methods are not widespread in spam filtering for enterprise-level mail servers, because of their high resources consumption and insufficient accuracy regarding false-positive errors. The developed solution offers precise and fast algorithm. Its classification quality is better than the quality of Naïve-Bayes method that is the most widespread machine learning method now. The problem of time efficiency that is typical for all learning based methods for spam filtering is solved using multi-agent architecture. It allows easy system scaling and building unified corporate spam detection system based on heterogeneous enterprise mail systems. Pilot program implementation and its experimental evaluation for standard data sets and for real mail flows have demonstrated that our approach outperforms existing learning and traditional spam filtering methods. That allows considering it as a promising platform for constructing enterprise spam filtering systems

An Efficient Framework of Utilizing the Latent Semantic Analysis in Text Extraction

The use of the latent semantic analysis (LSA) in text mining demands large space and time requirements. This paper proposes a new text extraction method that sets a framework on how to employ the statistical semantic analysis in the text extraction in an efficient way. The method uses the centrality feature and omits the segments of the text that have a high verbatim, statistical, or semantic similarity with previously processed segments. The identification of similarity is based on a new multi-layer similarity method that computes the similarity in three statistical layers, it uses the Jaccard similarity and the vector space model in the first and second layers respectively, and uses the LSA in the third layer. The multi-layer similarity restricts the use of the third layer for the segments that the first and second layers failed to estimate their similarities. Rouge tool is used in the evaluation, but because Rouge does not consider the extract’s size, we supplemented it with a new evaluation strategy based on the compression rate and the ratio of the sentences intersections between the automatic and the reference extracts. Our comparisons with classical LSA and traditional statistical extractions showed that we reduced the use of the LSA procedure by 52%, and we obtained 65% reduction on the original matrix dimensions, also, we obtained remarkable accuracy results. It is concluded that the employment of the centrality feature with the proposed multi-layer framework yields a significant solution in terms of efficiency and accuracy in the field of text extraction