An empirical study of feature selection for text categorization based on term weightage

This paper proposes a local feature selection (FS) measure namely, Categorical Descriptor Term (CTD) for text categorization. It is derived based on classic term weighting scheme, TFIDF. The method explicitly chooses feature set for each category by only selecting set of terms from relevant category. Although past literatures have suggested that the use of features from irrelevant categories can improve the measure of text categorization, we believe that by incorporating only relevant feature can be highly effective. The experimental comparison is carried out between CTD and five wellknown feature selection measures: Information Gain, Chi-Square, Correlation Coefficient, Odd Ratio and GSS Coefficient. The results also show that our proposed method can perform comparatively well with other FS measures, especially on collection with highly overlapped topics

An empirical study on CO2 emissions in ASEAN countries

This paper proposes a local feature selection (FS) measure namely, Categorical Descriptor Term (CTD) for text categorization. It is derived based on classic term weighting scheme, TFIDF. The method explicitly chooses feature set for each category by only selecting set of terms from relevant category. Although past literatures have suggested that the use of features from irrelevant categories can improve the measure of text categorization, we believe that by incorporating only relevant feature can be highly effective. The experimental comparison is carried out between CTD and five wellknown feature selection measures: Information Gain, Chi-Square, Correlation Coefficient, Odd Ratio and GSS Coefficient. The results also show that our proposed method can perform comparatively well with other FS measures, especially on collection with highly overlapped topics

Nomenclature and Benchmarking Models of Text Classification Models: Contemporary Affirmation of the Recent Literature

In this paper we present automated text classification in text mining that is gaining greater relevance in various fields every day Text mining primarily focuses on developing text classification systems able to automatically classify huge volume of documents comprising of unstructured and semi structured data The process of retrieval classification and summarization simplifies extract of information by the user The finding of the ideal text classifier feature generator and distinct dominant technique of feature selection leading all other previous research has received attention from researchers of diverse areas as information retrieval machine learning and the theory of algorithms To automatically classify and discover patterns from the different types of the documents 1 techniques like Machine Learning Natural Language Processing NLP and Data Mining are applied together In this paper we review some effective feature selection researches and show the results in a table for

Classificação de Documentos

Dissertação apresentada na Faculdade de Ciências e Tecnologia da Universidade No Lisboa para obtenção de grau de Mestre em Engenharia de InformáticaNo presente trabalho de investigação pretende-se automatizar o processo de classificação temática de documentos. Foram utilizadas três técnicas de selecção de termos, com três classificadores automáticos, e sete representações de documentos: palavra, multi-palavra, pentagrama, e cadeias dos primeiros 4, 5 e 6 caracteres individualmente, e globalmente. Entre as técnicas de selecção de termos encontra-se a medida do Terceiro Momento em relação à média. Esta medida foi recentemente proposta, por o Professor Joaquim Ferreira da Silva, e considerou-se importante realizar um estudo comparativo da sua performance em relação a outras medidas, já muito conhecidas e comprovada a sua aplicabilidade. As medidas escolhidas foram: Chi-Square e Information Gain. Existem medidas de selecção de termos que demonstram melhores resultados conforme o classificador utilizado, e por isso, as medidas foram experimentadas com diferentes classificadores: K-Nearest Neighbour, RIPPER e Support Vector Machines. São classificadores que na área de classificação demonstraram bons resultados, e assim, avaliou-se o seu desempenho com as diferentes medidas de selecção de termos. Nos resultados experimentais, em que foi utilizado o corpus da Reuters-21578, pode-se observar que o desempenho obtido com a técnica do terceiro momento é superior, ou equivalente, à obtida com as medidas de selecção de termos Chi-Square e Information Gain. Utilizando diferentes representações de documentos é possível obter um desempenho, com os três classificadores, equivalente ao obtido com a representação de documentos por palavra