Csorba Kristóf

Vajk István

University of Szeged

Probabilistic confidence prediction in document clusteringKristóf Csorba and István VajkIn this paper we propose a novel technique to predict the confidence of the clustering ofa document before performing the clustering procedure itself. If an application has time lim-itations for the document clustering and it has to cluster as much documents as possible, itcan use the predicted confidence value to sort the documents and process the ones with highexpected confidence first. This means that the system begins with the documents for whichthe probability of a certain result is high. If it has enough time, it can process the remainingdocuments as well, but only after the most beneficial ones. The base method consists of threekey techniques of the feature-space based document clustering: term frequency quantizing,singular value decomposition and double clustering.Term frequency quantizing removes the unnecessary variation of the term frequencies. Thisstep was introduced through the idea, that the exact occurrence number of a frequent term in adocument does not provide more information than a "frequent" status.Singular value decomposition was already shown to be capable to capture semantic relation-ships between terms based on occurrence behavior similarities. SVD is used to reduce thefeature space dimensionality before the clustering itself.Double clustering means, that instead of clustering the feature vectors of the documents, thefeature vectors of the individual terms are clustered to create termclusters first. Documentclustering is then performed in a feature-space based on these term-clusters.After these steps the document clustering is performed in the resulting feature-space byemploying the k-means algorithm using the cosine distance measure. During the clusteringbeside the assigned cluster an additional confidence value is provided for every document tomark ambiguous cases.Before the procedure above a probabilistic model of it can be employed to calculate the ex-pected confidence value. Before employing, this prediction method uses an initial training phase:after executing the original clustering method on a training set, the internal state is analyzed toretrieve probability of internal decisions. After the training the predictions for the further docu-ments are calculated with a single matrix multiplication. This prediction is faster, but thereforeless accurate. The predicted results can be used to reorder the documents to process the easiercases first which helps timecritical applications to get more sure results in the available time.In this paper we provide a description of the prediction method and experimental resultsin connection with absolute performance and about the correlation between the predicted andmeasured results.34

Probabilistic confidence prediction in document clustering : [abstract]

http://acta.bibl.u-szeged.hu/59842/1/cscs_2006_034.pdf

Probabilistic confidence prediction in document clustering

Abstract

Similar works

Full text

Available Versions

University of Szeged