Evaluating the impact of information distortion on normalized compression distance

The final publication is available at Springer via http://dx.doi.org/10.1007/978-3-540-87448-5_8Proceedings of Second International Castle Meeting, ICMCTA 2008, Castillo de la Mota, Medina del Campo, Spain, September 15-19, 2008.In this paper we apply different techniques of information distortion on a set of classical books written in English. We study the impact that these distortions have upon the Kolmogorov complexity and the clustering by compression technique (the latter based on Normalized Compression Distance, NCD). We show how to decrease the complexity of the considered books introducing several modifications in them. We measure how the information contained in each book is maintained using a clustering error measure. We find experimentally that the best way to keep the clustering error is by means of modifications in the most frequent words. We explain the details of these information distortions and we compare with other kinds of modifications like random word distortions and unfrequent word distortions. Finally, some phenomenological explanations from the different empirical results that have been carried out are presented.This work was supported by TIN 2004-04363-CO03-03, TIN 2007-65989, CAM S-SEM-0255-2006, TIN2007-64718 and TSI 2005-08255-C07-06. We would also like to thank Franscico Sánchez for his useful comments on this draft