A Cooperative Distributed Text Database Management Method Unifying Search and Compression Based on the Burrows-Wheeler Transformation

A new text database management method for distributed cooperative environments is proposed, which can collect texts in distributed sites through a network of narrow bandwidth and enables fulltext search in a unified efficient manner. This method is based on the two new developments in full-text search data structures and data compression. Specifically, the Burrows-Wheeler transformation is used as a basis of constructing the suffix array (or, PAT array) for full-text search and of performing the block sorting compression scheme. A cooperative environment makes it possible to employ these new methods in a uniform fashion. This framework may be also used in future for the Web text collection/search problem. The paper first describes this method, and then provides preliminary computational results concerning I/O implementation of suffix arrays and performing the suffix sorting. These preliminary computational results indicate practicality of our method