Performance of Data Structures for Small Sets of Strings

Fundamental structures such as trees and hash tables are used for managing data in a huge variety of circumstances. Making the right choice of structure is essential to efficiency. In previous work we have explored the performance of a range of data structures -- different forms of trees tries and hash tables -- for the task of managing sets of millions of strings and have developed new variants of each that are more efficient for this task than previous alternatives. In this paper we test the performance of the same data structures on small sets of strings in the context of document processing for index construction. Our results show that the new structures in particular our burst trie are the most efficient choice for this task thus demonstrating that they are suitable for managing sets of hundreds to millions of distinct strings and for input of hundreds to billions of occurrences