22 research outputs found
Combined Data Structure for Previous- and Next-Smaller-Values
Let be a static array storing elements from a totally ordered set. We
present a data structure of optimal size at most
bits that allows us to answer the following queries on in constant time,
without accessing : (1) previous smaller value queries, where given an index
, we wish to find the first index to the left of where is strictly
smaller than at , and (2) next smaller value queries, which search to the
right of . As an additional bonus, our data structure also allows to answer
a third kind of query: given indices , find the position of the minimum in
. Our data structure has direct consequences for the space-efficient
storage of suffix trees.Comment: to appear in Theoretical Computer Scienc
Lightweight Lempel-Ziv Parsing
We introduce a new approach to LZ77 factorization that uses O(n/d) words of
working space and O(dn) time for any d >= 1 (for polylogarithmic alphabet
sizes). We also describe carefully engineered implementations of alternative
approaches to lightweight LZ77 factorization. Extensive experiments show that
the new algorithm is superior in most cases, particularly at the lowest memory
levels and for highly repetitive data. As a part of the algorithm, we describe
new methods for computing matching statistics which may be of independent
interest.Comment: 12 page
Rpair: Rescaling RePair with Rsync
Data compression is a powerful tool for managing massive but repetitive datasets, especially schemes such as grammar-based compression that support computation over the data without decompressing it. In the best case such a scheme takes a dataset so big that it must be stored on disk and shrinks it enough that it can be stored and processed in internal memory. Even then, however, the scheme is essentially useless unless it can be built on the original dataset reasonably quickly while keeping the dataset on disk. In this paper we show how we can preprocess such datasets with context-triggered piecewise hashing such that afterwards we can apply RePair and other grammar-based compressors more easily. We first give our algorithm, then show how a variant of it can be used to approximate the LZ77 parse, then leverage that to prove theoretical bounds on compression, and finally give experimental evidence that our approach is competitive in practice
Indexing Finite Language Representation of Population Genotypes
With the recent advances in DNA sequencing, it is now possible to have
complete genomes of individuals sequenced and assembled. This rich and focused
genotype information can be used to do different population-wide studies, now
first time directly on whole genome level. We propose a way to index population
genotype information together with the complete genome sequence, so that one
can use the index to efficiently align a given sequence to the genome with all
plausible genotype recombinations taken into account. This is achieved through
converting a multiple alignment of individual genomes into a finite automaton
recognizing all strings that can be read from the alignment by switching the
sequence at any time. The finite automaton is indexed with an extension of
Burrows-Wheeler transform to allow pattern search inside the plausible
recombinant sequences. The size of the index stays limited, because of the high
similarity of individual genomes. The index finds applications in variation
calling and in primer design. On a variation calling experiment, we found about
1.0% of matches to novel recombinants just with exact matching, and up to 2.4%
with approximate matching.Comment: This is the full version of the paper that was presented at WABI
2011. The implementation is available at
http://www.cs.helsinki.fi/group/suds/gcsa
Suffix Sorting via Matching Statistics
Funding Information: Academy of Finland grants 339070 and 351150 Publisher Copyright: © Zsuzsanna Lipták, Francesco Masillo, and Simon J. Puglisi.We introduce a new algorithm for constructing the generalized suffix array of a collection of highly similar strings. As a first step, we construct a compressed representation of the matching statistics of the collection with respect to a reference string. We then use this data structure to distribute suffixes into a partial order, and subsequently to speed up suffix comparisons to complete the generalized suffix array. Our experimental evidence with a prototype implementation (a tool we call sacamats) shows that on string collections with highly similar strings we can construct the suffix array in time competitive with or faster than the fastest available methods. Along the way, we describe a heuristic for fast computation of the matching statistics of two strings, which may be of independent interest.Peer reviewe