12,508 research outputs found
Re-Pair Compression of Inverted Lists
Compression of inverted lists with methods that support fast intersection
operations is an active research topic. Most compression schemes rely on
encoding differences between consecutive positions with techniques that favor
small numbers. In this paper we explore a completely different alternative: We
use Re-Pair compression of those differences. While Re-Pair by itself offers
fast decompression at arbitrary positions in main and secondary memory, we
introduce variants that in addition speed up the operations required for
inverted list intersection. We compare the resulting data structures with
several recent proposals under various list intersection algorithms, to
conclude that our Re-Pair variants offer an interesting time/space tradeoff for
this problem, yet further improvements are required for it to improve upon the
state of the art
Efficient and Effective Query Auto-Completion
Query Auto-Completion (QAC) is an ubiquitous feature of modern textual search
systems, suggesting possible ways of completing the query being typed by the
user. Efficiency is crucial to make the system have a real-time responsiveness
when operating in the million-scale search space. Prior work has extensively
advocated the use of a trie data structure for fast prefix-search operations in
compact space. However, searching by prefix has little discovery power in that
only completions that are prefixed by the query are returned. This may impact
negatively the effectiveness of the QAC system, with a consequent monetary loss
for real applications like Web Search Engines and eCommerce. In this work we
describe the implementation that empowers a new QAC system at eBay, and discuss
its efficiency/effectiveness in relation to other approaches at the
state-of-the-art. The solution is based on the combination of an inverted index
with succinct data structures, a much less explored direction in the
literature. This system is replacing the previous implementation based on
Apache SOLR that was not always able to meet the required
service-level-agreement.Comment: Published in SIGIR 202
Universal Indexes for Highly Repetitive Document Collections
Indexing highly repetitive collections has become a relevant problem with the
emergence of large repositories of versioned documents, among other
applications. These collections may reach huge sizes, but are formed mostly of
documents that are near-copies of others. Traditional techniques for indexing
these collections fail to properly exploit their regularities in order to
reduce space.
We introduce new techniques for compressing inverted indexes that exploit
this near-copy regularity. They are based on run-length, Lempel-Ziv, or grammar
compression of the differential inverted lists, instead of the usual practice
of gap-encoding them. We show that, in this highly repetitive setting, our
compression methods significantly reduce the space obtained with classical
techniques, at the price of moderate slowdowns. Moreover, our best methods are
universal, that is, they do not need to know the versioning structure of the
collection, nor that a clear versioning structure even exists.
We also introduce compressed self-indexes in the comparison. These are
designed for general strings (not only natural language texts) and represent
the text collection plus the index structure (not an inverted index) in
integrated form. We show that these techniques can compress much further, using
a small fraction of the space required by our new inverted indexes. Yet, they
are orders of magnitude slower.Comment: This research has received funding from the European Union's Horizon
2020 research and innovation programme under the Marie Sk{\l}odowska-Curie
Actions H2020-MSCA-RISE-2015 BIRDS GA No. 69094
Worst-Case Efficient Sorting with QuickMergesort
The two most prominent solutions for the sorting problem are Quicksort and
Mergesort. While Quicksort is very fast on average, Mergesort additionally
gives worst-case guarantees, but needs extra space for a linear number of
elements. Worst-case efficient in-place sorting, however, remains a challenge:
the standard solution, Heapsort, suffers from a bad cache behavior and is also
not overly fast for in-cache instances.
In this work we present median-of-medians QuickMergesort (MoMQuickMergesort),
a new variant of QuickMergesort, which combines Quicksort with Mergesort
allowing the latter to be implemented in place. Our new variant applies the
median-of-medians algorithm for selecting pivots in order to circumvent the
quadratic worst case. Indeed, we show that it uses at most
comparisons for large enough.
We experimentally confirm the theoretical estimates and show that the new
algorithm outperforms Heapsort by far and is only around 10% slower than
Introsort (std::sort implementation of stdlibc++), which has a rather poor
guarantee for the worst case. We also simulate the worst case, which is only
around 10% slower than the average case. In particular, the new algorithm is a
natural candidate to replace Heapsort as a worst-case stopper in Introsort
Fast Parallel Operations on Search Trees
Using (a,b)-trees as an example, we show how to perform a parallel split with
logarithmic latency and parallel join, bulk updates, intersection, union (or
merge), and (symmetric) set difference with logarithmic latency and with
information theoretically optimal work. We present both asymptotically optimal
solutions and simplified versions that perform well in practice - they are
several times faster than previous implementations
Efficient Genomic Interval Queries Using Augmented Range Trees
Efficient large-scale annotation of genomic intervals is essential for
personal genome interpretation in the realm of precision medicine. There are 13
possible relations between two intervals according to Allen's interval algebra.
Conventional interval trees are routinely used to identify the genomic
intervals satisfying a coarse relation with a query interval, but cannot
support efficient query for more refined relations such as all Allen's
relations. We design and implement a novel approach to address this unmet need.
Through rewriting Allen's interval relations, we transform an interval query to
a range query, then adapt and utilize the range trees for querying. We
implement two types of range trees: a basic 2-dimensional range tree (2D-RT)
and an augmented range tree with fractional cascading (RTFC) and compare them
with the conventional interval tree (IT). Theoretical analysis shows that RTFC
can achieve the best time complexity for interval queries regarding all Allen's
relations among the three trees. We also perform comparative experiments on the
efficiency of RTFC, 2D-RT and IT in querying noncoding element annotations in a
large collection of personal genomes. Our experimental results show that 2D-RT
is more efficient than IT for interval queries regarding most of Allen's
relations, RTFC is even more efficient than 2D-RT. The results demonstrate that
RTFC is an efficient data structure for querying large-scale datasets regarding
Allen's relations between genomic intervals, such as those required by
interpreting genome-wide variation in large populations.Comment: 4 figures, 4 table
- …