5,483 research outputs found
Real-time Text Queries with Tunable Term Pair Indexes
Term proximity scoring is an established means in information retrieval for improving result quality of full-text queries. Integrating such proximity scores into efficient query processing, however, has not been equally well studied. Existing methods make use of precomputed lists of documents where tuples of terms, usually pairs, occur together, usually incurring a huge index size compared to term-only indexes. This paper introduces a joint framework for trading off index size and result quality, and provides optimization techniques for tuning precomputed indexes towards either maximal result quality or maximal query processing performance, given an upper bound for the index size. The framework allows to selectively materialize lists for pairs based on a query log to further reduce index size. Extensive experiments with two large text collections demonstrate runtime improvements of several orders of magnitude over existing text-based processing techniques with reasonable index sizes
GraphMatch: Efficient Large-Scale Graph Construction for Structure from Motion
We present GraphMatch, an approximate yet efficient method for building the
matching graph for large-scale structure-from-motion (SfM) pipelines. Unlike
modern SfM pipelines that use vocabulary (Voc.) trees to quickly build the
matching graph and avoid a costly brute-force search of matching image pairs,
GraphMatch does not require an expensive offline pre-processing phase to
construct a Voc. tree. Instead, GraphMatch leverages two priors that can
predict which image pairs are likely to match, thereby making the matching
process for SfM much more efficient. The first is a score computed from the
distance between the Fisher vectors of any two images. The second prior is
based on the graph distance between vertices in the underlying matching graph.
GraphMatch combines these two priors into an iterative "sample-and-propagate"
scheme similar to the PatchMatch algorithm. Its sampling stage uses Fisher
similarity priors to guide the search for matching image pairs, while its
propagation stage explores neighbors of matched pairs to find new ones with a
high image similarity score. Our experiments show that GraphMatch finds the
most image pairs as compared to competing, approximate methods while at the
same time being the most efficient.Comment: Published at IEEE 3DV 201
Off the Beaten Path: Let's Replace Term-Based Retrieval with k-NN Search
Retrieval pipelines commonly rely on a term-based search to obtain candidate
records, which are subsequently re-ranked. Some candidates are missed by this
approach, e.g., due to a vocabulary mismatch. We address this issue by
replacing the term-based search with a generic k-NN retrieval algorithm, where
a similarity function can take into account subtle term associations. While an
exact brute-force k-NN search using this similarity function is slow, we
demonstrate that an approximate algorithm can be nearly two orders of magnitude
faster at the expense of only a small loss in accuracy. A retrieval pipeline
using an approximate k-NN search can be more effective and efficient than the
term-based pipeline. This opens up new possibilities for designing effective
retrieval pipelines. Our software (including data-generating code) and
derivative data based on the Stack Overflow collection is available online
Word Embeddings for Entity-annotated Texts
Learned vector representations of words are useful tools for many information
retrieval and natural language processing tasks due to their ability to capture
lexical semantics. However, while many such tasks involve or even rely on named
entities as central components, popular word embedding models have so far
failed to include entities as first-class citizens. While it seems intuitive
that annotating named entities in the training corpus should result in more
intelligent word features for downstream tasks, performance issues arise when
popular embedding approaches are naively applied to entity annotated corpora.
Not only are the resulting entity embeddings less useful than expected, but one
also finds that the performance of the non-entity word embeddings degrades in
comparison to those trained on the raw, unannotated corpus. In this paper, we
investigate approaches to jointly train word and entity embeddings on a large
corpus with automatically annotated and linked entities. We discuss two
distinct approaches to the generation of such embeddings, namely the training
of state-of-the-art embeddings on raw-text and annotated versions of the
corpus, as well as node embeddings of a co-occurrence graph representation of
the annotated corpus. We compare the performance of annotated embeddings and
classical word embeddings on a variety of word similarity, analogy, and
clustering evaluation tasks, and investigate their performance in
entity-specific tasks. Our findings show that it takes more than training
popular word embedding models on an annotated corpus to create entity
embeddings with acceptable performance on common test cases. Based on these
results, we discuss how and when node embeddings of the co-occurrence graph
representation of the text can restore the performance.Comment: This paper is accepted in 41st European Conference on Information
Retrieva
No-But-Semantic-Match: Computing Semantically Matched XML Keyword Search Results
Users are rarely familiar with the content of a data source they are
querying, and therefore cannot avoid using keywords that do not exist in the
data source. Traditional systems may respond with an empty result, causing
dissatisfaction, while the data source in effect holds semantically related
content. In this paper we study this no-but-semantic-match problem on XML
keyword search and propose a solution which enables us to present the top-k
semantically related results to the user. Our solution involves two steps: (a)
extracting semantically related candidate queries from the original query and
(b) processing candidate queries and retrieving the top-k semantically related
results. Candidate queries are generated by replacement of non-mapped keywords
with candidate keywords obtained from an ontological knowledge base. Candidate
results are scored using their cohesiveness and their similarity to the
original query. Since the number of queries to process can be large, with each
result having to be analyzed, we propose pruning techniques to retrieve the
top- results efficiently. We develop two query processing algorithms based
on our pruning techniques. Further, we exploit a property of the candidate
queries to propose a technique for processing multiple queries in batch, which
improves the performance substantially. Extensive experiments on two real
datasets verify the effectiveness and efficiency of the proposed approaches.Comment: 24 pages, 21 figures, 6 tables, submitted to The VLDB Journal for
possible publicatio
HD-Index: Pushing the Scalability-Accuracy Boundary for Approximate kNN Search in High-Dimensional Spaces
Nearest neighbor searching of large databases in high-dimensional spaces is
inherently difficult due to the curse of dimensionality. A flavor of
approximation is, therefore, necessary to practically solve the problem of
nearest neighbor search. In this paper, we propose a novel yet simple indexing
scheme, HD-Index, to solve the problem of approximate k-nearest neighbor
queries in massive high-dimensional databases. HD-Index consists of a set of
novel hierarchical structures called RDB-trees built on Hilbert keys of
database objects. The leaves of the RDB-trees store distances of database
objects to reference objects, thereby allowing efficient pruning using distance
filters. In addition to triangular inequality, we also use Ptolemaic inequality
to produce better lower bounds. Experiments on massive (up to billion scale)
high-dimensional (up to 1000+) datasets show that HD-Index is effective,
efficient, and scalable.Comment: PVLDB 11(8):906-919, 201
- …