13,977 research outputs found
Ptolemaic Indexing
This paper discusses a new family of bounds for use in similarity search,
related to those used in metric indexing, but based on Ptolemy's inequality,
rather than the metric axioms. Ptolemy's inequality holds for the well-known
Euclidean distance, but is also shown here to hold for quadratic form metrics
in general, with Mahalanobis distance as an important special case. The
inequality is examined empirically on both synthetic and real-world data sets
and is also found to hold approximately, with a very low degree of error, for
important distances such as the angular pseudometric and several Lp norms.
Indexing experiments demonstrate a highly increased filtering power compared to
existing, triangular methods. It is also shown that combining the Ptolemaic and
triangular filtering can lead to better results than using either approach on
its own
Indexing Metric Spaces for Exact Similarity Search
With the continued digitalization of societal processes, we are seeing an
explosion in available data. This is referred to as big data. In a research
setting, three aspects of the data are often viewed as the main sources of
challenges when attempting to enable value creation from big data: volume,
velocity and variety. Many studies address volume or velocity, while much fewer
studies concern the variety. Metric space is ideal for addressing variety
because it can accommodate any type of data as long as its associated distance
notion satisfies the triangle inequality. To accelerate search in metric space,
a collection of indexing techniques for metric data have been proposed.
However, existing surveys each offers only a narrow coverage, and no
comprehensive empirical study of those techniques exists. We offer a survey of
all the existing metric indexes that can support exact similarity search, by i)
summarizing all the existing partitioning, pruning and validation techniques
used for metric indexes, ii) providing the time and storage complexity analysis
on the index construction, and iii) report on a comprehensive empirical
comparison of their similarity query processing performance. Here, empirical
comparisons are used to evaluate the index performance during search as it is
hard to see the complexity analysis differences on the similarity query
processing and the query performance depends on the pruning and validation
abilities related to the data distribution. This article aims at revealing
different strengths and weaknesses of different indexing techniques in order to
offer guidance on selecting an appropriate indexing technique for a given
setting, and directing the future research for metric indexes
Indexability, concentration, and VC theory
Degrading performance of indexing schemes for exact similarity search in high
dimensions has long since been linked to histograms of distributions of
distances and other 1-Lipschitz functions getting concentrated. We discuss this
observation in the framework of the phenomenon of concentration of measure on
the structures of high dimension and the Vapnik-Chervonenkis theory of
statistical learning.Comment: 17 pages, final submission to J. Discrete Algorithms (an expanded,
improved and corrected version of the SISAP'2010 invited paper, this e-print,
v3
HD-Index: Pushing the Scalability-Accuracy Boundary for Approximate kNN Search in High-Dimensional Spaces
Nearest neighbor searching of large databases in high-dimensional spaces is
inherently difficult due to the curse of dimensionality. A flavor of
approximation is, therefore, necessary to practically solve the problem of
nearest neighbor search. In this paper, we propose a novel yet simple indexing
scheme, HD-Index, to solve the problem of approximate k-nearest neighbor
queries in massive high-dimensional databases. HD-Index consists of a set of
novel hierarchical structures called RDB-trees built on Hilbert keys of
database objects. The leaves of the RDB-trees store distances of database
objects to reference objects, thereby allowing efficient pruning using distance
filters. In addition to triangular inequality, we also use Ptolemaic inequality
to produce better lower bounds. Experiments on massive (up to billion scale)
high-dimensional (up to 1000+) datasets show that HD-Index is effective,
efficient, and scalable.Comment: PVLDB 11(8):906-919, 201
A geometric framework for modelling similarity search
The aim of this paper is to propose a geometric framework for modelling
similarity search in large and multidimensional data spaces of general nature,
which seems to be flexible enough to address such issues as analysis of
complexity, indexability, and the `curse of dimensionality.' Such a framework
is provided by the concept of the so-called similarity workload, which is a
probability metric space (query domain) with a distinguished finite
subspace (dataset), together with an assembly of concepts, techniques, and
results from metric geometry. They include such notions as metric transform,
\e-entropy, and the phenomenon of concentration of measure on
high-dimensional structures. In particular, we discuss the relevance of the
latter to understanding the curse of dimensionality. As some of those concepts
and techniques are being currently reinvented by the database community, it
seems desirable to try and bridge the gap between database research and the
relevant work already done in geometry and analysis.Comment: 11 pages, LaTeX 2.
Efficient Document Indexing Using Pivot Tree
We present a novel method for efficiently searching top-k neighbors for
documents represented in high dimensional space of terms based on the cosine
similarity. Mostly, documents are stored as bag-of-words tf-idf representation.
One of the most used ways of computing similarity between a pair of documents
is cosine similarity between the vector representations, but cosine similarity
is not a metric distance measure as it doesn't follow triangle inequality,
therefore most metric searching methods can not be applied directly. We propose
an efficient method for indexing documents using a pivot tree that leads to
efficient retrieval. We also study the relation between precision and
efficiency for the proposed method and compare it with a state of the art in
the area of document searching based on inner product.Comment: 6 Pages, 2 Figure
- …