Efficient Vector Quantization for Fast Approximate Nearest Neighbor Search

Increasing sizes of databases and data stores mean that the traditional tasks, such as locating a nearest neighbor for a given data point, become too complex for classical solutions to handle. Exact solutions have been shown to scale poorly with dimensionality of the data. Approximate nearest neighbor search (ANN) is a practical compromise between accuracy and performance; it is widely applicable and is a subject of much research. Amongst a number of ANN approaches suggested in the recent years, the ones based on vector quantization stand out, achieving state-of-the-art results. Product quantization (PQ) decomposes vectors into subspaces for separate processing, allowing for fast lookup-based distance calculations. Additive quantization (AQ) drops most of PQ constraints, currently providing the best search accuracy on image descriptor datasets, but at a higher computational cost. This thesis work aims to reduce the complexity of AQ by changing a single most expensive step in the process – that of vector encoding. Both the outstanding search performance and high costs of AQ come from its generality, therefore by imposing some novel external constraints it is possible to achieve a better compromise: reduce complexity while retaining the accuracy advantage over other ANN methods. We propose a new encoding method for AQ – pyramid encoding. It requires significantly less calculations compared to the original “beam search” encoding, at the cost of an increased greediness of the optimization procedure. As its performance depends heavily on the initialization, the problem of choosing a starting point is also discussed. The results achieved by applying the proposed method are compared with the current state-of-the-art on two widely used benchmark datasets – GIST1M and SIFT1M, both generated from a real-world image data and therefore closely modeling practical applications. AQ with pyramid encoding, in addition to its computational benefits, is shown to achieve similar or better search performance than competing methods. However, its current advantages seem to be limited to data of a certain internal structure. Further analysis of this drawback provides us with the directions of possible future work

Neural Architecture Search by Estimation of Network Structure Distributions

The influence of deep learning is continuously expanding across different domains, and its new applications are ubiquitous. The question of neural network design thus increases in importance, as traditional empirical approaches are reaching their limits. Manual design of network architectures from scratch relies heavily on trial and error, while using existing pretrained models can introduce redundancies or vulnerabilities. Automated neural architecture design is able to overcome these problems, but the most successful algorithms operate on significantly constrained design spaces, assuming the target network to consist of identical repeating blocks. While such approach allows for faster search, it does so at the cost of expressivity. We instead propose an alternative probabilistic representation of a whole neural network structure under the assumption of independence between layer types. Our matrix of probabilities is equivalent to the population of models, but allows for discovery of structural irregularities, while being simple to interpret and analyze. We construct an architecture search algorithm, inspired by the estimation of distribution algorithms, to take advantage of this representation. The probability matrix is tuned towards generating high-performance models by repeatedly sampling the architectures and evaluating the corresponding networks, while gradually increasing the model depth. Our algorithm is shown to discover non-regular models which cannot be expressed via blocks, but are competitive both in accuracy and computational cost, while not utilizing complex dataflows or advanced training techniques, as well as remaining conceptually simple and highly extensible.Comment: 16 pages, 4 figures, 3 table

Switching ion binding selectivity of thiacalix[4]arene monocrowns at liquid–liquid and 2D-confined interfaces

Understanding the interaction of ions with organic receptors in confined space is of fundamental importance and could advance nanoelectronics and sensor design. In this work, metal ion complexation of conformationally varied thiacalix[4]monocrowns bearing lower-rim hydroxy (type I), dodecyloxy (type II), or methoxy (type III) fragments was evaluated. At the liquid–liquid interface, alkylated thiacalixcrowns-5(6) selectively extract alkali metal ions according to the induced-fit concept, whereas crown-4 receptors were ineffective due to distortion of the crown-ether cavity, as predicted by quantum-chemical calculations. In type-I ligands, alkali-metal ion extraction by the solvent-accessible crown-ether cavity was prevented, which resulted in competitive Ag+ extraction by sulfide bridges. Surprisingly, amphiphilic type-I/II conjugates moderately extracted other metal ions, which was attributed to calixarene aggregation in salt aqueous phase and supported by dynamic light scattering measurements. Cation–monolayer interactions at the air–water interface were monitored by surface pressure/potential measurements and UV/visible reflection–absorption spectroscopy. Topology-varied selectivity was evidenced, towards Sr2+ (crown-4), K+ (crown-5), and Ag+ (crown-6) in type-I receptors and Na+ (crown-4), Ca2+ (crown-5), and Cs+ (crown-6) in type-II receptors. Nuclear magnetic resonance and electronic absorption spectroscopy revealed exocyclic coordination in type-I ligands and cation–π interactions in type-II ligands

Switching Ion Binding Selectivity of Thiacalix[4]arene Monocrowns at Liquid–Liquid and 2D-Confined Interfaces

Understanding the interaction of ions with organic receptors in confined space is of fundamental importance and could advance nanoelectronics and sensor design. In this work, metal ion complexation of conformationally varied thiacalix[4]monocrowns bearing lower-rim hydroxy (type I), dodecyloxy (type II), or methoxy (type III) fragments was evaluated. At the liquid–liquid interface, alkylated thiacalixcrowns-5(6) selectively extract alkali metal ions according to the induced-fit concept, whereas crown-4 receptors were ineffective due to distortion of the crown-ether cavity, as predicted by quantum-chemical calculations. In type-I ligands, alkali-metal ion extraction by the solvent-accessible crown-ether cavity was prevented, which resulted in competitive Ag+ extraction by sulfide bridges. Surprisingly, amphiphilic type-I/II conjugates moderately extracted other metal ions, which was attributed to calixarene aggregation in salt aqueous phase and supported by dynamic light scattering measurements. Cation–monolayer interactions at the air–water interface were monitored by surface pressure/potential measurements and UV/visible reflection–absorption spectroscopy. Topology-varied selectivity was evidenced, towards Sr2+ (crown-4), K+ (crown-5), and Ag+ (crown-6) in type-I receptors and Na+ (crown-4), Ca2+ (crown-5), and Cs+ (crown-6) in type-II receptors. Nuclear magnetic resonance and electronic absorption spectroscopy revealed exocyclic coordination in type-I ligands and cation–π interactions in type-II ligands

Efficient Vector Quantization for Fast Approximate Nearest Neighbor Search

Increasing sizes of databases and data stores mean that the traditional tasks, such as locating a nearest neighbor for a given data point, become too complex for classical solutions to handle. Exact solutions have been shown to scale poorly with dimensionality of the data. Approximate nearest neighbor search (ANN) is a practical compromise between accuracy and performance; it is widely applicable and is a subject of much research. Amongst a number of ANN approaches suggested in the recent years, the ones based on vector quantization stand out, achieving state-of-the-art results. Product quantization (PQ) decomposes vectors into subspaces for separate processing, allowing for fast lookup-based distance calculations. Additive quantization (AQ) drops most of PQ constraints, currently providing the best search accuracy on image descriptor datasets, but at a higher computational cost. This thesis work aims to reduce the complexity of AQ by changing a single most expensive step in the process – that of vector encoding. Both the outstanding search performance and high costs of AQ come from its generality, therefore by imposing some novel external constraints it is possible to achieve a better compromise: reduce complexity while retaining the accuracy advantage over other ANN methods. We propose a new encoding method for AQ – pyramid encoding. It requires significantly less calculations compared to the original “beam search” encoding, at the cost of an increased greediness of the optimization procedure. As its performance depends heavily on the initialization, the problem of choosing a starting point is also discussed. The results achieved by applying the proposed method are compared with the current state-of-the-art on two widely used benchmark datasets – GIST1M and SIFT1M, both generated from a real-world image data and therefore closely modeling practical applications. AQ with pyramid encoding, in addition to its computational benefits, is shown to achieve similar or better search performance than competing methods. However, its current advantages seem to be limited to data of a certain internal structure. Further analysis of this drawback provides us with the directions of possible future work

On the Layer Selection in Small-Scale Deep Networks

Deep learning algorithms (in particular Convolutional Neural Networks, or CNNs) have shown their superiority in computer vision tasks and continue to push the state of the art in the most difficult problems of the field. However, deep models frequently lack interpretability. Current research efforts are often focused on increasingly complex and computationally expensive structures. These can be either handcrafted or generated by an algorithm, but in either case the specific choices of individual structural elements are hard to justify. This paper aims to analyze statistical properties of a large sample of small deep networks, where the choice of layer types is randomized. The limited representational power of such models forces them to specialize rather than generalize, resulting in several distinct structural patterns. Observing the empirical performance of structurally diverse weaker models thus allows for some practical insight into the connection between the data and the choice of suitable CNN architectures.acceptedVersionPeer reviewe