21,054 research outputs found
Differentiable Kernels in Generalized Matrix Learning Vector Quantization
In the present paper we investigate the application of differentiable kernel for generalized matrix learning vector quantization as an alternative kernel-based classifier, which additionally provides classification dependent data visualization. We show that the concept of differentiable kernels allows a prototype description in the data space but equipped with the kernel metric. Moreover, using the visualization properties of the original matrix learning vector quantization we are able to optimize the class visualization by inherent visualization mapping learning also in this new kernel-metric data space
S-TREE: Self-Organizing Trees for Data Clustering and Online Vector Quantization
This paper introduces S-TREE (Self-Organizing Tree), a family of models that use unsupervised learning to construct hierarchical representations of data and online tree-structured vector quantizers. The S-TREE1 model, which features a new tree-building algorithm, can be implemented with various cost functions. An alternative implementation, S-TREE2, which uses a new double-path search procedure, is also developed. S-TREE2 implements an online procedure that approximates an optimal (unstructured) clustering solution while imposing a tree-structure constraint. The performance of the S-TREE algorithms is illustrated with data clustering and vector quantization examples, including a Gauss-Markov source benchmark and an image compression application. S-TREE performance on these tasks is compared with the standard tree-structured vector quantizer (TSVQ) and the generalized Lloyd algorithm (GLA). The image reconstruction quality with S-TREE2 approaches that of GLA while taking less than 10% of computer time. S-TREE1 and S-TREE2 also compare favorably with the standard TSVQ in both the time needed to create the codebook and the quality of image reconstruction.Office of Naval Research (N00014-95-10409, N00014-95-0G57
ProxQuant: Quantized Neural Networks via Proximal Operators
To make deep neural networks feasible in resource-constrained environments
(such as mobile devices), it is beneficial to quantize models by using
low-precision weights. One common technique for quantizing neural networks is
the straight-through gradient method, which enables back-propagation through
the quantization mapping. Despite its empirical success, little is understood
about why the straight-through gradient method works.
Building upon a novel observation that the straight-through gradient method
is in fact identical to the well-known Nesterov's dual-averaging algorithm on a
quantization constrained optimization problem, we propose a more principled
alternative approach, called ProxQuant, that formulates quantized network
training as a regularized learning problem instead and optimizes it via the
prox-gradient method. ProxQuant does back-propagation on the underlying
full-precision vector and applies an efficient prox-operator in between
stochastic gradient steps to encourage quantizedness. For quantizing ResNets
and LSTMs, ProxQuant outperforms state-of-the-art results on binary
quantization and is on par with state-of-the-art on multi-bit quantization. For
binary quantization, our analysis shows both theoretically and experimentally
that ProxQuant is more stable than the straight-through gradient method (i.e.
BinaryConnect), challenging the indispensability of the straight-through
gradient method and providing a powerful alternative
Hashing for Similarity Search: A Survey
Similarity search (nearest neighbor search) is a problem of pursuing the data
items whose distances to a query item are the smallest from a large database.
Various methods have been developed to address this problem, and recently a lot
of efforts have been devoted to approximate search. In this paper, we present a
survey on one of the main solutions, hashing, which has been widely studied
since the pioneering work locality sensitive hashing. We divide the hashing
algorithms two main categories: locality sensitive hashing, which designs hash
functions without exploring the data distribution and learning to hash, which
learns hash functions according the data distribution, and review them from
various aspects, including hash function design and distance measure and search
scheme in the hash coding space
- …