32 research outputs found
Maximum Inner-Product Search using Tree Data-structures
The problem of {\em efficiently} finding the best match for a query in a
given set with respect to the Euclidean distance or the cosine similarity has
been extensively studied in literature. However, a closely related problem of
efficiently finding the best match with respect to the inner product has never
been explored in the general setting to the best of our knowledge. In this
paper we consider this general problem and contrast it with the existing
best-match algorithms. First, we propose a general branch-and-bound algorithm
using a tree data structure. Subsequently, we present a dual-tree algorithm for
the case where there are multiple queries. Finally we present a new data
structure for increasing the efficiency of the dual-tree algorithm. These
branch-and-bound algorithms involve novel bounds suited for the purpose of
best-matching with inner products. We evaluate our proposed algorithms on a
variety of data sets from various applications, and exhibit up to five orders
of magnitude improvement in query time over the naive search technique.Comment: Under submission in KDD 201
An ADMM Based Framework for AutoML Pipeline Configuration
We study the AutoML problem of automatically configuring machine learning
pipelines by jointly selecting algorithms and their appropriate
hyper-parameters for all steps in supervised learning pipelines. This black-box
(gradient-free) optimization with mixed integer & continuous variables is a
challenging problem. We propose a novel AutoML scheme by leveraging the
alternating direction method of multipliers (ADMM). The proposed framework is
able to (i) decompose the optimization problem into easier sub-problems that
have a reduced number of variables and circumvent the challenge of mixed
variable categories, and (ii) incorporate black-box constraints along-side the
black-box optimization objective. We empirically evaluate the flexibility (in
utilizing existing AutoML techniques), effectiveness (against open source
AutoML toolkits),and unique capability (of executing AutoML with practically
motivated black-box constraints) of our proposed scheme on a collection of
binary classification data sets from UCI ML& OpenML repositories. We observe
that on an average our framework provides significant gains in comparison to
other AutoML frameworks (Auto-sklearn & TPOT), highlighting the practical
advantages of this framework
Tree-Independent Dual-Tree Algorithms
Dual-tree algorithms are a widely used class of branch-and-bound algorithms.
Unfortunately, developing dual-tree algorithms for use with different trees and
problems is often complex and burdensome. We introduce a four-part logical
split: the tree, the traversal, the point-to-point base case, and the pruning
rule. We provide a meta-algorithm which allows development of dual-tree
algorithms in a tree-independent manner and easy extension to entirely new
types of trees. Representations are provided for five common algorithms; for
k-nearest neighbor search, this leads to a novel, tighter pruning bound. The
meta-algorithm also allows straightforward extensions to massively parallel
settings.Comment: accepted in ICML 201
Compositional Program Generation for Systematic Generalization
Compositional generalization is a key ability of humans that enables us to
learn new concepts from only a handful examples. Machine learning models,
including the now ubiquitous transformers, struggle to generalize in this way,
and typically require thousands of examples of a concept during training in
order to generalize meaningfully. This difference in ability between humans and
artificial neural architectures, motivates this study on a neuro-symbolic
architecture called the Compositional Program Generator (CPG). CPG has three
key features: modularity, type abstraction, and recursive composition, that
enable it to generalize both systematically to new concepts in a few-shot
manner, as well as productively by length on various sequence-to-sequence
language tasks. For each input, CPG uses a grammar of the input domain and a
parser to generate a type hierarchy in which each grammar rule is assigned its
own unique semantic module, a probabilistic copy or substitution program.
Instances with the same hierarchy are processed with the same composed program,
while those with different hierarchies may be processed with different
programs. CPG learns parameters for the semantic modules and is able to learn
the semantics for new types incrementally. Given a context-free grammar of the
input language and a dictionary mapping each word in the source language to its
interpretation in the output language, CPG can achieve perfect generalization
on the SCAN and COGS benchmarks, in both standard and extreme few-shot
settings.Comment: 7 pages of text with 1 page of reference
Model Sparsification Can Simplify Machine Unlearning
Recent data regulations necessitate machine unlearning (MU): The removal of
the effect of specific examples from the model. While exact unlearning is
possible by conducting a model retraining with the remaining data from scratch,
its computational cost has led to the development of approximate but efficient
unlearning schemes. Beyond data-centric MU solutions, we advance MU through a
novel model-based viewpoint: sparsification via weight pruning. Our results in
both theory and practice indicate that model sparsity can boost the
multi-criteria unlearning performance of an approximate unlearner, closing the
approximation gap, while continuing to be efficient. With this insight, we
develop two new sparsity-aware unlearning meta-schemes, termed `prune first,
then unlearn' and `sparsity-aware unlearning'. Extensive experiments show that
our findings and proposals consistently benefit MU in various scenarios,
including class-wise data scrubbing, random data scrubbing, and backdoor data
forgetting. One highlight is the 77% unlearning efficacy gain of fine-tuning
(one of the simplest approximate unlearning methods) in the proposed
sparsity-aware unlearning paradigm. Codes are available at
https://github.com/OPTML-Group/Unlearn-Sparse