50 research outputs found
Learning in the Real World: Constraints on Cost, Space, and Privacy
The sheer demand for machine learning in fields as varied as: healthcare, web-search ranking, factory automation, collision prediction, spam filtering, and many others, frequently outpaces the intended use-case of machine learning models. In fact, a growing number of companies hire machine learning researchers to rectify this very problem: to tailor and/or design new state-of-the-art models to the setting at hand.
However, we can generalize a large set of the machine learning problems encountered in practical settings into three categories: cost, space, and privacy. The first category (cost) considers problems that need to balance the accuracy of a machine learning model with the cost required to evaluate it. These include problems in web-search, where results need to be delivered to a user in under a second and be as accurate as possible. The second category (space) collects problems that require running machine learning algorithms on low-memory computing devices. For instance, in search-and-rescue operations we may opt to use many small unmanned aerial vehicles (UAVs) equipped with machine learning algorithms for object detection to find a desired search target. These algorithms should be small to fit within the physical memory limits of the UAV (and be energy efficient) while reliably detecting objects. The third category (privacy) considers problems where one wishes to run machine learning algorithms on sensitive data. It has been shown that seemingly innocuous analyses on such data can be exploited to reveal data individuals would prefer to keep private. Thus, nearly any algorithm that runs on patient or economic data falls under this set of problems.
We devise solutions for each of these problem categories including (i) a fast tree-based model for explicitly trading off accuracy and model evaluation time, (ii) a compression method for the k-nearest neighbor classifier, and (iii) a private causal inference algorithm that protects sensitive data
Grammar Variational Autoencoder
Deep generative models have been wildly successful at learning coherent
latent representations for continuous data such as video and audio. However,
generative modeling of discrete data such as arithmetic expressions and
molecular structures still poses significant challenges. Crucially,
state-of-the-art methods often produce outputs that are not valid. We make the
key observation that frequently, discrete data can be represented as a parse
tree from a context-free grammar. We propose a variational autoencoder which
encodes and decodes directly to and from these parse trees, ensuring the
generated outputs are always valid. Surprisingly, we show that not only does
our model more often generate valid outputs, it also learns a more coherent
latent space in which nearby points decode to similar discrete outputs. We
demonstrate the effectiveness of our learned models by showing their improved
performance in Bayesian optimization for symbolic regression and molecular
synthesis
Learning Binary Decision Trees by Argmin Differentiation
We address the problem of learning binary decision trees that partition data
for some downstream task. We propose to learn discrete parameters (i.e., for
tree traversals and node pruning) and continuous parameters (i.e., for tree
split functions and prediction functions) simultaneously using argmin
differentiation. We do so by sparsely relaxing a mixed-integer program for the
discrete parameters, to allow gradients to pass through the program to
continuous parameters. We derive customized algorithms to efficiently compute
the forward and backward passes. This means that our tree learning procedure
can be used as an (implicit) layer in arbitrary deep networks, and can be
optimized with arbitrary loss functions. We demonstrate that our approach
produces binary trees that are competitive with existing single tree and
ensemble approaches, in both supervised and unsupervised settings. Further,
apart from greedy approaches (which do not have competitive accuracies), our
method is faster to train than all other tree-learning baselines we compare
with. The code for reproducing the results is available at
https://github.com/vzantedeschi/LatentTrees
Learning Binary Decision Trees by Argmin Differentiation
We address the problem of learning binary decision trees that partition data
for some downstream task. We propose to learn discrete parameters (i.e., for
tree traversals and node pruning) and continuous parameters (i.e., for tree
split functions and prediction functions) simultaneously using argmin
differentiation. We do so by sparsely relaxing a mixed-integer program for the
discrete parameters, to allow gradients to pass through the program to
continuous parameters. We derive customized algorithms to efficiently compute
the forward and backward passes. This means that our tree learning procedure
can be used as an (implicit) layer in arbitrary deep networks, and can be
optimized with arbitrary loss functions. We demonstrate that our approach
produces binary trees that are competitive with existing single tree and
ensemble approaches, in both supervised and unsupervised settings. Further,
apart from greedy approaches (which do not have competitive accuracies), our
method is faster to train than all other tree-learning baselines we compare
with. The code for reproducing the results is available at
https://github.com/vzantedeschi/LatentTrees
Personalized Purchase Prediction of Market Baskets with Wasserstein-Based Sequence Matching
Personalization in marketing aims at improving the shopping experience of
customers by tailoring services to individuals. In order to achieve this,
businesses must be able to make personalized predictions regarding the next
purchase. That is, one must forecast the exact list of items that will comprise
the next purchase, i.e., the so-called market basket. Despite its relevance to
firm operations, this problem has received surprisingly little attention in
prior research, largely due to its inherent complexity. In fact,
state-of-the-art approaches are limited to intuitive decision rules for pattern
extraction. However, the simplicity of the pre-coded rules impedes performance,
since decision rules operate in an autoregressive fashion: the rules can only
make inferences from past purchases of a single customer without taking into
account the knowledge transfer that takes place between customers. In contrast,
our research overcomes the limitations of pre-set rules by contributing a novel
predictor of market baskets from sequential purchase histories: our predictions
are based on similarity matching in order to identify similar purchase habits
among the complete shopping histories of all customers. Our contributions are
as follows: (1) We propose similarity matching based on subsequential dynamic
time warping (SDTW) as a novel predictor of market baskets. Thereby, we can
effectively identify cross-customer patterns. (2) We leverage the Wasserstein
distance for measuring the similarity among embedded purchase histories. (3) We
develop a fast approximation algorithm for computing a lower bound of the
Wasserstein distance in our setting. An extensive series of computational
experiments demonstrates the effectiveness of our approach. The accuracy of
identifying the exact market baskets based on state-of-the-art decision rules
from the literature is outperformed by a factor of 4.0.Comment: Accepted for oral presentation at 25th ACM SIGKDD Conference on
Knowledge Discovery and Data Mining (KDD 2019
When Do Flat Minima Optimizers Work?
Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic and adaptive gradient-based optimizers. Two methods have received significant attention due to their scalability: 1. Stochastic Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem