1,398 research outputs found
Learning Deep Disentangled Embeddings with the F-Statistic Loss
Deep-embedding methods aim to discover representations of a domain that make
explicit the domain's class structure and thereby support few-shot learning.
Disentangling methods aim to make explicit compositional or factorial
structure. We combine these two active but independent lines of research and
propose a new paradigm suitable for both goals. We propose and evaluate a novel
loss function based on the statistic, which describes the separation of two
or more distributions. By ensuring that distinct classes are well separated on
a subset of embedding dimensions, we obtain embeddings that are useful for
few-shot learning. By not requiring separation on all dimensions, we encourage
the discovery of disentangled representations. Our embedding method matches or
beats state-of-the-art, as evaluated by performance on recall@ and few-shot
learning tasks. Our method also obtains performance superior to a variety of
alternatives on disentangling, as evaluated by two key properties of a
disentangled representation: modularity and explicitness. The goal of our work
is to obtain more interpretable, manipulable, and generalizable deep
representations of concepts and categories
Open-Ended Content-Style Recombination Via Leakage Filtering
We consider visual domains in which a class label specifies the content of an
image, and class-irrelevant properties that differentiate instances constitute
the style. We present a domain-independent method that permits the open-ended
recombination of style of one image with the content of another. Open ended
simply means that the method generalizes to style and content not present in
the training data. The method starts by constructing a content embedding using
an existing deep metric-learning technique. This trained content encoder is
incorporated into a variational autoencoder (VAE), paired with a to-be-trained
style encoder. The VAE reconstruction loss alone is inadequate to ensure a
decomposition of the latent representation into style and content. Our method
thus includes an auxiliary loss, leakage filtering, which ensures that no style
information remaining in the content representation is used for reconstruction
and vice versa. We synthesize novel images by decoding the style representation
obtained from one image with the content representation from another. Using
this method for data-set augmentation, we obtain state-of-the-art performance
on few-shot learning tasks
Discrete Event, Continuous Time RNNs
We investigate recurrent neural network architectures for event-sequence
processing. Event sequences, characterized by discrete observations stamped
with continuous-valued times of occurrence, are challenging due to the
potentially wide dynamic range of relevant time scales as well as interactions
between time scales. We describe four forms of inductive bias that should
benefit architectures for event sequences: temporal locality, position and
scale homogeneity, and scale interdependence. We extend the popular gated
recurrent unit (GRU) architecture to incorporate these biases via intrinsic
temporal dynamics, obtaining a continuous-time GRU. The CT-GRU arises by
interpreting the gates of a GRU as selecting a time scale of memory, and the
CT-GRU generalizes the GRU by incorporating multiple time scales of memory and
performing context-dependent selection of time scales for information storage
and retrieval. Event time-stamps drive decay dynamics of the CT-GRU, whereas
they serve as generic additional inputs to the GRU. Despite the very different
manner in which the two models consider time, their performance on eleven data
sets we examined is essentially identical. Our surprising results point both to
the robustness of GRU and LSTM architectures for handling continuous time, and
to the potency of incorporating continuous dynamics into neural architectures.Comment: 21 page
Adapted Deep Embeddings: A Synthesis of Methods for -Shot Inductive Transfer Learning
The focus in machine learning has branched beyond training classifiers on a
single task to investigating how previously acquired knowledge in a source
domain can be leveraged to facilitate learning in a related target domain,
known as inductive transfer learning. Three active lines of research have
independently explored transfer learning using neural networks. In weight
transfer, a model trained on the source domain is used as an initialization
point for a network to be trained on the target domain. In deep metric
learning, the source domain is used to construct an embedding that captures
class structure in both the source and target domains. In few-shot learning,
the focus is on generalizing well in the target domain based on a limited
number of labeled examples. We compare state-of-the-art methods from these
three paradigms and also explore hybrid adapted-embedding methods that use
limited target-domain data to fine tune embeddings constructed from
source-domain data. We conduct a systematic comparison of methods in a variety
of domains, varying the number of labeled instances available in the target
domain (), as well as the number of target-domain classes. We reach three
principal conclusions: (1) Deep embeddings are far superior, compared to weight
transfer, as a starting point for inter-domain transfer or model re-use (2) Our
hybrid methods robustly outperform every few-shot learning and every deep
metric learning method previously proposed, with a mean error reduction of 34%
over state-of-the-art. (3) Among loss functions for discovering embeddings, the
histogram loss (Ustinova & Lempitsky, 2016) is most robust. We hope our results
will motivate a unification of research in weight transfer, deep metric
learning, and few-shot learning
How deep is knowledge tracing?
In theoretical cognitive science, there is a tension between highly
structured models whose parameters have a direct psychological interpretation
and highly complex, general-purpose models whose parameters and representations
are difficult to interpret. The former typically provide more insight into
cognition but the latter often perform better. This tension has recently
surfaced in the realm of educational data mining, where a deep learning
approach to predicting students' performance as they work through a series of
exercises---termed deep knowledge tracing or DKT---has demonstrated a stunning
performance advantage over the mainstay of the field, Bayesian knowledge
tracing or BKT. In this article, we attempt to understand the basis for DKT's
advantage by considering the sources of statistical regularity in the data that
DKT can leverage but which BKT cannot. We hypothesize four forms of regularity
that BKT fails to exploit: recency effects, the contextualized trial sequence,
inter-skill similarity, and individual variation in ability. We demonstrate
that when BKT is extended to allow it more flexibility in modeling statistical
regularities---using extensions previously proposed in the literature---BKT
achieves a level of performance indistinguishable from that of DKT. We argue
that while DKT is a powerful, useful, general-purpose framework for modeling
student learning, its gains do not come from the discovery of novel
representations---the fundamental advantage of deep learning. To answer the
question posed in our title, knowledge tracing may be a domain that does not
require `depth'; shallow models like BKT can perform just as well and offer us
greater interpretability and explanatory power.Comment: 8 pages, 2 figure
Identity Crisis: Memorization and Generalization under Extreme Overparameterization
We study the interplay between memorization and generalization of
overparameterized networks in the extreme case of a single training example and
an identity-mapping task. We examine fully-connected and convolutional networks
(FCN and CNN), both linear and nonlinear, initialized randomly and then trained
to minimize the reconstruction error. The trained networks stereotypically take
one of two forms: the constant function (memorization) and the identity
function (generalization). We formally characterize generalization in
single-layer FCNs and CNNs. We show empirically that different architectures
exhibit strikingly different inductive biases. For example, CNNs of up to 10
layers are able to generalize from a single example, whereas FCNs cannot learn
the identity function reliably from 60k examples. Deeper CNNs often fail, but
nonetheless do astonishing work to memorize the training output: because CNN
biases are location invariant, the model must progressively grow an output
pattern from the image boundaries via the coordination of many layers. Our work
helps to quantify and visualize the sensitivity of inductive biases to
architectural choices such as depth, kernel width, and number of channels.Comment: ICLR 202
Convolutional Bipartite Attractor Networks
In human perception and cognition, a fundamental operation that brains
perform is interpretation: constructing coherent neural states from noisy,
incomplete, and intrinsically ambiguous evidence. The problem of interpretation
is well matched to an early and often overlooked architecture, the attractor
network---a recurrent neural net that performs constraint satisfaction,
imputation of missing features, and clean up of noisy data via energy
minimization dynamics. We revisit attractor nets in light of modern deep
learning methods and propose a convolutional bipartite architecture with a
novel training loss, activation function, and connectivity constraints. We
tackle larger problems than have been previously explored with attractor nets
and demonstrate their potential for image completion and super-resolution. We
argue that this architecture is better motivated than ever-deeper feedforward
models and is a viable alternative to more costly sampling-based generative
methods on a range of supervised and unsupervised tasks
Compositional Embeddings for Multi-Label One-Shot Learning
We present a compositional embedding framework that infers not just a single
class per input image, but a set of classes, in the setting of one-shot
learning. Specifically, we propose and evaluate several novel models consisting
of (1) an embedding function f trained jointly with a "composition" function g
that computes set union operations between the classes encoded in two embedding
vectors; and (2) embedding f trained jointly with a "query" function h that
computes whether the classes encoded in one embedding subsume the classes
encoded in another embedding. In contrast to prior work, these models must both
perceive the classes associated with the input examples and encode the
relationships between different class label sets, and they are trained using
only weak one-shot supervision consisting of the label-set relationships among
training examples. Experiments on the OmniGlot, Open Images, and COCO datasets
show that the proposed compositional embedding models outperform existing
embedding methods. Our compositional embedding models have applications to
multi-label object recognition for both one-shot and supervised learning
Learning to Generate Images with Perceptual Similarity Metrics
Deep networks are increasingly being applied to problems involving image
synthesis, e.g., generating images from textual descriptions and reconstructing
an input image from a compact representation. Supervised training of
image-synthesis networks typically uses a pixel-wise loss (PL) to indicate the
mismatch between a generated image and its corresponding target image. We
propose instead to use a loss function that is better calibrated to human
perceptual judgments of image quality: the multiscale structural-similarity
score (MS-SSIM). Because MS-SSIM is differentiable, it is easily incorporated
into gradient-descent learning. We compare the consequences of using MS-SSIM
versus PL loss on training deterministic and stochastic autoencoders. For three
different architectures, we collected human judgments of the quality of image
reconstructions. Observers reliably prefer images synthesized by
MS-SSIM-optimized models over those synthesized by PL-optimized models, for two
distinct PL measures ( and distances). We also explore the
effect of training objective on image encoding and analyze conditions under
which perceptually-optimized representations yield better performance on image
classification. Finally, we demonstrate the superiority of
perceptually-optimized networks for super-resolution imaging. Just as computer
vision has advanced through the use of convolutional architectures that mimic
the structure of the mammalian visual system, we argue that significant
additional advances can be made in modeling images through the use of training
objectives that are well aligned to characteristics of human perception
von Mises-Fisher Loss: An Exploration of Embedding Geometries for Supervised Learning
Recent work has argued that classification losses utilizing softmax
cross-entropy are superior not only for fixed-set classification tasks, but
also by outperforming losses developed specifically for open-set tasks
including few-shot learning and retrieval. Softmax classifiers have been
studied using different embedding geometries -- Euclidean, hyperbolic, and
spherical -- and claims have been made about the superiority of one or another,
but they have not been systematically compared with careful controls. We
conduct an empirical investigation of embedding geometry on softmax losses for
a variety of fixed-set classification and image retrieval tasks. An interesting
property observed for the spherical losses lead us to propose a probabilistic
classifier based on the von Mises-Fisher distribution, and we show that it is
competitive with state-of-the-art methods while producing improved
out-of-the-box calibration. We provide guidance regarding the trade-offs
between losses and how to choose among them.Comment: ICCV 202
- …