19 research outputs found
A Theory of Output-Side Unsupervised Domain Adaptation
When learning a mapping from an input space to an output space, the
assumption that the sample distribution of the training data is the same as
that of the test data is often violated. Unsupervised domain shift methods
adapt the learned function in order to correct for this shift. Previous work
has focused on utilizing unlabeled samples from the target distribution. We
consider the complementary problem in which the unlabeled samples are given
post mapping, i.e., we are given the outputs of the mapping of unknown samples
from the shifted domain. Two other variants are also studied: the two sided
version, in which unlabeled samples are give from both the input and the output
spaces, and the Domain Transfer problem, which was recently formalized. In all
cases, we derive generalization bounds that employ discrepancy terms
The Role of Minimal Complexity Functions in Unsupervised Learning of Semantic Mappings
We discuss the feasibility of the following learning problem: given unmatched
samples from two domains and nothing else, learn a mapping between the two,
which preserves semantics. Due to the lack of paired samples and without any
definition of the semantic information, the problem might seem ill-posed.
Specifically, in typical cases, it seems possible to build infinitely many
alternative mappings from every target mapping. This apparent ambiguity stands
in sharp contrast to the recent empirical success in solving this problem.
We identify the abstract notion of aligning two domains in a semantic way
with concrete terms of minimal relative complexity. A theoretical framework for
measuring the complexity of compositions of functions is developed in order to
show that it is reasonable to expect the minimal complexity mapping to be
unique. The measured complexity used is directly related to the depth of the
neural networks being learned and a semantically aligned mapping could then be
captured simply by learning using architectures that are not much bigger than
the minimal architecture.
Various predictions are made based on the hypothesis that semantic alignment
can be captured by the minimal mapping. These are verified extensively. In
addition, a new mapping algorithm is proposed and shown to lead to better
mapping results
Estimating the Success of Unsupervised Image to Image Translation
While in supervised learning, the validation error is an unbiased estimator
of the generalization (test) error and complexity-based generalization bounds
are abundant, no such bounds exist for learning a mapping in an unsupervised
way. As a result, when training GANs and specifically when using GANs for
learning to map between domains in a completely unsupervised way, one is forced
to select the hyperparameters and the stopping epoch by subjectively examining
multiple options. We propose a novel bound for predicting the success of
unsupervised cross domain mapping methods, which is motivated by the recently
proposed Simplicity Principle. The bound can be applied both in expectation,
for comparing hyperparameters and for selecting a stopping criterion, or per
sample, in order to predict the success of a specific cross-domain translation.
The utility of the bound is demonstrated in an extensive set of experiments
employing multiple recent algorithms. Our code is available at
https://github.com/sagiebenaim/gan_bound .Comment: The first and second authors contributed equall
Generalization Bounds for Unsupervised Cross-Domain Mapping with WGANs
The recent empirical success of unsupervised cross-domain mapping algorithms,
between two domains that share common characteristics, is not well-supported by
theoretical justifications. This lacuna is especially troubling, given the
clear ambiguity in such mappings. We work with the adversarial training method
called the Wasserstein GAN and derive a novel generalization bound, which
limits the risk between the learned mapping and the target mapping , by
a sum of two terms: (i) the risk between and the most distant alternative
mapping that was learned by the same cross-domain mapping algorithm, and (ii)
the minimal Wasserstein GAN divergence between the target domain and the domain
obtained by applying a hypothesis on the samples of the source domain,
where is a hypothesis selected by the same algorithm. The bound is
directly related to Occam's razor and encourages the selection of the minimal
architecture that supports a small Wasserstein GAN divergence. The bound leads
to multiple algorithmic consequences, including a method for hyperparameters
selection and for an early stopping in cross-domain mapping GANs. We also
demonstrate a novel capability for unsupervised learning of estimating
confidence in the mapping of every specific sample. Lastly, we show how
non-minimal architectures can be effectively trained by an inverted knowledge
distillation, in which a minimal architecture is used to train a larger one,
leading to higher quality outputs.Comment: arXiv admin note: text overlap with arXiv:1709.0007
Unsupervised Learning of the Set of Local Maxima
This paper describes a new form of unsupervised learning, whose input is a
set of unlabeled points that are assumed to be local maxima of an unknown value
function v in an unknown subset of the vector space. Two functions are learned:
(i) a set indicator c, which is a binary classifier, and (ii) a comparator
function h that given two nearby samples, predicts which sample has the higher
value of the unknown function v. Loss terms are used to ensure that all
training samples x are a local maxima of v, according to h and satisfy c(x)=1.
Therefore, c and h provide training signals to each other: a point x' in the
vicinity of x satisfies c(x)=-1 or is deemed by h to be lower in value than x.
We present an algorithm, show an example where it is more efficient to use
local maxima as an indicator function than to employ conventional
classification, and derive a suitable generalization bound. Our experiments
show that the method is able to outperform one-class classification algorithms
in the task of anomaly detection and also provide an additional signal that is
extracted in a completely unsupervised way.Comment: ICLR 201
On Random Kernels of Residual Architectures
We derive finite width and depth corrections for the Neural Tangent Kernel
(NTK) of ResNets and DenseNets. Our analysis reveals that finite size residual
architectures are initialized much closer to the "kernel regime" than their
vanilla counterparts: while in networks that do not use skip connections,
convergence to the NTK requires one to fix the depth, while increasing the
layers' width. Our findings show that in ResNets, convergence to the NTK may
occur when depth and width simultaneously tend to infinity, provided with a
proper initialization. In DenseNets, however, convergence of the NTK to its
limit as the width tends to infinity is guaranteed, at a rate that is
independent of both the depth and scale of the weights. Our experiments
validate the theoretical results and demonstrate the advantage of deep ResNets
and DenseNets for kernel regression with random gradient features
Evaluation Metrics for Conditional Image Generation
We present two new metrics for evaluating generative models in the
class-conditional image generation setting. These metrics are obtained by
generalizing the two most popular unconditional metrics: the Inception Score
(IS) and the Fre'chet Inception Distance (FID). A theoretical analysis shows
the motivation behind each proposed metric and links the novel metrics to their
unconditional counterparts. The link takes the form of a product in the case of
IS or an upper bound in the FID case. We provide an extensive empirical
evaluation, comparing the metrics to their unconditional variants and to other
metrics, and utilize them to analyze existing generative models, thus providing
additional insights about their performance, from unlearned classes to mode
collapse.Comment: To be published in "INTERNATIONAL JOURNAL OF COMPUTER VISION
Domain Intersection and Domain Difference
We present a method for recovering the shared content between two visual
domains as well as the content that is unique to each domain. This allows us to
map from one domain to the other, in a way in which the content that is
specific for the first domain is removed and the content that is specific for
the second is imported from any image in the second domain. In addition, our
method enables generation of images from the intersection of the two domains as
well as their union, despite having no such samples during training. The method
is shown analytically to contain all the sufficient and necessary constraints.
It also outperforms the literature methods in an extensive set of experiments.
Our code is available at
https://github.com/sagiebenaim/DomainIntersectionDifference
On Infinite-Width Hypernetworks
{\em Hypernetworks} are architectures that produce the weights of a
task-specific {\em primary network}. A notable application of hypernetworks in
the recent literature involves learning to output functional representations.
In these scenarios, the hypernetwork learns a representation corresponding to
the weights of a shallow MLP, which typically encodes shape or image
information. While such representations have seen considerable success in
practice, they remain lacking in the theoretical guarantees in the wide regime
of the standard architectures. In this work, we study wide over-parameterized
hypernetworks. We show that unlike typical architectures, infinitely wide
hypernetworks do not guarantee convergence to a global minima under gradient
descent. We further show that convexity can be achieved by increasing the
dimensionality of the hypernetwork's output, to represent wide MLPs. In the
dually infinite-width regime, we identify the functional priors of these
architectures by deriving their corresponding GP and NTK kernels, the latter of
which we refer to as the {\em hyperkernel}. As part of this study, we make a
mathematical contribution by deriving tight bounds on high order Taylor
expansion terms of standard fully connected ReLU networks.Comment: The first two authors contributed equall
Emerging Disentanglement in Auto-Encoder Based Unsupervised Image Content Transfer
We study the problem of learning to map, in an unsupervised way, between
domains A and B, such that the samples b in B contain all the information that
exists in samples a in A and some additional information. For example, ignoring
occlusions, B can be people with glasses, A people without, and the glasses,
would be the added information. When mapping a sample a from the first domain
to the other domain, the missing information is replicated from an independent
reference sample b in B. Thus, in the above example, we can create, for every
person without glasses a version with the glasses observed in any face image.
Our solution employs a single two-pathway encoder and a single decoder for
both domains. The common part of the two domains and the separate part are
encoded as two vectors, and the separate part is fixed at zero for domain A.
The loss terms are minimal and involve reconstruction losses for the two
domains and a domain confusion term. Our analysis shows that under mild
assumptions, this architecture, which is much simpler than the literature
guided-translation methods, is enough to ensure disentanglement between the two
domains. We present convincing results in a few visual domains, such as
no-glasses to glasses, adding facial hair based on a reference image, etc