3 research outputs found
Semi-Supervised Learning with Declaratively Specified Entropy Constraints
We propose a technique for declaratively specifying strategies for
semi-supervised learning (SSL). The proposed method can be used to specify
ensembles of semi-supervised learning, as well as agreement constraints and
entropic regularization constraints between these learners, and can be used to
model both well-known heuristics such as co-training and novel domain-specific
heuristics. In addition to representing individual SSL heuristics, we show that
multiple heuristics can also be automatically combined using Bayesian
optimization methods. We show consistent improvements on a suite of
well-studied SSL benchmarks, including a new state-of-the-art result on a
difficult relation extraction task
Lautum Regularization for Semi-supervised Transfer Learning
Transfer learning is a very important tool in deep learning as it allows
propagating information from one "source dataset" to another "target dataset",
especially in the case of a small number of training examples in the latter.
Yet, discrepancies between the underlying distributions of the source and
target data are commonplace and are known to have a substantial impact on
algorithm performance. In this work we suggest a novel information theoretic
approach for the analysis of the performance of deep neural networks in the
context of transfer learning. We focus on the task of semi-supervised transfer
learning, in which unlabeled samples from the target dataset are available
during the network training on the source dataset. Our theory suggests that one
may improve the transferability of a deep neural network by imposing a Lautum
information based regularization that relates the network weights to the target
data. We demonstrate the effectiveness of the proposed approach in various
transfer learning experiments
Learning from Rules Generalizing Labeled Exemplars
In many applications labeled data is not readily available, and needs to be
collected via pain-staking human supervision. We propose a rule-exemplar method
for collecting human supervision to combine the efficiency of rules with the
quality of instance labels. The supervision is coupled such that it is both
natural for humans and synergistic for learning. We propose a training
algorithm that jointly denoises rules via latent coverage variables, and trains
the model through a soft implication loss over the coverage and label
variables. The denoised rules and trained model are used jointly for inference.
Empirical evaluation on five different tasks shows that (1) our algorithm is
more accurate than several existing methods of learning from a mix of clean and
noisy supervision, and (2) the coupled rule-exemplar supervision is effective
in denoising rules.Comment: ICLR 2020 (Spotlight