Transfer learning is a very important tool in deep learning as it allows
propagating information from one "source dataset" to another "target dataset",
especially in the case of a small number of training examples in the latter.
Yet, discrepancies between the underlying distributions of the source and
target data are commonplace and are known to have a substantial impact on
algorithm performance. In this work we suggest a novel information theoretic
approach for the analysis of the performance of deep neural networks in the
context of transfer learning. We focus on the task of semi-supervised transfer
learning, in which unlabeled samples from the target dataset are available
during the network training on the source dataset. Our theory suggests that one
may improve the transferability of a deep neural network by imposing a Lautum
information based regularization that relates the network weights to the target
data. We demonstrate the effectiveness of the proposed approach in various
transfer learning experiments
