6 research outputs found
Contrastive Learning for Lifted Networks
In this work we address supervised learning of neural networks via lifted
network formulations. Lifted networks are interesting because they allow
training on massively parallel hardware and assign energy models to
discriminatively trained neural networks. We demonstrate that the training
methods for lifted networks proposed in the literature have significant
limitations and show how to use a contrastive loss to address those
limitations. We demonstrate that this contrastive training approximates
back-propagation in theory and in practice and that it is superior to the
training objective regularly used for lifted networks.Comment: 9 pages, BMVC 201
Lifted Regression/Reconstruction Networks
In this work we propose lifted regression/reconstruction networks(LRRNs), which combine lifted neural networks with a guaranteed Lipschitz continuity property for the output layer. Lifted neural networks explicitly optimize an energy model to infer the unit activations and therefore—in contrast to standard feed-forward neural networks—allow bidirectional feedback between layers. So far lifted neural networks have been modelled around standard feed-forward architectures. We propose to take further advantage of the feedback property by letting the layers simultaneously perform regression and reconstruction. The resulting lifted network architecture allows to control the desired amount of Lipschitz continuity, which is an important feature to obtain adversarially robust regression and classification methods. We analyse and numerically demonstrate applications for unsupervised and supervised learnin
Lifted Regression/Reconstruction Networks
In this work we propose lifted regression/reconstruction networks (LRRNs),
which combine lifted neural networks with a guaranteed Lipschitz continuity
property for the output layer. Lifted neural networks explicitly optimize an
energy model to infer the unit activations and therefore---in contrast to
standard feed-forward neural networks---allow bidirectional feedback between
layers. So far lifted neural networks have been modelled around standard
feed-forward architectures. We propose to take further advantage of the
feedback property by letting the layers simultaneously perform regression and
reconstruction. The resulting lifted network architecture allows to control the
desired amount of Lipschitz continuity, which is an important feature to obtain
adversarially robust regression and classification methods. We analyse and
numerically demonstrate applications for unsupervised and supervised learning.Comment: 12 pages, 8 figure
Truncated Inference for Latent Variable Optimization Problems: Application to Robust Estimation and Learning
Optimization problems with an auxiliary latent variable structure in addition
to the main model parameters occur frequently in computer vision and machine
learning. The additional latent variables make the underlying optimization task
expensive, either in terms of memory (by maintaining the latent variables), or
in terms of runtime (repeated exact inference of latent variables). We aim to
remove the need to maintain the latent variables and propose two formally
justified methods, that dynamically adapt the required accuracy of latent
variable inference. These methods have applications in large scale robust
estimation and in learning energy-based models from labeled data.Comment: 16 page
Contrastive learning for lifted networks
In this work we address supervised learning via lifted network formulations. Lifted networks are interesting because they allow training on massively parallel hardware and assign energy models to discriminatively trained neural networks. We demonstrate that training methods for lifted networks proposed in the literature have significant limitations, and therefore we propose to use a contrastive loss to train lifted networks. We show that this contrastive training approximates back-propagation in theory and in practice, and that it is superior to the regular training objective for lifted networks