64 research outputs found
Reversible GANs for Memory-efficient Image-to-Image Translation
The Pix2pix and CycleGAN losses have vastly improved the qualitative and
quantitative visual quality of results in image-to-image translation tasks. We
extend this framework by exploring approximately invertible architectures which
are well suited to these losses. These architectures are approximately
invertible by design and thus partially satisfy cycle-consistency before
training even begins. Furthermore, since invertible architectures have constant
memory complexity in depth, these models can be built arbitrarily deep. We are
able to demonstrate superior quantitative output on the Cityscapes and Maps
datasets at near constant memory budget
Learning Layer-wise Equivariances Automatically using Gradients
Convolutions encode equivariance symmetries into neural networks leading to
better generalisation performance. However, symmetries provide fixed hard
constraints on the functions a network can represent, need to be specified in
advance, and can not be adapted. Our goal is to allow flexible symmetry
constraints that can automatically be learned from data using gradients.
Learning symmetry and associated weight connectivity structures from scratch is
difficult for two reasons. First, it requires efficient and flexible
parameterisations of layer-wise equivariances. Secondly, symmetries act as
constraints and are therefore not encouraged by training losses measuring data
fit. To overcome these challenges, we improve parameterisations of soft
equivariance and learn the amount of equivariance in layers by optimising the
marginal likelihood, estimated using differentiable Laplace approximations. The
objective balances data fit and model complexity enabling layer-wise symmetry
discovery in deep networks. We demonstrate the ability to automatically learn
layer-wise equivariances on image classification tasks, achieving equivalent or
improved performance over baselines with hard-coded symmetry
The molecular weight of the A-chains of α-crystallin
Contains fulltext :
142593.pdf (publisher's version ) (Open Access
Bovine -crystallin: sequence of the C-terminal cyanogen bromide fragment of the A chain
Contains fulltext :
142263.pdf (publisher's version ) (Open Access
Learning Invariant Weights in Neural Networks
Assumptions about invariances or symmetries in data can significantly
increase the predictive power of statistical models. Many commonly used models
in machine learning are constraint to respect certain symmetries in the data,
such as translation equivariance in convolutional neural networks, and
incorporation of new symmetry types is actively being studied. Yet, efforts to
learn such invariances from the data itself remains an open research problem.
It has been shown that marginal likelihood offers a principled way to learn
invariances in Gaussian Processes. We propose a weight-space equivalent to this
approach, by minimizing a lower bound on the marginal likelihood to learn
invariances in neural networks resulting in naturally higher performing models
Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations
Data augmentation is commonly applied to improve performance of deep learning
by enforcing the knowledge that certain transformations on the input preserve
the output. Currently, the used data augmentation is chosen by human effort and
costly cross-validation, which makes it cumbersome to apply to new datasets. We
develop a convenient gradient-based method for selecting the data augmentation
without validation data and during training of a deep neural network. Our
approach relies on phrasing data augmentation as an invariance in the prior
distribution and learning it using Bayesian model selection, which has been
shown to work in Gaussian processes, but not yet for deep neural networks. We
propose a differentiable Kronecker-factored Laplace approximation to the
marginal likelihood as our objective, which can be optimised without human
supervision or validation data. We show that our method can successfully
recover invariances present in the data, and that this improves generalisation
and data efficiency on image datasets
- …