6,152 research outputs found
Distance-Based Regularisation of Deep Networks for Fine-Tuning
We investigate approaches to regularisation during fine-tuning of deep neural
networks. First we provide a neural network generalisation bound based on
Rademacher complexity that uses the distance the weights have moved from their
initial values. This bound has no direct dependence on the number of weights
and compares favourably to other bounds when applied to convolutional networks.
Our bound is highly relevant for fine-tuning, because providing a network with
a good initialisation based on transfer learning means that learning can modify
the weights less, and hence achieve tighter generalisation. Inspired by this,
we develop a simple yet effective fine-tuning algorithm that constrains the
hypothesis class to a small sphere centred on the initial pre-trained weights,
thus obtaining provably better generalisation performance than conventional
transfer learning. Empirical evaluation shows that our algorithm works well,
corroborating our theoretical results. It outperforms both state of the art
fine-tuning competitors, and penalty-based alternatives that we show do not
directly constrain the radius of the search space
The learnability of unknown quantum measurements
© Rinton Press. In this work, we provide an elegant framework to analyze learning matrices in the Schatten class by taking advantage of a recently developed methodology—matrix concentration inequalities. We establish the fat-shattering dimension, Rademacher/Gaussian complexity, and the entropy number of learning bounded operators and trace class operators. By characterising the tasks of learning quantum states and two-outcome quantum measurements into learning matrices in the Schatten-1 and ∞ classes, our proposed approach directly solves the sample complexity problems of learning quantum states and quantum measurements. Our main result in the paper is that, for learning an unknown quantum measurement, the upper bound, given by the fat-shattering dimension, is linearly proportional to the dimension of the underlying Hilbert space. Learning an unknown quantum state becomes a dual problem to ours, and as a byproduct, we can recover Aaronson’s famous result [Proc. R. Soc. A 463, 3089–3144 (2007)] solely using a classical machine learning technique. In addition, other famous complexity measures like covering numbers and Rademacher/Gaussian complexities are derived explicitly under the same framework. We are able to connect measures of sample complexity with various areas in quantum information science, e.g. quantum state/measurement tomography, quantum state discrimination and quantum random access codes, which may be of independent interest. Lastly, with the assistance of general Bloch-sphere representation, we show that learning quantum measurements/states can be mathematically formulated as a neural network. Consequently, classical ML algorithms can be applied to efficiently accomplish the two quantum learning tasks
Approximation-Generalization Trade-offs under (Approximate) Group Equivariance
The explicit incorporation of task-specific inductive biases through symmetry
has emerged as a general design precept in the development of high-performance
machine learning models. For example, group equivariant neural networks have
demonstrated impressive performance across various domains and applications
such as protein and drug design. A prevalent intuition about such models is
that the integration of relevant symmetry results in enhanced generalization.
Moreover, it is posited that when the data and/or the model may only exhibit
or symmetry, the optimal or
best-performing model is one where the model symmetry aligns with the data
symmetry. In this paper, we conduct a formal unified investigation of these
intuitions. To begin, we present general quantitative bounds that demonstrate
how models capturing task-specific symmetries lead to improved generalization.
In fact, our results do not require the transformations to be finite or even
form a group and can work with partial or approximate equivariance. Utilizing
this quantification, we examine the more general question of model
mis-specification i.e. when the model symmetries don't align with the data
symmetries. We establish, for a given symmetry group, a quantitative comparison
between the approximate/partial equivariance of the model and that of the data
distribution, precisely connecting model equivariance error and data
equivariance error. Our result delineates conditions under which the model
equivariance error is optimal, thereby yielding the best-performing model for
the given task and data
A Model of Inductive Bias Learning
A major problem in machine learning is that of inductive bias: how to choose
a learner's hypothesis space so that it is large enough to contain a solution
to the problem being learnt, yet small enough to ensure reliable generalization
from reasonably-sized training sets. Typically such bias is supplied by hand
through the skill and insights of experts. In this paper a model for
automatically learning bias is investigated. The central assumption of the
model is that the learner is embedded within an environment of related learning
tasks. Within such an environment the learner can sample from multiple tasks,
and hence it can search for a hypothesis space that contains good solutions to
many of the problems in the environment. Under certain restrictions on the set
of all hypothesis spaces available to the learner, we show that a hypothesis
space that performs well on a sufficiently large number of training tasks will
also perform well when learning novel tasks in the same environment. Explicit
bounds are also derived demonstrating that learning multiple tasks within an
environment of related tasks can potentially give much better generalization
than learning a single task
Deep learning model-aware regulatization with applications to Inverse Problems
There are various inverse problems – including reconstruction problems arising in medical imaging - where one is often aware of the forward operator that maps variables of interest to the observations. It is therefore natural to ask whether such knowledge of the forward operator can be exploited in deep learning approaches increasingly used to solve inverse problems. In this paper, we provide one such way via an analysis of the generalisation error of deep learning approaches to inverse problems. In particular, by building on the algorithmic robustness framework, we offer a generalisation error bound that encapsulates key ingredients associated with the learning problem such as the complexity of the data space, the size of the training set, the Jacobian of the deep neural network and the Jacobian of the composition of the forward operator with the neural network. We then propose a ‘plug-and-play’ regulariser that leverages the knowledge of the forward map to improve the generalization of the network. We likewise also use a new method allowing us to tightly upper bound the Jacobians of the relevant operators that is much more computationally efficient than existing ones. We demonstrate the efficacy of our model-aware regularised deep learning algorithms against other state-of-the-art approaches on inverse problems involving various sub-sampling operators such as those used in classical compressed sensing tasks, image super-resolution problems and accelerated Magnetic Resonance Imaging (MRI) setups
Sparse machine learning methods with applications in multivariate signal processing
This thesis details theoretical and empirical work that draws from two main subject areas: Machine
Learning (ML) and Digital Signal Processing (DSP). A unified general framework is given for the application
of sparse machine learning methods to multivariate signal processing. In particular, methods that
enforce sparsity will be employed for reasons of computational efficiency, regularisation, and compressibility.
The methods presented can be seen as modular building blocks that can be applied to a variety
of applications. Application specific prior knowledge can be used in various ways, resulting in a flexible
and powerful set of tools. The motivation for the methods is to be able to learn and generalise from a set
of multivariate signals.
In addition to testing on benchmark datasets, a series of empirical evaluations on real world
datasets were carried out. These included: the classification of musical genre from polyphonic audio
files; a study of how the sampling rate in a digital radar can be reduced through the use of Compressed
Sensing (CS); analysis of human perception of different modulations of musical key from
Electroencephalography (EEG) recordings; classification of genre of musical pieces to which a listener
is attending from Magnetoencephalography (MEG) brain recordings. These applications demonstrate
the efficacy of the framework and highlight interesting directions of future research
- …