Denoising Adversarial Autoencoders: Classifying Skin Lesions Using Limited Labelled Training Data

We propose a novel deep learning model for classifying medical images in the setting where there is a large amount of unlabelled medical data available, but labelled data is in limited supply. We consider the specific case of classifying skin lesions as either malignant or benign. In this setting, the proposed approach -- the semi-supervised, denoising adversarial autoencoder -- is able to utilise vast amounts of unlabelled data to learn a representation for skin lesions, and small amounts of labelled data to assign class labels based on the learned representation. We analyse the contributions of both the adversarial and denoising components of the model and find that the combination yields superior classification performance in the setting of limited labelled training data.Comment: Under consideration for the IET Computer Vision Journal special issue on "Computer Vision in Cancer Data Analysis

Scalable approximate inference methods for Bayesian deep learning

This thesis proposes multiple methods for approximate inference in deep Bayesian neural networks split across three parts. The first part develops a scalable Laplace approximation based on a block- diagonal Kronecker factored approximation of the Hessian. This approximation accounts for parameter correlations – overcoming the overly restrictive independence assumption of diagonal methods – while avoiding the quadratic scaling in the num- ber of parameters of the full Laplace approximation. The chapter further extends the method to online learning where datasets are observed one at a time. As the experiments demonstrate, modelling correlations between the parameters leads to improved performance over the diagonal approximation in uncertainty estimation and continual learning, in particular in the latter setting the improvements can be substantial. The second part explores two parameter-efficient approaches for variational inference in neural networks, one based on factorised binary distributions over the weights, one extending ideas from sparse Gaussian processes to neural network weight matrices. The former encounters similar underfitting issues as mean-field Gaussian approaches, which can be alleviated by a MAP-style method in a hierarchi- cal model. The latter, based on an extension of Matheron’s rule to matrix normal distributions, achieves comparable uncertainty estimation performance to ensembles with the accuracy of a deterministic network while using only 25% of the number of parameters of a single ResNet-50. The third part introduces TyXe, a probabilistic programming library built on top of Pyro to facilitate turning PyTorch neural networks into Bayesian ones. In contrast to existing frameworks, TyXe avoids introducing a layer abstraction, allowing it to support arbitrary architectures. This is demonstrated in a range of applications, from image classification with torchvision ResNets over node labelling with DGL graph neural networks to incorporating uncertainty into neural radiance fields with PyTorch3d

TAnoGAN: Time Series Anomaly Detection with Generative Adversarial Networks

Anomaly detection in time series data is a significant problem faced in many application areas such as manufacturing, medical imaging and cyber-security. Recently, Generative Adversarial Networks (GAN) have gained attention for generation and anomaly detection in image domain. In this paper, we propose a novel GAN-based unsupervised method called TAnoGan for detecting anomalies in time series when a small number of data points are available. We evaluate TAnoGan with 46 real-world time series datasets that cover a variety of domains. Extensive experimental results show that TAnoGan performs better than traditional and neural network models.Comment: Made some minor changes. This is the accepted version of the paper at AusDM'2

Practical probabilistic systems for satellite image segmentation and classifcation

Thesis (MEng)--Stellenbosch University, 2020.ENGLISH ABSTRACT: This thesis undertakes the task of satellite image classification from a probabilistic perspective. Our probabilistic approach is motivated by using uncertainty to address the lack of data and variability in satellite image data. In the interest of producing accurate models, we adopt Bayesian neural networks (BNNs) as the primary focus for classification models which offer a way of combining uncertainty estimation with the expressive power of deep learning. Furthermore, due to the limited communication bandwidth of a satellite, we require the model to run on-board a satellite which introduces major computational constraints. BNNs can also be designed to introduce sparsity providing a computationally efficient solution. Despite these advantages, BNNs are rarely used in practice as they are difficult to train. We discuss the most recent advances in variational techniques, including Monte-Carlo variational inference, stochastic optimisation, the reparametrisation trick, and local reparametrisation trick. However, even with these advances BNNs often still suffer from crippling gradient variance. In an attempt to understand this we study the relationship between probabilistic modelling and stochastic regularisation techniques, setting the foundation for practical uncertainty estimators, compression techniques and a signal propagation analysis of BNNs. Using this understanding we present an innovation using signal propagation theory to propose a self-stabilising prior that improves robustness in training. We then discuss techniques for incorporating spatial information making use of probabilistic graphical models (PGMs). We connect the output of pixel classifications of a BNN to a PGM, developing a probabilistic system. This uses the uncertainty of the classifier, together with the contextual information of neighbouring pixels, to have a de-noising effect on the classifier output. Finally, we experimentally evaluate a series of Bayesian and deterministic models for satellite image classification. We see that Bayesian methods excel in situations where data is scarce. We also see that BNNs are able to achieve levels of accuracy comparable to modern deep learning while either remaining well-calibrated in comparison to deterministic methods, or able to yield extremely sparse solutions requiring only 3 % of the original weights. In addition, we qualitatively illustrate the value of models that recognise their fallibility and incorporating them into probabilistic systems which can reason automatically and dynamically incorporate information from different sources depending on the certainty of each source.AFRIKAANSE OPSOMMING: Hierdie tesis onderneem satelliet beeld klassifikasie vanuit ’n probabilistiese benadering. Ons probabilistiese benadering is gemotiveer deur die gebruik van onsekerheid om die gebrek aan en veranderlikheid in satelliet data te adresseer. Om akkurate modelle te verseker maak ons hoofsaaklik gebruik van “Bayesian neural networks” (BNNs). BNNs verskaf ’n manier om onsekerheid skatting met die modellering krag van “deep learning” te kombineer. Daarbenewens, weens beperkte kommunikasie bandwydte van ’n satelliet, behoort die model op die te kan satelliet opereer wat groot rekenkundige beperkings voorstel. BNNs kan ook ontwerp word om parameters te verwyder wat gevolglik koste effektiewe oplossings verskaf. Ten spyte van hierdie voordele word BNNs selde gebruik want in praktyk kan die opleiding van die modelle geweldig moeilik wees. Ons bespreek onlangse vernuwings in variasionele tegnieke, wat “Monte-Carlo variational inference”, “stochastic optimisation”, die “reparametrisation trick” en “local reparametrisation trick” insluit. Ons bestudeer ook die verwantskap tussen BNNs en stogastiese regularisering tegnieke wat die fondament vir praktiese onsekerheid skatters, kompressie tegnieke en ’n sein voortplanting analise van BNNs lˆe. Hierdie tegnieke het Bayesiese diep-leer moontlik gemaak, maar die tegnieke ly steeds aan skadelike gradi¨ent variansie. Ons spreek hierdie aan met ’n innovasie met die gebruik van sein voortplanting teorie om ’n self-stabiliserende prior voor te stel wat opleiding robuust maak. Daarna bespreek ons die gebruik van probabilistiese grafiese modelle (PGMs) om ruimtelike inligting te inkorporeer. Ons verbind die uitset van die klassifikasie model aan ’n PGM, om ’n probabilistiese stelsel te ontwikkel. Dit gebruik die onsekerheid van die klassifiseerder in kombinasie met die kontekstuele inligting van die naburige pixels wat die uitset skoon maak. Laastens maak ons ’n eksperimentele evaluering van ’n reeks van Bayesiese en deterministiese modelle op satelliet beeld klassifikasie. Ons neem waar dat Bayesiese modelle presteer in situasies waar data skaars is. Ons sien ook dat BNNs diep-leer vlakke van akkuraatheid bereik terwyl hulle ´of, goed gekalibreer bly in vergelyking met deterministiese metodes, ´of in staat is om uiters koste effektiewe oplossings te lewer, wat net 3 % van die oorspronklike parameters vereis. Daarbenewens, ondersoek ons die waarde van modelle wat hul feilbaarheid kan herken wat stelsels gee wat dinamies inligting van verskeie bronne kan inkorporeer en outomaties redeneer.Master

Uncertainty in Neural Networks; Bayesian Ensembles, Priors & Prediction Intervals

The breakout success of deep neural networks (NNs) in the 2010's marked a new era in the quest to build artificial intelligence (AI). With NNs as the building block of these systems, excellent performance has been achieved on narrow, well-defined tasks where large amounts of data are available. However, these systems lack certain capabilities that are important for broad use in real-world applications. One such capability is the communication of uncertainty in a NN's predictions and decisions. In applications such as healthcare recommendation or heavy machinery prognostics, it is vital that AI systems be aware of and express their uncertainty – this creates safer, more cautious, and ultimately more useful systems. This thesis explores how to engineer NNs to communicate robust uncertainty estimates on their predictions, whilst minimising the impact on usability. One way to encourage uncertainty estimates to be robust is to adopt the Bayesian framework, which offers a principled approach to handling uncertainty. Two of the major contributions in this thesis relate to Bayesian NNs (BNNs). Specifying appropriate priors is an important step in any Bayesian model, yet it is not clear how to do this in BNNs. The first contribution shows that the connection between BNNs and Gaussian Processes (GPs) provides an effective lens to study BNN priors. NN architectures are derived which mirror the combining of GP kernels to create priors tailored to a task. The second major contribution is a novel way to perform approximate Bayesian inference in BNNs using a modified version of ensembling. Novel analysis improves an understanding of a technique known as randomised MAP sampling. It's shown this is particularly effective when strong correlations exist between parameters, making it well suited to NNs. The third major contribution of the thesis is a non-Bayesian technique that trains a NN to directly output prediction intervals for regression tasks through a tailored objective function. This advances over related works that were incompatible with gradient descent, and ignored one source of uncertainty.EPSRC, Alan Turing Institut

Cumulative Distribution Functions As The Foundation For Probabilistic Models

This thesis discusses applications of probabilistic and connectionist models for constructing and training cumulative distribution functions (CDFs). First, it is shown how existing tools from the copula literature can be combined to build probabilistic models. It is found that this simple construction leads to numerical and scalability issues that make training and inference challenging. Next, several innovative ideas, combining neural networks, automatic differentiation and copula functions, introduce how to assemble black-box probabilistic models. The basic building block is a cumulative distribution function that is straightforward to construct, composed of arithmetic operations and nonlinear functions. There is no need to assume any specific parametric probability density function (PDF), making the model flexible and normalisation unnecessary. The only requirement is to design a computational graph that parameterises monotonically non-decreasing functions with a constrained range. Training can be then performed using standard tools from any neural network software library. Finally, factorial hidden Markov models (FHMMs) for sequential data are presented. It is shown how to leverage cumulative distribution functions in the form of the Gaussian copula and amortised stochastic variational method to encode hidden Markov chains coherently. This approach enables efficient learning and inference to model long sequences of high-dimensional data with long-range dependencies. Tackling such complex problems was impossible with the established FHMM approximate inference algorithm. It is empirically verified on several problems that some of the estimators introduced in this work can perform comparably or better than the currently popular models. Especially for tasks requiring tail-area or marginal probabilities that can be read directly from a cumulative distribution function