An Overview of Deep Semi-Supervised Learning

Deep neural networks demonstrated their ability to provide remarkable performances on a wide range of supervised learning tasks (e.g., image classification) when trained on extensive collections of labeled data (e.g., ImageNet). However, creating such large datasets requires a considerable amount of resources, time, and effort. Such resources may not be available in many practical cases, limiting the adoption and the application of many deep learning methods. In a search for more data-efficient deep learning methods to overcome the need for large annotated datasets, there is a rising research interest in semi-supervised learning and its applications to deep neural networks to reduce the amount of labeled data required, by either developing novel methods or adopting existing semi-supervised learning frameworks for a deep learning setting. In this paper, we provide a comprehensive overview of deep semi-supervised learning, starting with an introduction to the field, followed by a summarization of the dominant semi-supervised approaches in deep learning.Comment: Preprin

Representation Learning: A Review and New Perspectives

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning

Robust Learning from Multiple Information Sources

In the big data era, the ability to handle high-volume, high-velocity and high-variety information assets has become a basic requirement for data analysts. Traditional learning models, which focus on medium size, single source data, often fail to achieve reliable performance if data come from multiple heterogeneous sources (views). As a result, robust multi-view data processing methods that are insensitive to corruptions and anomalies in the data set are needed. This thesis develops robust learning methods for three problems that arise from real-world applications: robust training on a noisy training set, multi-view learning in the presence of between-view inconsistency and network topology inference using partially observed data. The central theme behind all these methods is the use of information-theoretic measures, including entropies and information divergences, as parsimonious representations of uncertainties in the data, as robust optimization surrogates that allows for efficient learning, and as flexible and reliable discrepancy measures for data fusion. More specifically, the thesis makes the following contributions: 1. We propose a maximum entropy-based discriminative learning model that incorporates the minimal entropy (ME) set anomaly detection technique. The resulting probabilistic model can perform both nonparametric classification and anomaly detection simultaneously. An efficient algorithm is then introduced to estimate the posterior distribution of the model parameters while selecting anomalies in the training data. 2. We consider a multi-view classification problem on a statistical manifold where class labels are provided by probabilistic density functions (p.d.f.) and may not be consistent among different views due to the existence of noise corruption. A stochastic consensus-based multi-view learning model is proposed to fuse predictive information for multiple views together. By exploring the non-Euclidean structure of the statistical manifold, a joint consensus view is constructed that is robust to single-view noise corruption and between-view inconsistency. 3. We present a method for estimating the parameters (partial correlations) of a Gaussian graphical model that learns a sparse sub-network topology from partially observed relational data. This model is applicable to the situation where the partial correlations between pairs of variables on a measured sub-network (internal data) are to be estimated when only summary information about the partial correlations between variables outside of the sub-network (external data) are available. The proposed model is able to incorporate the dependence structure between latent variables from external sources and perform latent feature selection efficiently. From a multi-view learning perspective, it can be seen as a two-view learning system given asymmetric information flow from both the internal view and the external view.PHDElectrical & Computer Eng PhDUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/138599/1/tianpei_1.pd

Generative adversarial network for predictive maintenance of a packaging machine

Generative models have been designed to discover and learn the latent structure of the input data in order to generate new samples based on the regularities discovered in the data. Starting from the first simplest models such as the Restricted Boltzmann Machines up to the Variational Autoencoders and Generative Adversarial Networks (or GAN), these models have experienced a surprising development in generating data as similar to reality as possible. The potential of these models, especially in Deep Learning, has led to the most disparate applications: generation of images, videos and music, image-to-image translations, text-to-image translation and conversion of low resolution images to high resolution, to name a few. In this thesis work, carried out during the internship period of the Master's Degree, the main focus is on GANs, a generative model that makes use of the principles of supervised training through the use of two competing "sub models": a generator, trained to produce new realistic samples, and a discriminator, which tries to distinguish between real and generated data. Usually, when this model is employed, the focus is mainly on the role of the generator used to produce new data. In this case, however, the idea is to use the discriminator as a binary classifier in the context of Predictive Maintenance of a packaging machine. In other words, the discriminator obtained as a result of GAN training is used to classify the state of the machine as either normal or critical. After an initial pre-processing and exploration of the datasets, the results obtained are compared with other classifiers. Finally, the limits and possible developments of this approach are discussed.Generative models have been designed to discover and learn the latent structure of the input data in order to generate new samples based on the regularities discovered in the data. Starting from the first simplest models such as the Restricted Boltzmann Machines up to the Variational Autoencoders and Generative Adversarial Networks (or GAN), these models have experienced a surprising development in generating data as similar to reality as possible. The potential of these models, especially in Deep Learning, has led to the most disparate applications: generation of images, videos and music, image-to-image translations, text-to-image translation and conversion of low resolution images to high resolution, to name a few. In this thesis work, carried out during the internship period of the Master's Degree, the main focus is on GANs, a generative model that makes use of the principles of supervised training through the use of two competing "sub models": a generator, trained to produce new realistic samples, and a discriminator, which tries to distinguish between real and generated data. Usually, when this model is employed, the focus is mainly on the role of the generator used to produce new data. In this case, however, the idea is to use the discriminator as a binary classifier in the context of Predictive Maintenance of a packaging machine. In other words, the discriminator obtained as a result of GAN training is used to classify the state of the machine as either normal or critical. After an initial pre-processing and exploration of the datasets, the results obtained are compared with other classifiers. Finally, the limits and possible developments of this approach are discussed

Semi-supervised triplet loss based learning of ambient audio embeddings

International audienceDeep neural networks are particularly useful to learn relevant repre-sentations from data. Recent studies have demonstrated the poten-tial of unsupervised representation learning for ambient sound anal-ysis using various flavors of the triplet loss. They have comparedthis approach to supervised learning. However, in real situations,it is common to have a small labeled dataset and a large unlabeledone. In this paper, we combine unsupervised and supervised tripletloss based learning into a semi-supervised representation learningapproach. We propose two flavors of this approach, whereby thepositive samples for those triplets whose anchors are unlabeled areobtained either by applying a transformation to the anchor, or byselecting the nearest sample in the training set. We compare ourapproach to supervised and unsupervised representation learning aswell as the ratio between the amount of labeled and unlabeled data.We evaluate all the above approaches on an audio tagging task usingthe DCASE 2018 Task 4 dataset, and we show the impact of thisratio on the tagging performance

Discriminate-and-Rectify Encoders: Learning from Image Transformation Sets

The complexity of a learning task is increased by transformations in the input space that preserve class identity. Visual object recognition for example is affected by changes in viewpoint, scale, illumination or planar transformations. While drastically altering the visual appearance, these changes are orthogonal to recognition and should not be reflected in the representation or feature encoding used for learning. We introduce a framework for weakly supervised learning of image embeddings that are robust to transformations and selective to the class distribution, using sets of transforming examples (orbit sets), deep parametrizations and a novel orbit-based loss. The proposed loss combines a discriminative, contrastive part for orbits with a reconstruction error that learns to rectify orbit transformations. The learned embeddings are evaluated in distance metric-based tasks, such as one-shot classification under geometric transformations, as well as face verification and retrieval under more realistic visual variability. Our results suggest that orbit sets, suitably computed or observed, can be used for efficient, weakly-supervised learning of semantically relevant image embeddings.This material is based upon work supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216