Unbiased risk estimation and scoring rules

Stein unbiased risk estimation is generalized twice, from the Gaussian shift model to nonparametric families of smooth densities, and from the quadratic risk to more general divergence type distances. The development relies on a connection with local proper scoring rules.Comment: This is the author's version of a work that was accepted for publication in Comptes rendus Mathematiqu

Bregman divergence as general framework to estimate unnormalized statistical models

We show that the Bregman divergence provides a rich framework to estimate unnormalized statistical models for continuous or discrete random variables, that is, models which do not integrate or sum to one, respectively. We prove that recent estimation methods such as noise-contrastive estimation, ratio matching, and score matching belong to the proposed framework, and explain their interconnection based on supervised learning. Further, we discuss the role of boosting in unsupervised learning

Representation Learning: A Review and New Perspectives

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning

Auto-Encoder based Deep Representation Model for Image Anomaly Detection

Image anomaly detection is to distinguish a small portion of images that are different from the user-defined normal ones. In this work, we focus on auto-encoders based anomaly detection models, which assess the probability of anomaly by measuring reconstruction errors. One of the critical steps in image anomaly detection is to extract robust and distinguishable representations that could separate abnormal patterns from normal ones. However, current auto-encoder based methods fail to extract such distinguishable representations because their optimization objectives are not tailored for this specific task. Besides, the architectures of those models are unable to capture features that are robust to irrelevant distortions but sensitive to abnormal patterns. In this work, two auto-encoder based models are proposed to address the aforementioned issues in optimization objectives and model architectures, respectively. The first model learns to extract distinct representations for abnormal patterns by imposing sparse regularizations on the latent space during the optimization process. This sparse regularization makes the extracted abnormal features unable to be represented as sparse as the normal ones. The second model detects abnormal patterns using Asymmetric Convolution Blocks, which strengthens the crisscross part of the convolutional kernel, making the extracted features less sensitive to geometric transformations. The experimental results demonstrate the superiority of both proposed models over other auto-encoder based anomaly detection models on popular datasets. The proposed methods could also be easily incorporated into most anomaly detection methods in a plug-and-play manner

Étude de techniques d'apprentissage non-supervisé pour l'amélioration de l'entraînement supervisé de modèles connexionnistes

Thèse numérisée par la Division de la gestion de documents et des archives de l'Université de Montréal