Deep learning with 3D and label geometry

A fine-grained understanding of an image is two-fold: visual understanding and semantic understanding. The former strives to understand the intrinsic properties of the object in the image, whereas the latter aims at associating the diverse objects with certain semantics. All of these form the basis of an in-depth understanding of images. Today’s default architectures of deep convolutional networks have already shown a remarkable ability in capturing the 2D visual appearances of images, and mapping visual content to semantic classes thereafter. However, research on fine-grained image understanding, such as inferring the intrinsic 3D information and more structured semantics, is less explored. In this thesis, we look at the problems by asking "How to better utilize geometry for better image understanding?" In the first part, we research visual image understanding with 3D geometry. We show that it is possible to automatically explain a variety of visual contents in the image with texture-free 3D shapes. Furthermore, we develop a deep learning framework to reliably recover a set of 3D geometric attributes, such as the pose of an object and the surface normal of its shape, from a 2D image. In the second part, we explore label geometry for semantic image understanding. We find that a set of image classification problems have geometrically similar probability spaces. Therefore, label geometry is introduced, unifying one-vs.-rest classification, multi-label classification, and out-of-distribution classification in one framework. Moreover, we show that learned hierarchical label geometries can balance the accuracy and specificity of an image classifier

Recent advances in directional statistics

Mainstream statistical methodology is generally applicable to data observed in Euclidean space. There are, however, numerous contexts of considerable scientific interest in which the natural supports for the data under consideration are Riemannian manifolds like the unit circle, torus, sphere and their extensions. Typically, such data can be represented using one or more directions, and directional statistics is the branch of statistics that deals with their analysis. In this paper we provide a review of the many recent developments in the field since the publication of Mardia and Jupp (1999), still the most comprehensive text on directional statistics. Many of those developments have been stimulated by interesting applications in fields as diverse as astronomy, medicine, genetics, neurology, aeronautics, acoustics, image analysis, text mining, environmetrics, and machine learning. We begin by considering developments for the exploratory analysis of directional data before progressing to distributional models, general approaches to inference, hypothesis testing, regression, nonparametric curve estimation, methods for dimension reduction, classification and clustering, and the modelling of time series, spatial and spatio-temporal data. An overview of currently available software for analysing directional data is also provided, and potential future developments discussed.Comment: 61 page

Learning Likelihoods with Conditional Normalizing Flows

Normalizing Flows (NFs) are able to model complicated distributions p(y) with strong inter-dimensional correlations and high multimodality by transforming a simple base density p(z) through an invertible neural network under the change of variables formula. Such behavior is desirable in multivariate structured prediction tasks, where handcrafted per-pixel loss-based methods inadequately capture strong correlations between output dimensions. We present a study of conditional normalizing flows (CNFs), a class of NFs where the base density to output space mapping is conditioned on an input x, to model conditional densities p(y|x). CNFs are efficient in sampling and inference, they can be trained with a likelihood-based objective, and CNFs, being generative flows, do not suffer from mode collapse or training instabilities. We provide an effective method to train continuous CNFs for binary problems and in particular, we apply these CNFs to super-resolution and vessel segmentation tasks demonstrating competitive performance on standard benchmark datasets in terms of likelihood and conventional metrics.Comment: 18 pages, 8 Tables, 9 Figures, Preprin