Multi-Source Neural Variational Inference

Learning from multiple sources of information is an important problem in machine-learning research. The key challenges are learning representations and formulating inference methods that take into account the complementarity and redundancy of various information sources. In this paper we formulate a variational autoencoder based multi-source learning framework in which each encoder is conditioned on a different information source. This allows us to relate the sources via the shared latent variables by computing divergence measures between individual source's posterior approximations. We explore a variety of options to learn these encoders and to integrate the beliefs they compute into a consistent posterior approximation. We visualise learned beliefs on a toy dataset and evaluate our methods for learning shared representations and structured output prediction, showing trade-offs of learning separate encoders for each information source. Furthermore, we demonstrate how conflict detection and redundancy can increase robustness of inference in a multi-source setting.Comment: AAAI 2019, Association for the Advancement of Artificial Intelligence (AAAI) 201

Doctor of Philosophy

dissertationScene labeling is the problem of assigning an object label to each pixel of a given image. It is the primary step towards image understanding and unifies object recognition and image segmentation in a single framework. A perfect scene labeling framework detects and densely labels every region and every object that exists in an image. This task is of substantial importance in a wide range of applications in computer vision. Contextual information plays an important role in scene labeling frameworks. A contextual model utilizes the relationships among the objects in a scene to facilitate object detection and image segmentation. Using contextual information in an effective way is one of the main questions that should be answered in any scene labeling framework. In this dissertation, we develop two scene labeling frameworks that rely heavily on contextual information to improve the performance over state-of-the-art methods. The first model, called the multiclass multiscale contextual model (MCMS), uses contextual information from multiple objects and at different scales for learning discriminative models in a supervised setting. The MCMS model incorporates crossobject and interobject information into one probabilistic framework, and thus is able to capture geometrical relationships and dependencies among multiple objects in addition to local information from each single object present in an image. The second model, called the contextual hierarchical model (CHM), learns contextual information in a hierarchy for scene labeling. At each level of the hierarchy, a classifier is trained based on downsampled input images and outputs of previous levels. The CHM then incorporates the resulting multiresolution contextual information into a classifier to segment the input image at original resolution. This training strategy allows for optimization of a joint posterior probability at multiple resolutions through the hierarchy. We demonstrate the performance of CHM on different challenging tasks such as outdoor scene labeling and edge detection in natural images and membrane detection in electron microscopy images. We also introduce two novel classification methods. WNS-AdaBoost speeds up the training of AdaBoost by providing a compact representation of a training set. Disjunctive normal random forest (DNRF) is an ensemble method that is able to learn complex decision boundaries and achieves low generalization error by optimizing a single objective function for each weak classifier in the ensemble. Finally, a segmentation framework is introduced that exploits both shape information and regional statistics to segment irregularly shaped intracellular structures such as mitochondria in electron microscopy images

Disjunctive normal shape Boltzmann machine

Shape Boltzmann machine (a type of Deep Boltzmann machine) is a powerful tool for shape modelling; however, has some drawbacks in representation of local shape parts. Disjunctive Normal Shape Model (DNSM) is a strong shape model that can effectively represent local parts of objects. In this paper, we propose a new shape model based on Shape Boltzmann Machine and Disjunctive Normal Shape Model which we call Disjunctive Normal Shape Boltzmann Machine (DNSBM). DNSBM learns binary distributions of shapes by taking both local and global shape constraints into account using a type of Deep Boltzmann Machine. The samples generated using DNSBM look realistic. Moreover, DNSBM is capable of generating novel samples that differ from training examples by exploiting the local shape representation capability of DNSM. We demonstrate the performance of DNSBM for shape completion on two different data sets in which exploitation of local shape parts is important for capturing the statistical variability of the underlying shape distributions. Experimental results show that DNSBM is a strong model for representing shapes that are composed of local parts

A Survey of Methods for Converting Unstructured Data to CSG Models

The goal of this document is to survey existing methods for recovering CSG representations from unstructured data such as 3D point-clouds or polygon meshes. We review and discuss related topics such as the segmentation and fitting of the input data. We cover techniques from solid modeling and CAD for polyhedron to CSG and B-rep to CSG conversion. We look at approaches coming from program synthesis, evolutionary techniques (such as genetic programming or genetic algorithm), and deep learning methods. Finally, we conclude with a discussion of techniques for the generation of computer programs representing solids (not just CSG models) and higher-level representations (such as, for example, the ones based on sketch and extrusion or feature based operations).Comment: 29 page

Bayesian methods for segmentation of objects from multimodal and complex shape densities using statistical shape priors

In many image segmentation problems involving limited and low-quality data, employing statistical prior information about the shapes of the objects to be segmented can significantly improve the segmentation result. However, defining probability densities in the space of shapes is an open and challenging problem, especially if the object to be segmented comes from a shape density involving multiple modes (classes). In the literature, there are some techniques that exploit nonparametric shape priors to learn multimodal prior densities from a training set. These methods solve the problem of segmenting objects of limited and low-quality to some extent by performing maximum a posteriori (MAP) estimation. However, these methods assume that the boundaries found by using the observed data can provide at least a good initialization for MAP estimation so that convergence to a desired mode of the posterior density is achieved. There are two major problems with this assumption that we focus in this thesis. First, as the data provide less information, these approaches can get stuck at a local optimum which may not be the desired solution. Second, even though a good initialization directs the segmenting curve to a local optimum solution that looks like the desired segmentation, it does not provide a picture of other probable solutions, potentially from different modes of the posterior density, based on the data and the priors. In this thesis, we propose methods for segmentation of objects that come from multimodal posterior densities and suffer from severe noise, occlusion and missing data. The first framework that we propose represents the segmentation problem in terms of the joint posterior density of shapes and features. We incorporate the learned joint shape and feature prior distribution into a maximum a posteri- ori estimation framework for segmentation. In our second proposed framework, we approach the segmentation problem from the approximate Bayesian inference perspective. We propose two different Markov chain Monte Carlo (MCMC) sampling based image segmentation approaches that generates samples from the posterior density. As a final contribution of this thesis, we propose a new shape model that learns binary shape distributions by exploiting local shape priors and the Boltzmann machine. Although the proposed generative shape model has not been used in the context of object segmentation in this thesis, it has great potential to be used for this purpose. The source code of the methods introduced in this thesis will be available in https://github.com/eerdil

Dendritic spine shape analysis based on two-photon microscopy images

Neuronal morphology and function are highly coupled. In particular, dendritic spine morphology is strongly governed by the incoming neuronal activity. Previously, volumes of dendritic spines have been considered as a primary parameter to study spine morphology and gain insight into structure-function coupling. However, this reductionist approach fails to incorporate the broad spine structure repertoire. First step towards integrating the rich spine morphology information into functional coupling is to classify spine shapes into main spine types suggested in the literature. Due to the lack of reliable automated analysis tools, classification is currently performed manually, which is a time-intensive task and prone to subjectivity. Availability of automated spine shape analysis tools can accelerate this process and help neuroscientists understand underlying structure and function relationship. Several studies on spine shape classification have been reported in the literature, however, there is an on-going debate on whether distinct spine shape classes exist or whether spines should be modeled through a continuum of shape variations. Another challenge is the subjectivity and bias that is introduced due to the supervised nature of classification approaches. This thesis focuses on morphological, shape, and appearance features based methods to perform dendritic spine shape analysis using both clustering and classification approaches. We apply manifold learning methods for dendritic spine classification and observe that ISOMAP implicitly computes prominent features suitable for classification purposes. We also apply linear representation based approach for spine classification and conclude that sparse representation provides slightly better classification performance. We propose 2D and 3D morphological features based approach for spine shape analysis and demonstrate the advantage of 3D morphological features. We also use a deep learning based approach for spine classification and show that mid-level features extracted from Convolutional Neural Networks (CNNs) perform as well as hand-crafted features. We propose a kernel density estimation (KDE) based framework for dendritic spine classification. We evaluate our proposed approaches by comparing labels assigned by a neuroscience expert. Our KDE based framework also enables neuroscientists to analyze separability of spine shape classes in the likelihood ratio space, which leads to further insights about the nature of the spine shape analysis problem. Furthermore, we also propose a methodology for unsupervised learning and clustering of spine shapes. In particular, we use x-means to perform cluster analysis that selects the number of clusters automatically using the Bayesian information criterion (BIC). The objective of clustering in this context is two-fold: confirm the hypothesis of some distinct shape classes and discover new natural groups. We observe that although there are many spines which easily fit into the definition of standard shape types (confirming the hypothesis), there are also a significant number of others which do not comply with standard shape types and demonstrate intermediate properties

Articulated motion and deformable objects

This guest editorial introduces the twenty two papers accepted for this Special Issue on Articulated Motion and Deformable Objects (AMDO). They are grouped into four main categories within the field of AMDO: human motion analysis (action/gesture), human pose estimation, deformable shape segmentation, and face analysis. For each of the four topics, a survey of the recent developments in the field is presented. The accepted papers are briefly introduced in the context of this survey. They contribute novel methods, algorithms with improved performance as measured on benchmarking datasets, as well as two new datasets for hand action detection and human posture analysis. The special issue should be of high relevance to the reader interested in AMDO recognition and promote future research directions in the field

Doctor of Philosophy in Computing

dissertationImage segmentation is the problem of partitioning an image into disjoint segments that are perceptually or semantically homogeneous. As one of the most fundamental computer vision problems, image segmentation is used as a primary step for high-level vision tasks, such as object recognition and image understanding, and has even wider applications in interdisciplinary areas, such as longitudinal brain image analysis. Hierarchical models have gained popularity as a key component in image segmentation frameworks. By imposing structures, a hierarchical model can efficiently utilize features from larger image regions and make optimal inference for final segmentation feasible. We develop a hierarchical merge tree (HMT) model for image segmentation. Motivated by the application in large-scale segmentation of neuronal structures in electron microscopy (EM) images, our model provides a compact representation of region merging hypotheses and utilizes higher order information for efficient segmentation inference. Taking advantage of supervised learning, our model is free from parameter tuning and outperforms previous state-of-the-art methods on both two-dimensional (2D) and three-dimensional EM image data sets without any change. We also extend HMT to the hierarchical merge forest (HMF) model. By identifying region correspondences, HMF utilizes inter-section information to correct intra-section errors and improves 2D EM segmentation accuracy. HMT is a generic segmentation model. We demonstrate this by applying it to natural image segmentation problems. We propose a constrained conditional model formulation with a globally optimal inference algorithm for HMT and an iterative merge tree sampling algorithm that significantly improves its performance. Experimental results show our approach achieves state-of-the-art accuracy for object-independent image segmentation. Finally, we propose a semi-supervised HMT (SSHMT) model to reduce the high demand for labeled data by supervised learning. We introduce a differentiable unsupervised loss term that enforces consistent boundary predictions and develop a Bayesian learning model that combines supervised and unsupervised information. We show that with a very small amount of labeled data, SSHMT consistently performs close to the supervised HMT with full labeled data sets and significantly outperforms HMT trained with the same labeled subsets