Optimal Clustering under Uncertainty

Classical clustering algorithms typically either lack an underlying probability framework to make them predictive or focus on parameter estimation rather than defining and minimizing a notion of error. Recent work addresses these issues by developing a probabilistic framework based on the theory of random labeled point processes and characterizing a Bayes clusterer that minimizes the number of misclustered points. The Bayes clusterer is analogous to the Bayes classifier. Whereas determining a Bayes classifier requires full knowledge of the feature-label distribution, deriving a Bayes clusterer requires full knowledge of the point process. When uncertain of the point process, one would like to find a robust clusterer that is optimal over the uncertainty, just as one may find optimal robust classifiers with uncertain feature-label distributions. Herein, we derive an optimal robust clusterer by first finding an effective random point process that incorporates all randomness within its own probabilistic structure and from which a Bayes clusterer can be derived that provides an optimal robust clusterer relative to the uncertainty. This is analogous to the use of effective class-conditional distributions in robust classification. After evaluating the performance of robust clusterers in synthetic mixtures of Gaussians models, we apply the framework to granular imaging, where we make use of the asymptotic granulometric moment theory for granular images to relate robust clustering theory to the application.Comment: 19 pages, 5 eps figures, 1 tabl

An Information Fusion Perspective

A fundamental issue concerned the effectiveness of the Bayesian filter is raised.The observation-only (O2) inference is presented for dynamic state estimation.The "probability of filter benefit" is defined and quantitatively analyzed.Convincing simulations demonstrate that many filters can be easily ineffective. The general solution for dynamic state estimation is to model the system as a hidden Markov process and then employ a recursive estimator of the prediction-correction format (of which the best known is the Bayesian filter) to statistically fuse the time-series observations via models. The performance of the estimator greatly depends on the quality of the statistical mode assumed. In contrast, this paper presents a modeling-free solution, referred to as the observation-only (O2) inference, which infers the state directly from the observations. A Monte Carlo sampling approach is correspondingly proposed for unbiased nonlinear O2 inference. With faster computational speed, the performance of the O2 inference has identified a benchmark to assess the effectiveness of conventional recursive estimators where an estimator is defined as effective only when it outperforms on average the O2 inference (if applicable). It has been quantitatively demonstrated, from the perspective of information fusion, that a prior "biased" information (which inevitably accompanies inaccurate modelling) can be counterproductive for a filter, resulting in an ineffective estimator. Classic state space models have shown that a variety of Kalman filters and particle filters can easily be ineffective (inferior to the O2 inference) in certain situations, although this has been omitted somewhat in the literature

Clustering for filtering: multi-object detection and estimation using multiple/massive sensors

Advanced multi-sensor systems are expected to combat the challenges that arise in object recognition and state estimation in harsh environments with poor or even no prior information, while bringing new challenges mainly related to data fusion and computational burden. Unlike the prevailing Markov-Bayes framework that is the basis of a large variety of stochastic filters and the approximate, we propose a clustering-based methodology for multi-sensor multi-object detection and estimation (MODE), named clustering for filtering (C4F), which abandons unrealistic assumptions with respect to the objects, background and sensors. Rather, based on cluster analysis of the input multi-sensor data, the C4F approach needs no prior knowledge about the latent objects (whether quantity or dynamics), can handle time-varying uncertainties regarding the background and sensors such as noises, clutter and misdetection, and does so computationally fast. This offers an inherently robust and computationally efficient alternative to conventional Markov–Bayes filters for dealing with the scenario with little prior knowledge but rich observation data. Simulations based on representative scenarios of both complete and little prior information have demonstrated the superiority of our C4F approach

Optimal Bayesian Transfer Learning for Classification and Regression

Machine learning methods and algorithms working under the assumption of identically and independently distributed (i.i.d.) data cannot be applicable when dealing with massive data collected from different sources or by various technologies, where heterogeneity of data is inevitable. In such scenarios where we are far from simple homogeneous and uni-modal distributions, we should address the data heterogeneity in a smart way in order to take the best advantages of data coming from different sources. In this dissertation we study two main sources of data heterogeneity, time and domain. We address the time by modeling the dynamics of data and the domain difference by transfer learning. Gene expression data have been used for many years for phenotype classification, for instance, classification of healthy versus cancerous tissues or classification of various types of diseases. The traditional methods use static gene expression data measured in one time point. We propose to take into account the dynamics of gene interactions through time, which can be governed by gene regulatory networks (GRN), and design the classifiers using gene expression trajectories instead of static data. Thanks to recent advanced sequencing technologies such as single-cell, we are now able to look inside a single cell and capture the dynamics of gene expressions. As a result, we design optimal classifiers using single-cell gene expression trajectories, whose dynamics are modeled via Boolean networks with perturbation (BNp). We solve this problem using both expectation maximization (EM) and Bayesian framework and show the great efficacy of these methods over classification via bulk RNA-Seq data. Transfer learning (TL) has recently attracted significant research attention, as it simultaneously learns from different source domains, which have plenty of labeled data, and transfers the relevant knowledge to the target domain with limited labeled data to improve the prediction performance. We study transfer learning with a novel Bayesian viewpoint. Transfer learning appears where we do not have enough data in our target domain to train the machine learning algorithms well but have good amount of data in other relevant source domains. The probability distributions of the source and target domains might be totally different but they share some knowledge underlying the similar tasks between the domains and are related to each other in some sense. The ultimate goal of transfer learning is to find the amount of relatedness between the domains and then transfer the amount of knowledge to the target domain which can help improve the classification task in the data-poor target domain. Negative transfer is the most vital issue in transfer learning and happens when the TL algorithm is not able to detect that the source domain is not related to the target domain for a specific task. For addressing all these issues with a solid theoretical backbone, we propose a novel transfer learning method based on a Bayesian framework. We propose a Bayesian transfer learning framework, where the source and target domains are related through the joint prior distribution of the model parameters. The modeling of joint prior densities enables better understanding of the transferability between domains. Using such an idea, we propose optimal Bayesian transfer learning (OBTL) for both continuous and count data as well as optimal Bayesian transfer regression (OBTR), which are able to optimally transfer the relevant knowledge from a data-rich source domain to a data-poor target domain, whereby improving the classification accuracy in the target domain with limited data

Optimal Bayesian Transfer Learning for Classification and Regression

Machine learning methods and algorithms working under the assumption of identically and independently distributed (i.i.d.) data cannot be applicable when dealing with massive data collected from different sources or by various technologies, where heterogeneity of data is inevitable. In such scenarios where we are far from simple homogeneous and uni-modal distributions, we should address the data heterogeneity in a smart way in order to take the best advantages of data coming from different sources. In this dissertation we study two main sources of data heterogeneity, time and domain. We address the time by modeling the dynamics of data and the domain difference by transfer learning. Gene expression data have been used for many years for phenotype classification, for instance, classification of healthy versus cancerous tissues or classification of various types of diseases. The traditional methods use static gene expression data measured in one time point. We propose to take into account the dynamics of gene interactions through time, which can be governed by gene regulatory networks (GRN), and design the classifiers using gene expression trajectories instead of static data. Thanks to recent advanced sequencing technologies such as single-cell, we are now able to look inside a single cell and capture the dynamics of gene expressions. As a result, we design optimal classifiers using single-cell gene expression trajectories, whose dynamics are modeled via Boolean networks with perturbation (BNp). We solve this problem using both expectation maximization (EM) and Bayesian framework and show the great efficacy of these methods over classification via bulk RNA-Seq data. Transfer learning (TL) has recently attracted significant research attention, as it simultaneously learns from different source domains, which have plenty of labeled data, and transfers the relevant knowledge to the target domain with limited labeled data to improve the prediction performance. We study transfer learning with a novel Bayesian viewpoint. Transfer learning appears where we do not have enough data in our target domain to train the machine learning algorithms well but have good amount of data in other relevant source domains. The probability distributions of the source and target domains might be totally different but they share some knowledge underlying the similar tasks between the domains and are related to each other in some sense. The ultimate goal of transfer learning is to find the amount of relatedness between the domains and then transfer the amount of knowledge to the target domain which can help improve the classification task in the data-poor target domain. Negative transfer is the most vital issue in transfer learning and happens when the TL algorithm is not able to detect that the source domain is not related to the target domain for a specific task. For addressing all these issues with a solid theoretical backbone, we propose a novel transfer learning method based on a Bayesian framework. We propose a Bayesian transfer learning framework, where the source and target domains are related through the joint prior distribution of the model parameters. The modeling of joint prior densities enables better understanding of the transferability between domains. Using such an idea, we propose optimal Bayesian transfer learning (OBTL) for both continuous and count data as well as optimal Bayesian transfer regression (OBTR), which are able to optimally transfer the relevant knowledge from a data-rich source domain to a data-poor target domain, whereby improving the classification accuracy in the target domain with limited data

Optimal Experimental Design in the Context of Objective-Based Uncertainty Quantification

In many real-world engineering applications, model uncertainty is inherent. Largescale dynamical systems cannot be perfectly modeled due to systems complexity, lack of enough training data, perturbation, or noise. Hence, it is often of interest to acquire more data through additional experiments to enhance system model. On the other hand, high cost of experiments and limited operational resources make it necessary to devise a cost-effective plan to conduct experiments. In this dissertation, we are concerned with the problem of prioritizing experiments, called experimental design, aimed at uncertainty reduction in dynamical systems. We take an objective-based view where both uncertainty and modeling objective are taken into account for experimental design. To do so, we utilize the concept of mean objective cost of uncertainty to quantify uncertainty. The first part of this dissertation is devoted to the experimental design for gene regulatory networks. Owing to the complexity of these networks, accurate inference is practically challenging. Moreover, from a translational perspective it is crucial that gene regulatory network uncertainty be quantified and reduced in a manner that pertains to the additional cost of network intervention that it induces. We propose a criterion to rank potential experiments based on the concept of mean objective cost of uncertainty. To lower the computational cost of the experimental design, we also propose a network reduction scheme by introducing a novel cost function that takes into account the disruption in the ranking of potential experiments caused by gene deletion. We investigate the performance of both the optimal and the approximate experimental design methods on synthetic and real gene regulatory networks. In the second part, we turn our attention to canonical expansions. Canonical expansions are convenient representations that can facilitate the study of random processes. We discuss objective-based experimental design in the context of canonical expansions for three major applications: filtering, signal detection, and signal compression. We present the general experimental design framework for linear filtering and specifically solve it for Wiener filtering. Then we focus on Karhunen-Loève expansion to study experimental design for signal detection and signal compression applications when the noise variance and the signal covariance matrix are unknown, respectively. In particular, we find the closed-form solution for the intrinsically Bayesian robust Karhunen-Loève compression which is required for the experimental design in the case of signal compression