16 research outputs found
Recommended from our members
A unified view of high-dimensional bridge regression
In many application areas ranging from bioinformatics to imaging, we are interested in recovering a sparse coefficient in the high-dimensional linear model, when the sample size n is comparable to or less than the dimension p. One of the most popular classes of estimators is the Lq-regularized least squares (LQLS), a.k.a. bridge regression. There have been extensive studies towards understanding the performance of the best subset selection (q=0), LASSO (q=1) and ridge (q=2), three widely known estimators from the LQLS family. This thesis aims at giving a unified view of LQLS for all the non-negative values of q. In contrast to most existing works which obtain order-wise error bounds with loose constants, we derive asymptotically exact error formulas characterized through a series of fixed point equations. A delicate analysis of the fixed point equations enables us to gain fruitful insights into the statistical properties of LQLS across the entire spectrum of Lq-regularization. Our work not only validates the scope of folklore understanding of Lq-minimization, but also provides new insights into high-dimensional statistics as a whole. We will elaborate on our theoretical findings mainly from parameter estimation point of view. At the end of the thesis, we briefly mention bridge regression for variable selection and prediction.
We start by considering the parameter estimation problem and evaluate the performance of LQLS by characterizing the asymptotic mean square error (AMSE). The expression we derive for AMSE does not have explicit forms and hence is not useful in comparing LQLS for different values of q, or providing information in evaluating the effect of relative sample size n/p or the sparsity level of the coefficient. To simplify the expression, we first perform the phase transition (PT) analysis, a widely accepted analysis diagram, of LQLS. Our results reveal some of the limitations and misleading features of the PT framework. To overcome these limitations, we propose the small-error analysis of LQLS. Our new analysis framework not only sheds light on the results of the phase transition analysis, but also describes when phase transition analysis is reliable, and presents a more accurate comparison among different Lq-regularizations.
We then extend our low noise sensitivity analysis to linear models without sparsity structure. Our analysis, as a generalization of phase transition analysis, reveals a clear picture of bridge regression for estimating generic coefficients. Moreover, by a simple transformation we connect our low-noise sensitivity framework to the classical asymptotic regime in which n/p goes to infinity, and give some insightful implications beyond what classical asymptotic analysis of bridge regression can offer.
Furthermore, following the same idea of the new analysis framework, we are able to obtain an explicit characterization of AMSE in the form of second-order expansions under the large noise regime. The expansions provide us some intriguing messages. For example, ridge will outperform LASSO in terms of estimating sparse coefficients when the measurement noise is large.
Finally, we present a short analysis of LQLS, for the purpose of variable selection and prediction. We propose a two-stage variable selection technique based on the LQLS estimators, and describe its superiority and close connection to parameter estimation. For prediction, we illustrate the intricate relation between the tuning parameter selection for optimal in-sample prediction and optimal parameter estimation
Unsupervised Multi-task and Transfer Learning on Gaussian Mixture Models
Unsupervised learning has been widely used in many real-world applications.
One of the simplest and most important unsupervised learning models is the
Gaussian mixture model (GMM). In this work, we study the multi-task learning
problem on GMMs, which aims to leverage potentially similar GMM parameter
structures among tasks to obtain improved learning performance compared to
single-task learning. We propose a multi-task GMM learning procedure based on
the EM algorithm that not only can effectively utilize unknown similarity
between related tasks but is also robust against a fraction of outlier tasks
from arbitrary sources. The proposed procedure is shown to achieve minimax
optimal rate of convergence for both parameter estimation error and the excess
mis-clustering error, in a wide range of regimes. Moreover, we generalize our
approach to tackle the problem of transfer learning for GMMs, where similar
theoretical results are derived. Finally, we demonstrate the effectiveness of
our methods through simulations and a real data analysis. To the best of our
knowledge, this is the first work studying multi-task and transfer learning on
GMMs with theoretical guarantees.Comment: 149 pages, 7 figures, 2 table
Spectral clustering via adaptive layer aggregation for multi-layer networks
One of the fundamental problems in network analysis is detecting community
structure in multi-layer networks, of which each layer represents one type of
edge information among the nodes. We propose integrative spectral clustering
approaches based on effective convex layer aggregations. Our aggregation
methods are strongly motivated by a delicate asymptotic analysis of the
spectral embedding of weighted adjacency matrices and the downstream -means
clustering, in a challenging regime where community detection consistency is
impossible. In fact, the methods are shown to estimate the optimal convex
aggregation, which minimizes the mis-clustering error under some specialized
multi-layer network models. Our analysis further suggests that clustering using
Gaussian mixture models is generally superior to the commonly used -means in
spectral clustering. Extensive numerical studies demonstrate that our adaptive
aggregation techniques, together with Gaussian mixture model clustering, make
the new spectral clustering remarkably competitive compared to several
popularly used methods.Comment: 71 page
Unsupervised Federated Learning: A Federated Gradient EM Algorithm for Heterogeneous Mixture Models with Robustness against Adversarial Attacks
While supervised federated learning approaches have enjoyed significant
success, the domain of unsupervised federated learning remains relatively
underexplored. In this paper, we introduce a novel federated gradient EM
algorithm designed for the unsupervised learning of mixture models with
heterogeneous mixture proportions across tasks. We begin with a comprehensive
finite-sample theory that holds for general mixture models, then apply this
general theory on Gaussian Mixture Models (GMMs) and Mixture of Regressions
(MoRs) to characterize the explicit estimation error of model parameters and
mixture proportions. Our proposed federated gradient EM algorithm demonstrates
several key advantages: adaptability to unknown task similarity, resilience
against adversarial attacks on a small fraction of data sources, protection of
local data privacy, and computational and communication efficiency.Comment: 43 pages, 1 figur