8,245 research outputs found
Understanding Regularized Spectral Clustering via Graph Conductance
This paper uses the relationship between graph conductance and spectral
clustering to study (i) the failures of spectral clustering and (ii) the
benefits of regularization. The explanation is simple. Sparse and stochastic
graphs create a lot of small trees that are connected to the core of the graph
by only one edge. Graph conductance is sensitive to these noisy `dangling
sets'. Spectral clustering inherits this sensitivity. The second part of the
paper starts from a previously proposed form of regularized spectral clustering
and shows that it is related to the graph conductance on a `regularized graph'.
We call the conductance on the regularized graph CoreCut. Based upon previous
arguments that relate graph conductance to spectral clustering (e.g. Cheeger
inequality), minimizing CoreCut relaxes to regularized spectral clustering.
Simple inspection of CoreCut reveals why it is less sensitive to small cuts in
the graph. Together, these results show that unbalanced partitions from
spectral clustering can be understood as overfitting to noise in the periphery
of a sparse and stochastic graph. Regularization fixes this overfitting. In
addition to this statistical benefit, these results also demonstrate how
regularization can improve the computational speed of spectral clustering. We
provide simulations and data examples to illustrate these results.Comment: 14 pages, 8 figure
An optimization approach to locally-biased graph algorithms
Locally-biased graph algorithms are algorithms that attempt to find local or
small-scale structure in a large data graph. In some cases, this can be
accomplished by adding some sort of locality constraint and calling a
traditional graph algorithm; but more interesting are locally-biased graph
algorithms that compute answers by running a procedure that does not even look
at most of the input graph. This corresponds more closely to what practitioners
from various data science domains do, but it does not correspond well with the
way that algorithmic and statistical theory is typically formulated. Recent
work from several research communities has focused on developing locally-biased
graph algorithms that come with strong complementary algorithmic and
statistical theory and that are useful in practice in downstream data science
applications. We provide a review and overview of this work, highlighting
commonalities between seemingly-different approaches, and highlighting
promising directions for future work.Comment: 19 pages, 13 figure
Hierarchical Clustering with Structural Constraints
Hierarchical clustering is a popular unsupervised data analysis method. For
many real-world applications, we would like to exploit prior information about
the data that imposes constraints on the clustering hierarchy, and is not
captured by the set of features available to the algorithm. This gives rise to
the problem of "hierarchical clustering with structural constraints".
Structural constraints pose major challenges for bottom-up approaches like
average/single linkage and even though they can be naturally incorporated into
top-down divisive algorithms, no formal guarantees exist on the quality of
their output. In this paper, we provide provable approximation guarantees for
two simple top-down algorithms, using a recently introduced optimization
viewpoint of hierarchical clustering with pairwise similarity information
[Dasgupta, 2016]. We show how to find good solutions even in the presence of
conflicting prior information, by formulating a constraint-based regularization
of the objective. We further explore a variation of this objective for
dissimilarity information [Cohen-Addad et al., 2018] and improve upon current
techniques. Finally, we demonstrate our approach on a real dataset for the
taxonomy application.Comment: In Proc. 35th International Conference on Machine Learning (ICML
2018
Spectral Resolution Clustering for Brain Parcellation
We take an image science perspective on the problem of determining brain
network connectivity given functional activity. But adapting the concept of
image resolution to this problem, we provide a new perspective on network
partitioning for individual brain parcellation. The typical goal here is to
determine densely-interconnected subnetworks within a larger network by
choosing the best edges to cut. We instead define these subnetworks as
resolution cells, where highly-correlated activity within the cells makes edge
weights difficult to determine from the data. Subdividing the resolution
estimates into disjoint resolution cells via clustering yields a new variation,
and new perspective, on spectral clustering. This provides insight and
strategies for open questions such as the selection of model order and the
optimal choice of preprocessing steps for functional imaging data. The approach
is demonstrated using functional imaging data, where we find the proposed
approach produces parcellations which are more predictive across multiple scans
versus conventional methods, as well as versus alternative forms of spectral
clustering
An information-theoretic derivation of min-cut based clustering
Min-cut clustering, based on minimizing one of two heuristic cost-functions
proposed by Shi and Malik, has spawned tremendous research, both analytic and
algorithmic, in the graph partitioning and image segmentation communities over
the last decade. It is however unclear if these heuristics can be derived from
a more general principle facilitating generalization to new problem settings.
Motivated by an existing graph partitioning framework, we derive relationships
between optimizing relevance information, as defined in the Information
Bottleneck method, and the regularized cut in a K-partitioned graph. For fast
mixing graphs, we show that the cost functions introduced by Shi and Malik can
be well approximated as the rate of loss of predictive information about the
location of random walkers on the graph. For graphs generated from a stochastic
algorithm designed to model community structure, the optimal information
theoretic partition and the optimal min-cut partition are shown to be the same
with high probability.Comment: 7 pages, 3 figures, two-column, submitted to IEEE Transactions on
Pattern Analysis and Machine Intelligenc
A Semi-supervised Spatial Spectral Regularized Manifold Local Scaling Cut With HGF for Dimensionality Reduction of Hyperspectral Images
Hyperspectral images (HSI) contain a wealth of information over hundreds of
contiguous spectral bands, making it possible to classify materials through
subtle spectral discrepancies. However, the classification of this rich
spectral information is accompanied by the challenges like high dimensionality,
singularity, limited training samples, lack of labeled data samples,
heteroscedasticity and nonlinearity. To address these challenges, we propose a
semi-supervised graph based dimensionality reduction method named
`semi-supervised spatial spectral regularized manifold local scaling cut'
(S3RMLSC). The underlying idea of the proposed method is to exploit the limited
labeled information from both the spectral and spatial domains along with the
abundant unlabeled samples to facilitate the classification task by retaining
the original distribution of the data. In S3RMLSC, a hierarchical guided filter
(HGF) is initially used to smoothen the pixels of the HSI data to preserve the
spatial pixel consistency. This step is followed by the construction of linear
patches from the nonlinear manifold by using the maximal linear patch (MLP)
criterion. Then the inter-patch and intra-patch dissimilarity matrices are
constructed in both spectral and spatial domains by regularized manifold local
scaling cut (RMLSC) and neighboring pixel manifold local scaling cut (NPMLSC)
respectively. Finally, we obtain the projection matrix by optimizing the
updated semi-supervised spatial-spectral between-patch and total-patch
dissimilarity. The effectiveness of the proposed DR algorithm is illustrated
with publicly available real-world HSI datasets
Scalable Spectral Algorithms for Community Detection in Directed Networks
Community detection has been one of the central problems in network studies
and directed network is particularly challenging due to asymmetry among its
links. In this paper, we found that incorporating the direction of links
reveals new perspectives on communities regarding to two different roles,
source and terminal, that a node plays in each community. Intriguingly, such
communities appear to be connected with unique spectral property of the graph
Laplacian of the adjacency matrix and we exploit this connection by using
regularized SVD methods. We propose harvesting algorithms, coupled with
regularized SVDs, that are linearly scalable for efficient identification of
communities in huge directed networks. The proposed algorithm shows great
performance and scalability on benchmark networks in simulations and
successfully recovers communities in real network applications.Comment: Single column, 40 pages, 6 figures and 7 table
A Nonlinear Orthogonal Non-Negative Matrix Factorization Approach to Subspace Clustering
A recent theoretical analysis shows the equivalence between non-negative
matrix factorization (NMF) and spectral clustering based approach to subspace
clustering. As NMF and many of its variants are essentially linear, we
introduce a nonlinear NMF with explicit orthogonality and derive general
kernel-based orthogonal multiplicative update rules to solve the subspace
clustering problem. In nonlinear orthogonal NMF framework, we propose two
subspace clustering algorithms, named kernel-based non-negative subspace
clustering KNSC-Ncut and KNSC-Rcut and establish their connection with spectral
normalized cut and ratio cut clustering. We further extend the nonlinear
orthogonal NMF framework and introduce a graph regularization to obtain a
factorization that respects a local geometric structure of the data after the
nonlinear mapping. The proposed NMF-based approach to subspace clustering takes
into account the nonlinear nature of the manifold, as well as its intrinsic
local geometry, which considerably improves the clustering performance when
compared to the several recently proposed state-of-the-art methods
A Sparse Non-negative Matrix Factorization Framework for Identifying Functional Units of Tongue Behavior from MRI
Muscle coordination patterns of lingual behaviors are synergies generated by
deforming local muscle groups in a variety of ways. Functional units are
functional muscle groups of local structural elements within the tongue that
compress, expand, and move in a cohesive and consistent manner. Identifying the
functional units using tagged-Magnetic Resonance Imaging (MRI) sheds light on
the mechanisms of normal and pathological muscle coordination patterns,
yielding improvement in surgical planning, treatment, or rehabilitation
procedures. Here, to mine this information, we propose a matrix factorization
and probabilistic graphical model framework to produce building blocks and
their associated weighting map using motion quantities extracted from
tagged-MRI. Our tagged-MRI imaging and accurate voxel-level tracking provide
previously unavailable internal tongue motion patterns, thus revealing the
inner workings of the tongue during speech or other lingual behaviors. We then
employ spectral clustering on the weighting map to identify the cohesive
regions defined by the tongue motion that may involve multiple or undocumented
regions. To evaluate our method, we perform a series of experiments. We first
use two-dimensional images and synthetic data to demonstrate the accuracy of
our method. We then use three-dimensional synthetic and \textit{in vivo} tongue
motion data using protrusion and simple speech tasks to identify
subject-specific and data-driven functional units of the tongue in localized
regions.Comment: Accepted at IEEE TMI (https://ieeexplore.ieee.org/document/8467354
Spectral-graph Based Classifications: Linear Regression for Classification and Normalized Radial Basis Function Network
Spectral graph theory has been widely applied in unsupervised and
semi-supervised learning. In this paper, we find for the first time, to our
knowledge, that it also plays a concrete role in supervised classification. It
turns out that two classifiers are inherently related to the theory: linear
regression for classification (LRC) and normalized radial basis function
network (nRBFN), corresponding to linear and nonlinear kernel respectively. The
spectral graph theory provides us with a new insight into a fundamental aspect
of classification: the tradeoff between fitting error and overfitting risk.
With the theory, ideal working conditions for LRC and nRBFN are presented,
which ensure not only zero fitting error but also low overfitting risk. For
quantitative analysis, two concepts, the fitting error and the spectral risk
(indicating overfitting), have been defined. Their bounds for nRBFN and LRC are
derived. A special result shows that the spectral risk of nRBFN is lower
bounded by the number of classes and upper bounded by the size of radial basis.
When the conditions are not met exactly, the classifiers will pursue the
minimum fitting error, running into the risk of overfitting. It turns out that
-norm regularization can be applied to control overfitting. Its effect
is explored under the spectral context. It is found that the two terms in the
-regularized objective are one-one correspondent to the fitting error
and the spectral risk, revealing a tradeoff between the two quantities.
Concerning practical performance, we devise a basis selection strategy to
address the main problem hindering the applications of (n)RBFN. With the
strategy, nRBFN is easy to implement yet flexible. Experiments on 14 benchmark
data sets show the performance of nRBFN is comparable to that of SVM, whereas
the parameter tuning of nRBFN is much easier, leading to reduction of model
selection time
- …