9 research outputs found
C3: Cross-instance guided Contrastive Clustering
Clustering is the task of gathering similar data samples into clusters
without using any predefined labels. It has been widely studied in machine
learning literature, and recent advancements in deep learning have revived
interest in this field. Contrastive clustering (CC) models are a staple of deep
clustering in which positive and negative pairs of each data instance are
generated through data augmentation. CC models aim to learn a feature space
where instance-level and cluster-level representations of positive pairs are
grouped together. Despite improving the SOTA, these algorithms ignore the
cross-instance patterns, which carry essential information for improving
clustering performance. This increases the false-negative-pair rate of the
model while decreasing its true-positive-pair rate. In this paper, we propose a
novel contrastive clustering method, Cross-instance guided Contrastive
Clustering (C3), that considers the cross-sample relationships to increase the
number of positive pairs and mitigate the impact of false negative, noise, and
anomaly sample on the learned representation of data. In particular, we define
a new loss function that identifies similar instances using the instance-level
representation and encourages them to aggregate together. Moreover, we propose
a novel weighting method to select negative samples in a more efficient way.
Extensive experimental evaluations show that our proposed method can outperform
state-of-the-art algorithms on benchmark computer vision datasets: we improve
the clustering accuracy by 6.6%, 3.3%, 5.0%, 1.3% and 0.3% on CIFAR-10,
CIFAR-100, ImageNet-10, ImageNet-Dogs, and Tiny-ImageNet.Comment: 10 pages, 8 Figures, 1 Table
Contrastive Clustering
In this paper, we propose a one-stage online clustering method called
Contrastive Clustering (CC) which explicitly performs the instance- and
cluster-level contrastive learning. To be specific, for a given dataset, the
positive and negative instance pairs are constructed through data augmentations
and then projected into a feature space. Therein, the instance- and
cluster-level contrastive learning are respectively conducted in the row and
column space by maximizing the similarities of positive pairs while minimizing
those of negative ones. Our key observation is that the rows of the feature
matrix could be regarded as soft labels of instances, and accordingly the
columns could be further regarded as cluster representations. By simultaneously
optimizing the instance- and cluster-level contrastive loss, the model jointly
learns representations and cluster assignments in an end-to-end manner.
Extensive experimental results show that CC remarkably outperforms 17
competitive clustering methods on six challenging image benchmarks. In
particular, CC achieves an NMI of 0.705 (0.431) on the CIFAR-10 (CIFAR-100)
dataset, which is an up to 19\% (39\%) performance improvement compared with
the best baseline
Information Maximization Clustering via Multi-View Self-Labelling
Image clustering is a particularly challenging computer vision task, which
aims to generate annotations without human supervision. Recent advances focus
on the use of self-supervised learning strategies in image clustering, by first
learning valuable semantics and then clustering the image representations.
These multiple-phase algorithms, however, increase the computational time and
their final performance is reliant on the first stage. By extending the
self-supervised approach, we propose a novel single-phase clustering method
that simultaneously learns meaningful representations and assigns the
corresponding annotations. This is achieved by integrating a discrete
representation into the self-supervised paradigm through a classifier net.
Specifically, the proposed clustering objective employs mutual information, and
maximizes the dependency between the integrated discrete representation and a
discrete probability distribution. The discrete probability distribution is
derived though the self-supervised process by comparing the learnt latent
representation with a set of trainable prototypes. To enhance the learning
performance of the classifier, we jointly apply the mutual information across
multi-crop views. Our empirical results show that the proposed framework
outperforms state-of-the-art techniques with the average accuracy of 89.1% and
49.0%, respectively, on CIFAR-10 and CIFAR-100/20 datasets. Finally, the
proposed method also demonstrates attractive robustness to parameter settings,
making it ready to be applicable to other datasets
Temporal Action Segmentation: An Analysis of Modern Techniques
Temporal action segmentation (TAS) in videos aims at densely identifying
video frames in minutes-long videos with multiple action classes. As a
long-range video understanding task, researchers have developed an extended
collection of methods and examined their performance using various benchmarks.
Despite the rapid growth of TAS techniques in recent years, no systematic
survey has been conducted in these sectors. This survey analyzes and summarizes
the most significant contributions and trends. In particular, we first examine
the task definition, common benchmarks, types of supervision, and prevalent
evaluation measures. In addition, we systematically investigate two essential
techniques of this topic, i.e., frame representation and temporal modeling,
which have been studied extensively in the literature. We then conduct a
thorough review of existing TAS works categorized by their levels of
supervision and conclude our survey by identifying and emphasizing several
research gaps. In addition, we have curated a list of TAS resources, which is
available at https://github.com/nus-cvml/awesome-temporal-action-segmentation.Comment: 19 pages, 9 figures, 8 table
Deep Clustering: A Comprehensive Survey
Cluster analysis plays an indispensable role in machine learning and data
mining. Learning a good data representation is crucial for clustering
algorithms. Recently, deep clustering, which can learn clustering-friendly
representations using deep neural networks, has been broadly applied in a wide
range of clustering tasks. Existing surveys for deep clustering mainly focus on
the single-view fields and the network architectures, ignoring the complex
application scenarios of clustering. To address this issue, in this paper we
provide a comprehensive survey for deep clustering in views of data sources.
With different data sources and initial conditions, we systematically
distinguish the clustering methods in terms of methodology, prior knowledge,
and architecture. Concretely, deep clustering methods are introduced according
to four categories, i.e., traditional single-view deep clustering,
semi-supervised deep clustering, deep multi-view clustering, and deep transfer
clustering. Finally, we discuss the open challenges and potential future
opportunities in different fields of deep clustering
λμ’ , μ΄μ’ , κ·Έλ¦¬κ³ λ무 ννμ κ·Έλνλ₯Ό μν λΉμ§λ νν νμ΅
νμλ
Όλ¬Έ(λ°μ¬) -- μμΈλνκ΅λνμ : 곡과λν μ κΈ°Β·μ 보곡νλΆ, 2022. 8. μ΅μ§μ.κ·Έλν λ°μ΄ν°μ λν λΉμ§λ νν νμ΅μ λͺ©μ μ κ·Έλνμ ꡬ쑰μ λ
Έλμ μμ±μ μ λ°μνλ μ μ©ν λ
Έλ λ¨μ νΉμ κ·Έλν λ¨μμ λ²‘ν° νν ννμ νμ΅νλ κ²μ΄λ€. μ΅κ·Ό, κ·Έλν λ°μ΄ν°μ λν΄ κ°λ ₯ν νν νμ΅ λ₯λ ₯μ κ°μΆ κ·Έλν μ κ²½λ§μ νμ©ν λΉμ§λ κ·Έλν νν νμ΅ λͺ¨λΈμ μ€κ³κ° μ£Όλͺ©μ λ°κ³ μλ€. λ§μ λ°©λ²λ€μ ν μ’
λ₯μ μ£μ§μ ν μ’
λ₯μ λ
Έλκ° μ‘΄μ¬νλ λμ’
κ·Έλνμ λν νμ΅μ μ§μ€μ νλ€. νμ§λ§ μ΄ μΈμμ μλ§μ μ’
λ₯μ κ΄κ³κ° μ‘΄μ¬νκΈ° λλ¬Έμ, κ·Έλν λν ꡬ쑰μ , μλ―Έλ‘ μ μμ±μ ν΅ν΄ λ€μν μ’
λ₯λ‘ λΆλ₯ν μ μλ€. κ·Έλμ, κ·Έλνλ‘λΆν° μ μ©ν ννμ νμ΅νκΈ° μν΄μλ λΉμ§λ νμ΅ νλ μμν¬λ μ
λ ₯ κ·Έλνμ νΉμ§μ μ λλ‘ κ³ λ €ν΄μΌλ§ νλ€. λ³Έ νμλ
Όλ¬Έμμ μ°λ¦¬λ λ리 μ ν μ μλ μΈκ°μ§ κ·Έλν κ΅¬μ‘°μΈ λμ’
κ·Έλν, νΈλ¦¬ ννμ κ·Έλν, κ·Έλ¦¬κ³ μ΄μ’
κ·Έλνμ λν κ·Έλν μ κ²½λ§μ νμ©νλ λΉμ§λ νμ΅ λͺ¨λΈλ€μ μ μνλ€.
μ²μμΌλ‘, μ°λ¦¬λ λμ’
κ·Έλνμ λ
Έλμ λνμ¬ μ μ°¨μ ννμ νμ΅νλ κ·Έλν 컨볼루μ
μ€ν μΈμ½λ λͺ¨λΈμ μ μνλ€. κΈ°μ‘΄μ κ·Έλν μ€ν μΈμ½λλ ꡬ쑰μ μ μ²΄κ° νμ΅μ΄ λΆκ°λ₯ν΄μ μ νμ μΈ νν νμ΅ λ₯λ ₯μ κ°μ§ μ μλ λ°λ©΄μ, μ μνλ μ€ν μΈμ½λλ λ
Έλμ νΌμ³λ₯Ό 볡μνλ©°,ꡬ쑰μ μ μ²΄κ° νμ΅μ΄ κ°λ₯νλ€. λ
Έλμ νΌμ³λ₯Ό 볡μνκΈ° μν΄μ, μ°λ¦¬λ μΈμ½λ λΆλΆμ μν μ΄ μ΄μν λ
ΈλλΌλ¦¬ μ μ¬ν ννμ κ°μ§κ² νλ λΌνλΌμμ μ€λ¬΄λ©μ΄λΌλ κ²μ μ£Όλͺ©νμ¬ λμ½λ λΆλΆμμλ μ΄μ λ
Έλμ ννκ³Ό λ©μ΄μ§κ² νλ λΌνλΌμμ μ€νλμ νλλ‘ μ€κ³νμλ€. λν λΌνλΌμμ μ€νλμ κ·Έλλ‘ μ μ©νλ©΄ λΆμμ μ±μ μ λ°ν μ μκΈ° λλ¬Έμ, μ£μ§μ κ°μ€μΉ κ°μ μμ κ°μ μ€ μ μλ λΆνΈν κ·Έλνλ₯Ό νμ©νμ¬ μμ μ μΈ λΌνλΌμμ μ€νλμ ννλ₯Ό μ μνμλ€. λμ’
κ·Έλνμ λν λ
Έλ ν΄λ¬μ€ν°λ§κ³Ό λ§ν¬ μμΈ‘ μ€νμ ν΅νμ¬ μ μνλ λ°©λ²μ΄ μμ μ μΌλ‘ μ°μν μ±λ₯μ 보μμ νμΈνμλ€.
λμ§Έλ‘, μ°λ¦¬λ νΈλ¦¬μ ννλ₯Ό κ°μ§λ κ³μΈ΅μ μΈ κ΄κ³λ₯Ό κ°μ§κ³ μλ κ·Έλνμ λ
Έλ ννμ μ ννκ² νμ΅νκΈ° μνμ¬ μ곑μ 곡κ°μμ λμνλ μ€ν μΈμ½λ λͺ¨λΈμ μ μνλ€. μ ν΄λ¦¬λμΈ κ³΅κ°μ νΈλ¦¬λ₯Ό μ¬μνκΈ°μ λΆμ μ νλ€λ μ΅κ·Όμ λΆμμ ν΅νμ¬, μ곑μ 곡κ°μμ κ·Έλν μ κ²½λ§μ λ μ΄μ΄λ₯Ό νμ©νμ¬ λ
Έλμ μ μ°¨μ ννμ νμ΅νκ² λλ€. μ΄ λ, κ·Έλν μ κ²½λ§μ΄ μ곑μ κΈ°ννμμ κ³μΈ΅ μ 보λ₯Ό λ΄κ³ μλ 거리μ κ°μ νμ©νμ¬ λ
Έλμ μ΄μμ¬μ΄μ μ€μλλ₯Ό νμ©νλλ‘ μ€κ³νμλ€. μ°λ¦¬λ λ
Όλ¬Έ μΈμ© κ΄κ³ λ€νΈμν¬, κ³ν΅λ, μ΄λ―Έμ§ μ¬μ΄μ λ€νΈμν¬λ±μ λν΄ μ μν λͺ¨λΈμ μ μ©νμ¬ λ
Έλ ν΄λ¬μ€ν°λ§κ³Ό λ§ν¬ μμΈ‘ μ€νμ νμμΌλ©°, νΈλ¦¬μ ννλ₯Ό κ°μ§λ κ·Έλνμ λν΄μ μ μν λͺ¨λΈμ΄ μ ν΄λ¦¬λμΈ κ³΅κ°μμ μννλ λͺ¨λΈμ λΉν΄ ν₯μλ μ±λ₯μ 보μλ€λ κ²μ νμΈνμλ€.
λ§μ§λ§μΌλ‘, μ°λ¦¬λ μ¬λ¬ μ’
λ₯μ λ
Έλμ μ£μ§λ₯Ό κ°μ§λ μ΄μ’
κ·Έλνμ λν λμ‘° νμ΅ λͺ¨λΈμ μ μνλ€. μ°λ¦¬λ κΈ°μ‘΄μ λ°©λ²λ€μ΄ νμ΅νκΈ° μ΄μ μ μΆ©λΆν λλ©μΈ μ§μμ μ¬μ©νμ¬ μ€κ³ν λ©νν¨μ€λ λ©νκ·Έλνμ μμ‘΄νλ€λ λ¨μ κ³Ό λ§μ μ΄μ’
κ·Έλνμ μ£μ§κ° λ€λ₯Έ λ
Έλ μ’
λ₯μ¬μ΄μ κ΄κ³μ μ§μ€νκ³ μλ€λ μ μ μ£Όλͺ©νμλ€. μ΄λ₯Ό ν΅ν΄ μ°λ¦¬λ μ¬μ κ³Όμ μ΄ νμμμΌλ©° λ€λ₯Έ μ’
λ₯ μ¬μ΄μ κ΄κ³μ λνμ¬ κ°μ μ’
λ₯ μ¬μ΄μ κ΄κ³λ λμμ ν¨μ¨μ μΌλ‘ νμ΅νκ² νλ λ©νλ
ΈλλΌλ κ°λ
μ μ μνμλ€. λν λ©νλ
Έλλ₯Ό κΈ°λ°μΌλ‘νλ κ·Έλν μ κ²½λ§κ³Ό λμ‘° νμ΅ λͺ¨λΈμ μ μνμλ€. μ°λ¦¬λ μ μν λͺ¨λΈμ λ©νν¨μ€λ₯Ό μ¬μ©νλ μ΄μ’
κ·Έλν νμ΅ λͺ¨λΈκ³Ό λ
Έλ ν΄λ¬μ€ν°λ§ λ±μ μ€ν μ±λ₯μΌλ‘ λΉκ΅ν΄λ³΄μμ λ, λΉλ±νκ±°λ λμ μ±λ₯μ 보μμμ νμΈνμλ€.The goal of unsupervised graph representation learning is extracting useful node-wise or graph-wise vector representation that is aware of the intrinsic structures of the graph and its attributes. These days, designing methodology of unsupervised graph representation learning based on graph neural networks has growing attention due to their powerful representation ability. Many methods are focused on a homogeneous graph that is a network with a single type of node and a single type of edge. However, as many types of relationships exist in this world, graphs can also be classified into various types by structural and semantic properties. For this reason, to learn useful representations from graphs, the unsupervised learning framework must consider the characteristics of the input graph. In this dissertation, we focus on designing unsupervised learning models using graph neural networks for three graph structures that are widely available: homogeneous graphs, tree-like graphs, and heterogeneous graphs.
First, we propose a symmetric graph convolutional autoencoder which produces a low-dimensional latent representation from a homogeneous graph. In contrast to the existing graph autoencoders with asymmetric decoder parts, the proposed autoencoder has a newly designed decoder which builds a completely symmetric autoencoder form. For the reconstruction of node features, the decoder is designed based on Laplacian sharpening as the counterpart of Laplacian smoothing of the encoder, which allows utilizing the graph structure in the whole processes of the proposed autoencoder architecture. In order to prevent the numerical instability of the network caused by the Laplacian sharpening introduction, we further propose a new numerically stable form of the Laplacian sharpening by incorporating the signed graphs. The experimental results of clustering, link prediction and visualization tasks on homogeneous graphs strongly support that the proposed model is stable and outperforms various state-of-the-art algorithms.
Second, we analyze how unsupervised tasks can benefit from learned representations in hyperbolic space. To explore how well the hierarchical structure of unlabeled data can be represented in hyperbolic spaces, we design a novel hyperbolic message passing autoencoder whose overall auto-encoding is performed in hyperbolic space. The proposed model conducts auto-encoding the networks via fully utilizing hyperbolic geometry in message passing. Through extensive quantitative and qualitative analyses, we validate the properties and benefits of the unsupervised hyperbolic representations of tree-like graphs.
Third, we propose the novel concept of metanode for message passing to learn both heterogeneous and homogeneous relationships between any two nodes without meta-paths and meta-graphs. Unlike conventional methods, metanodes do not require a predetermined step to manipulate the given relations between different types to enrich relational information. Going one step further, we propose a metanode-based message passing layer and a contrastive learning model using the proposed layer. In our experiments, we show the competitive performance of the proposed metanode-based message passing method on node clustering and node classification tasks, when compared to state-of-the-art methods for message passing networks for heterogeneous graphs.1 Introduction 1
2 Representation Learning on Graph-Structured Data 4
2.1 Basic Introduction 4
2.1.1 Notations 5
2.2 Traditional Approaches 5
2.2.1 Graph Statistic 5
2.2.2 Neighborhood Overlap 7
2.2.3 Graph Kernel 9
2.2.4 Spectral Approaches 10
2.3 Node Embeddings I: Factorization and Random Walks 15
2.3.1 Factorization-based Methods 15
2.3.2 Random Walk-based Methods 16
2.4 Node Embeddings II: Graph Neural Networks 17
2.4.1 Overview of Framework 17
2.4.2 Representative Models 18
2.5 Learning in Unsupervised Environments 21
2.5.1 Predictive Coding 21
2.5.2 Contrastive Coding 22
2.6 Applications 24
2.6.1 Classifications 24
2.6.2 Link Prediction 26
3 Autoencoder Architecture for Homogeneous Graphs 27
3.1 Overview 27
3.2 Preliminaries 30
3.2.1 Spectral Convolution on Graphs 30
3.2.2 Laplacian Smoothing 32
3.3 Methodology 33
3.3.1 Laplacian Sharpening 33
3.3.2 Numerically Stable Laplacian Sharpening 34
3.3.3 Subspace Clustering Cost for Image Clustering 37
3.3.4 Training 39
3.4 Experiments 40
3.4.1 Datasets 40
3.4.2 Experimental Settings 42
3.4.3 Comparing Methods 42
3.4.4 Node Clustering 43
3.4.5 Image Clustering 45
3.4.6 Ablation Studies 46
3.4.7 Link Prediction 47
3.4.8 Visualization 47
3.5 Summary 49
4 Autoencoder Architecture for Tree-like Graphs 50
4.1 Overview 50
4.2 Preliminaries 52
4.2.1 Hyperbolic Embeddings 52
4.2.2 Hyperbolic Geometry 53
4.3 Methodology 55
4.3.1 Geometry-Aware Message Passing 56
4.3.2 Nonlinear Activation 57
4.3.3 Loss Function 58
4.4 Experiments 58
4.4.1 Datasets 59
4.4.2 Compared Methods 61
4.4.3 Experimental Details 62
4.4.4 Node Clustering and Link Prediction 64
4.4.5 Image Clustering 66
4.4.6 Structure-Aware Unsupervised Embeddings 68
4.4.7 Hyperbolic Distance to Filter Training Samples 71
4.4.8 Ablation Studies 74
4.5 Further Discussions 75
4.5.1 Connection to Contrastive Learning 75
4.5.2 Failure Cases of Hyperbolic Embedding Spaces 75
4.6 Summary 77
5 Contrastive Learning for Heterogeneous Graphs 78
5.1 Overview 78
5.2 Preliminaries 82
5.2.1 Meta-path 82
5.2.2 Representation Learning on Heterogeneous Graphs 82
5.2.3 Contrastive methods for Heterogeneous Graphs 83
5.3 Methodology 84
5.3.1 Definitions 84
5.3.2 Metanode-based Message Passing Layer 86
5.3.3 Contrastive Learning Framework 88
5.4 Experiments 89
5.4.1 Experimental Details 90
5.4.2 Node Classification 94
5.4.3 Node Clustering 96
5.4.4 Visualization 96
5.4.5 Effectiveness of Metanodes 97
5.5 Summary 99
6 Conclusions 101λ°