22,722 research outputs found
Prototypical Contrastive Learning of Unsupervised Representations
This paper presents Prototypical Contrastive Learning (PCL), an unsupervised
representation learning method that addresses the fundamental limitations of
instance-wise contrastive learning. PCL not only learns low-level features for
the task of instance discrimination, but more importantly, it implicitly
encodes semantic structures of the data into the learned embedding space.
Specifically, we introduce prototypes as latent variables to help find the
maximum-likelihood estimation of the network parameters in an
Expectation-Maximization framework. We iteratively perform E-step as finding
the distribution of prototypes via clustering and M-step as optimizing the
network via contrastive learning. We propose ProtoNCE loss, a generalized
version of the InfoNCE loss for contrastive learning, which encourages
representations to be closer to their assigned prototypes. PCL outperforms
state-of-the-art instance-wise contrastive learning methods on multiple
benchmarks with substantial improvement in low-resource transfer learning. Code
and pretrained models are available at https://github.com/salesforce/PCL
Contrastive Prompt Learning-based Code Search based on Interaction Matrix
Code search aims to retrieve the code snippet that highly matches the given
query described in natural language. Recently, many code pre-training
approaches have demonstrated impressive performance on code search. However,
existing code search methods still suffer from two performance constraints:
inadequate semantic representation and the semantic gap between natural
language (NL) and programming language (PL). In this paper, we propose CPLCS, a
contrastive prompt learning-based code search method based on the cross-modal
interaction mechanism. CPLCS comprises:(1) PL-NL contrastive learning, which
learns the semantic matching relationship between PL and NL representations;
(2) a prompt learning design for a dual-encoder structure that can alleviate
the problem of inadequate semantic representation; (3) a cross-modal
interaction mechanism to enhance the fine-grained mapping between NL and PL. We
conduct extensive experiments to evaluate the effectiveness of our approach on
a real-world dataset across six programming languages. The experiment results
demonstrate the efficacy of our approach in improving semantic representation
quality and mapping ability between PL and NL
DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection
Time series anomaly detection is critical for a wide range of applications.
It aims to identify deviant samples from the normal sample distribution in time
series. The most fundamental challenge for this task is to learn a
representation map that enables effective discrimination of anomalies.
Reconstruction-based methods still dominate, but the representation learning
with anomalies might hurt the performance with its large abnormal loss. On the
other hand, contrastive learning aims to find a representation that can clearly
distinguish any instance from the others, which can bring a more natural and
promising representation for time series anomaly detection. In this paper, we
propose DCdetector, a multi-scale dual attention contrastive representation
learning model. DCdetector utilizes a novel dual attention asymmetric design to
create the permutated environment and pure contrastive loss to guide the
learning process, thus learning a permutation invariant representation with
superior discrimination abilities. Extensive experiments show that DCdetector
achieves state-of-the-art results on multiple time series anomaly detection
benchmark datasets. Code is publicly available at
https://github.com/DAMO-DI-ML/KDD2023-DCdetector
Contrastive Audio-Visual Masked Autoencoder
In this paper, we first extend the recent Masked Auto-Encoder (MAE) model
from a single modality to audio-visual multi-modalities. Subsequently, we
propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by combining
contrastive learning and masked data modeling, two major self-supervised
learning frameworks, to learn a joint and coordinated audio-visual
representation. Our experiments show that the contrastive audio-visual
correspondence learning objective not only enables the model to perform
audio-visual retrieval tasks, but also helps the model learn a better joint
representation. As a result, our fully self-supervised pretrained CAV-MAE
achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the
previous best supervised pretrained model on AudioSet in the audio-visual event
classification task. Code and pretrained models are at
https://github.com/yuangongnd/cav-mae.Comment: Accepted at ICLR 2023 as a notable top 25% paper. Code and pretrained
models are at https://github.com/yuangongnd/cav-ma
NCAGC: A Neighborhood Contrast Framework for Attributed Graph Clustering
Attributed graph clustering is one of the most fundamental tasks among graph
learning field, the goal of which is to group nodes with similar
representations into the same cluster without human annotations. Recent studies
based on graph contrastive learning method have achieved remarkable results
when exploit graph-structured data. However, most existing methods 1) do not
directly address the clustering task, since the representation learning and
clustering process are separated; 2) depend too much on data augmentation,
which greatly limits the capability of contrastive learning; 3) ignore the
contrastive message for clustering tasks, which adversely degenerate the
clustering results. In this paper, we propose a Neighborhood Contrast Framework
for Attributed Graph Clustering, namely NCAGC, seeking for conquering the
aforementioned limitations. Specifically, by leveraging the Neighborhood
Contrast Module, the representation of neighbor nodes will be 'push closer' and
become clustering-oriented with the neighborhood contrast loss. Moreover, a
Contrastive Self-Expression Module is built by minimizing the node
representation before and after the self-expression layer to constraint the
learning of self-expression matrix. All the modules of NCAGC are optimized in a
unified framework, so the learned node representation contains
clustering-oriented messages. Extensive experimental results on four attributed
graph datasets demonstrate the promising performance of NCAGC compared with 16
state-of-the-art clustering methods. The code is available at
https://github.com/wangtong627/NCAGC
- …