We present a new Self-Supervised Learning (SSL) approach to pre-train
encoders on unlabeled audio data that reduces the need for large amounts of
labeled data for audio and speech classification. Our primary aim is to learn
audio representations that can generalize across a large variety of speech and
non-speech tasks in a low-resource un-labeled audio pre-training setting.
Inspired by the recent success of clustering and contrasting learning paradigms
for SSL-based speech representation learning, we propose SLICER (Symmetrical
Learning of Instance and Cluster-level Efficient Representations), which brings
together the best of both clustering and contrasting learning paradigms. We use
a symmetric loss between latent representations from student and teacher
encoders and simultaneously solve instance and cluster-level contrastive
learning tasks. We obtain cluster representations online by just projecting the
input spectrogram into an output subspace with dimensions equal to the number
of clusters. In addition, we propose a novel mel-spectrogram augmentation
procedure, k-mix, based on mixup, which does not require labels and aids
unsupervised representation learning for audio. Overall, SLICER achieves
state-of-the-art results on the LAPE Benchmark \cite{9868132}, significantly
outperforming DeLoRes-M and other prior approaches, which are pre-trained on
10× larger of unsupervised data. We will make all our codes available on
GitHub.Comment: Submitted to ICASSP 202