3 research outputs found
A Self-Supervised Approach for Cluster Assessment of High-Dimensional Data
Estimating the number of clusters and underlying cluster structure in a
dataset is a crucial task. Real-world data are often unlabeled, complex and
high-dimensional, which makes it difficult for traditional clustering
algorithms to perform well. In recent years, a matrix reordering based
algorithm, called "visual assessment of tendency" (VAT), and its variants have
attracted many researchers from various domains to estimate the number of
clusters and inherent cluster structure present in the data. However, these
algorithms fail when applied to high-dimensional data due to the curse of
dimensionality, as they rely heavily on the notions of closeness and farness
between data points. To address this issue, we propose a deep-learning based
framework for cluster structure assessment in complex, image datasets. First,
our framework generates representative embeddings for complex data using a
self-supervised deep neural network, and then, these low-dimensional embeddings
are fed to VAT/iVAT algorithms to estimate the underlying cluster structure. In
this process, we ensured not to use any prior knowledge for the number of
clusters (i.e k). We present our results on four real-life image datasets, and
our findings indicate that our framework outperforms state-of-the-art VAT/iVAT
algorithms in terms of clustering accuracy and normalized mutual information
(NMI).Comment: Submitted to IEEE SMC 202
Convergence of ADAM with Constant Step Size in Non-Convex Settings: A Simple Proof
In neural network training, RMSProp and ADAM remain widely favoured
optimization algorithms. One of the keys to their performance lies in selecting
the correct step size, which can significantly influence their effectiveness.
It is worth noting that these algorithms performance can vary considerably,
depending on the chosen step sizes. Additionally, questions about their
theoretical convergence properties continue to be a subject of interest. In
this paper, we theoretically analyze a constant stepsize version of ADAM in the
non-convex setting. We show sufficient conditions for the stepsize to achieve
almost sure asymptotic convergence of the gradients to zero with minimal
assumptions. We also provide runtime bounds for deterministic ADAM to reach
approximate criticality when working with smooth, non-convex functions.Comment: 9 pages including references and appendi
Learning Low-Rank Latent Spaces with Simple Deterministic Autoencoder: Theoretical and Empirical Insights
The autoencoder is an unsupervised learning paradigm that aims to create a
compact latent representation of data by minimizing the reconstruction loss.
However, it tends to overlook the fact that most data (images) are embedded in
a lower-dimensional space, which is crucial for effective data representation.
To address this limitation, we propose a novel approach called Low-Rank
Autoencoder (LoRAE). In LoRAE, we incorporated a low-rank regularizer to
adaptively reconstruct a low-dimensional latent space while preserving the
basic objective of an autoencoder. This helps embed the data in a
lower-dimensional space while preserving important information. It is a simple
autoencoder extension that learns low-rank latent space. Theoretically, we
establish a tighter error bound for our model. Empirically, our model's
superiority shines through various tasks such as image generation and
downstream classification. Both theoretical and practical outcomes highlight
the importance of acquiring low-dimensional embeddings.Comment: Accepted @ IEEE/CVF WACV 202