Global Optimality in Representation Learning

A majority of data processing techniques across a wide range of technical disciplines require a representation of the data that is meaningful for the task at hand in order to succeed. In some cases one has enough prior knowledge about the problem that a fixed transformation of the data or set of features can be pre-calculated, but for most challenging problems with high dimensional data, it is often not known what representation of the data would give the best performance. To address this issue, the field of representation learning seeks to learn meaningful representations directly from data and includes methods such as matrix factorization, tensor factorization, and neural networks. Such techniques have achieved considerable empirical success in many fields, but common to a vast majority of these approaches are the significant disadvantages that 1) the associated optimization problems are typically non-convex due to a multilinear form or other convexity destroying transformation and 2) one is forced to specify the size of the learned representation a priori. This thesis presents a very general framework which allows for the mathematical analysis of a wide range of non-convex representation learning problems. The framework allows the derivation of sufficient conditions to guarantee that a local minimizer of the non-convex optimization problem is a global minimizer and that from any initialization it is possible to find a global minimizer using a purely local descent algorithm. Further, the framework also allows for a wide range of regularization to be incorporated into the model to capture known features of data and to adaptively fit the size of the learned representation to the data instead of defining it a priori. Multiple implications of this work are discussed as they relate to modern practices in deep learning, and the advantages of the approach are demonstrated in applications of automated spatio-temporal segmentation of neural calcium imaging data and reconstructing hyperspectral image volumes from compressed measurements

Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models

The advent of large pre-trained models has brought about a paradigm shift in both visual representation learning and natural language processing. However, clustering unlabeled images, as a fundamental and classic machine learning problem, still lacks effective solution, particularly for large-scale datasets. In this paper, we propose a novel image clustering pipeline that leverages the powerful feature representation of large pre-trained models such as CLIP and cluster images effectively and efficiently at scale. We show that the pre-trained features are significantly more structured by further optimizing the rate reduction objective. The resulting features may significantly improve the clustering accuracy, e.g., from 57\% to 66\% on ImageNet-1k. Furthermore, by leveraging CLIP's image-text binding, we show how the new clustering method leads to a simple yet effective self-labeling algorithm that successfully works on unlabeled large datasets such as MS-COCO and LAION-Aesthetics. We will release the code in https://github.com/LeslieTrue/CPP.Comment: 21 pages, 13 figure

Global Optimality in Representation Learning

A majority of data processing techniques across a wide range of technical disciplines require a representation of the data that is meaningful for the task at hand in order to succeed. In some cases one has enough prior knowledge about the problem that a fixed transformation of the data or set of features can be pre-calculated, but for most challenging problems with high dimensional data, it is often not known what representation of the data would give the best performance. To address this issue, the field of representation learning seeks to learn meaningful representations directly from data and includes methods such as matrix factorization, tensor factorization, and neural networks. Such techniques have achieved considerable empirical success in many fields, but common to a vast majority of these approaches are the significant disadvantages that 1) the associated optimization problems are typically non-convex due to a multilinear form or other convexity destroying transformation and 2) one is forced to specify the size of the learned representation a priori. This thesis presents a very general framework which allows for the mathematical analysis of a wide range of non-convex representation learning problems. The framework allows the derivation of sufficient conditions to guarantee that a local minimizer of the non-convex optimization problem is a global minimizer and that from any initialization it is possible to find a global minimizer using a purely local descent algorithm. Further, the framework also allows for a wide range of regularization to be incorporated into the model to capture known features of data and to adaptively fit the size of the learned representation to the data instead of defining it a priori. Multiple implications of this work are discussed as they relate to modern practices in deep learning, and the advantages of the approach are demonstrated in applications of automated spatio-temporal segmentation of neural calcium imaging data and reconstructing hyperspectral image volumes from compressed measurements