OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression

This paper presents a language-powered paradigm for ordinal regression. Existing methods usually treat each rank as a category and employ a set of weights to learn these concepts. These methods are easy to overfit and usually attain unsatisfactory performance as the learned concepts are mainly derived from the training set. Recent large pre-trained vision-language models like CLIP have shown impressive performance on various visual tasks. In this paper, we propose to learn the rank concepts from the rich semantic CLIP latent space. Specifically, we reformulate this task as an image-language matching problem with a contrastive objective, which regards labels as text and obtains a language prototype from a text encoder for each rank. While prompt engineering for CLIP is extremely time-consuming, we propose OrdinalCLIP, a differentiable prompting method for adapting CLIP for ordinal regression. OrdinalCLIP consists of learnable context tokens and learnable rank embeddings; The learnable rank embeddings are constructed by explicitly modeling numerical continuity, resulting in well-ordered, compact language prototypes in the CLIP space. Once learned, we can only save the language prototypes and discard the huge language model, resulting in zero additional computational overhead compared with the linear head counterpart. Experimental results show that our paradigm achieves competitive performance in general ordinal regression tasks, and gains improvements in few-shot and distribution shift settings for age estimation. The code is available at https://github.com/xk-huang/OrdinalCLIP.Comment: Accepted by NeurIPS2022. Code is available at https://github.com/xk-huang/OrdinalCLI

An Ordinal Approach to Affective Computing

Both depression prediction and emotion recognition systems are often based on ordinal ground truth due to subjectively annotated datasets. Yet, both have so far been posed as classification or regression problems. These naive approaches have fundamental issues because they are not focused on ordering, unlike ordinal regression, which is the most appropriate for truly ordinal ground truth. Ordinal regression to date offers comparatively fewer, more limited methods when compared with other branches in machine learning, and its usage has been limited to specific research domains. Accordingly, this thesis presents investigations into ordinal approaches for affective computing by describing a consistent framework to understand all ordinal system designs, proposing ordinal systems for large datasets, and introducing tools and principles to select suitable system designs and evaluation methods. First, three learning approaches are compared using the support vector framework to establish the empirical advantages of ordinal regression, which is lacking from the current literature. Results on depression and emotion corpora indicate that ordinal regression with proper tuning can improve existing depression and emotion systems. Ordinal logistic regression (OLR), which is an extension of logistic regression for ordinal scales, contributes to a number of model structures, from which the best structure must be chosen. Exploiting the newly proposed computationally efficient greedy algorithm for model structure selection (GREP), OLR outperformed or was comparable with state-of-the-art depression systems on two benchmark depression speech datasets. Deep learning has dominated many affective computing fields, and hence ordinal deep learning is an attractive prospect. However, it is under-studied even in the machine learning literature, which motivates an in-depth analysis of appropriate network architectures and loss functions. One of the significant outcomes of this analysis is the introduction of RankCNet, a novel ordinal network which utilises a surrogate loss function of rank correlation. Not only the modelling algorithm but the choice of evaluation measure depends on the nature of the ground truth. Rank correlation measures, which are sensitive to ordering, are more apt for ordinal problems than common classification or regression measures that ignore ordering information. Although rank-based evaluation for ordinal problems is not new, so far in affective computing, ordinality of the ground truth has been widely ignored during evaluation. Hence, a systematic analysis in the affective computing context is presented, to provide clarity and encourage careful choice of evaluation measures. Another contribution is a neural network framework with a novel multi-term loss function to assess the ordinality of ordinally-annotated datasets, which can guide the selection of suitable learning and evaluation methods. Experiments on multiple synthetic and affective speech datasets reveal that the proposed system can offer reliable and meaningful predictions about the ordinality of a given dataset. Overall, the novel contributions and findings presented in this thesis not only improve prediction accuracy but also encourage future research towards ordinal affective computing: a different paradigm, but often the most appropriate

Ordinal regression based on data relationship

Ordinal regression is a supervised learning problem which aims to classify instances into ordinal categories. It is different from multi-class classification because there is an ordinal relationship between the categories. Moreover, it is different from metric regression because the target values to be predicted are discrete and the distances between different categories are not defined. Ordinal regression is an active research area because of numerous governmental, commercial and scientific applications, such as quality assessment, disease grading, credit rating, and risk stratification. The main challenge of ordinal regression is to model the ordinal information carried by the labels. Traditionally, there are two angles to tackle the ordinal regression problem: from metric regression perspective and classification perspective. However, most of existing works under both above categories are pointwise methods, in which the relationship between pairs or lists of data points is not explored sufficiently. Furthermore, learning models, especially deep neural network based models, on small datasets is challenging, but many real-world ordinal regression problems are in fact small data problems. The aim of this research is to propose ordinal regression algorithms by exploring data relationship and give consideration to suitability for small datasets and scalability for large datasets. This thesis proposes four approaches for ordinal regression problems based on data relationship. The first approach is a pairwise ordinal regression approach for small datasets. In the training phase, the labeled instances are paired up to train a binary classifier, and the relationship between two data points in each pair is represented by a pairwise kernel. In the testing phase, a decoder algorithm is developed to recover the ordinal information from the binary outputs. By pairing up the training points, the size of the training dataset is squared, which alleviates the lack of training points in small datasets. A proof is presented that if the pairwise kernel fulfills certain properties, the time complexity solving the QP problem can be reduced from O(n^4) to O(n^2) without any loss of accuracy, where n is the number of training points. Motivated by the study of the pairwise relationship, the second approach extends the data relationship representation from pairs to triplets based on deep neural networks. The intuition is to predict rank labels by answering the question: “Is the rank of an input greater than k −1 and smaller than k + 1?”. Therefore, the proposed approach transforms the ordinal regression problem to binary classification problems answering above question and uses triplets with instances from different categories to train deep neural networks such that high-level features describing their ordinal relationship can be extracted automatically. In the testing phase, triplets are formed by a testing instance and other instances with known ranks. A decoder is designed to estimate the rank of the testing instance based on the outputs of the network. Because of the data argumentation by permutation, deep learning can work for ordinal regression even on small datasets. The third proposed approach is a constrained deep neural network for ordinal regression, which aims to automatically extract high-level features for representing intraclass information and interclass ordinal relationship simultaneously. A constrained optimization formulation is proposed for the ordinal regression problem which minimizes the negative loglikelihood for multiple categories constrained by the order between instances. Mathematically, it is equivalent to an unconstrained formulation with a pairwise regularizer. An implementation based on a convolutional neural network framework is proposed to solve the problem such that high-level features can be extracted automatically, and the optimal solution can be learned through the traditional back-propagation method. The proposed pairwise constraints as a regularizer make the algorithm work even on small datasets, and a proposed efficient implementation makes it be scalable for large datasets. Furthermore, an ordinal network architecture is proposed for ordinal regression. The proposed approach embeds the ordinal relationship into the edges between nodes of the same layers in the neural network. Existing deep learning based ordinal regression approaches are implemented by traditional architectures for classification, in which no edges exist between nodes of the same layers. The proposed architecture performs as a latent function mapping the instances to a real line, and the target categories are the intervals on this line which are decided by multiple boundaries. Most significant benefit is that the ordinal network is able to predict the rank labels directly by the outputs of the network without explicit predictions of the multiple boundaries. This research breaks the limits of traditional ordinal regression approaches, and shows the effective and efficient performance of the proposed approaches comparing with the state-of-the-art ordinal regression approaches.Doctor of Philosoph