OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression

The goal of ordinal regression in computer vision is to estimate the rank number or continue value for a given image. Although we can directly regress this value with a single neuron, a popular baseline is to regard it as a classification task and employ additional constraints trying to make use of the ordinal property. Mathematically, we denote the rank space as $\mathcal{R}=\{r_{j}|0\leq j<C\}$, where $r_{j}$ denotes a rank category and $C$ is the number of ranks categories. During training, we randomly sample a batch of images $\bm{X}=\{\bm{x}_{i}|0\leq i<B\}$ and the associated labels $\bm{Y}=\{\bm{y}_{i}|0\leq i<B\}$, where $B$ represents the batch size and $\bm{y}_{i}\in\{0,1\}^{C}$. The classification baseline treats each rank as a class number. Assuming the rank label for a sample $\bm{x}_{i}$ is $r_{j}$, we construct the one-hot label by setting $y_{i,l}=1$ if $l=j$, otherwise $y_{i,l}=0$, where $l$ is an index and $0\leq l<C$.

The images are first sent to an image encoder to extract image features $\bm{f}=\{\bm{f}_{0},\bm{f}_{1},\ldots,\bm{f}_{B-1}\}$, where $\bm{f}_{i}\in\mathbb{R}^{d}$ is the extracted features of image $\bm{x}_{i}$. We use $d$ to denote the feature dimension. Then the image features are sent to a linear layer with a set of parameters $\bm{W}=[\bm{w}_{0},\bm{w}_{1},\ldots,\bm{w}_{C-1}]^{\top}\in\mathbb{R}^{C\times d}$ and $\bm{b}=[b_{0},b_{1},\ldots,b_{C-1}]\in\mathbb{R}^{C}$:

where $l_{i,j}$ is the predicted logit for rank $r_{j}$ of image $\bm{x}_{i}$. Then the model is optimized with a cross-entropy loss:

Many existing methods develop additional technologies based on the classification formulation, such as label distribution [10, 8, 7], mean-variance loss [32], and so on. However, these methods simply treat each rank as a number and learn the rank concept mainly from the training data leading to the overfitting problem along with unsatisfactory performance. In this paper, we propose a language-powered paradigm, which learns the rank concept from language to address the above issues.

We believe that natural language contains rich prior knowledge. To preserve ordinality, we propose to link the learned rank category to its language concept. To leverage the language power, we employ the text encoder of CLIP [37] as the language model. CLIP follows a vision-language pre-training framework and learns representations from image-text pairs, resulting in a joint vision-language latent space. As the CLIP model shows strong performance in downstream tasks, we implement our method with the CLIP-powered text encoder.

We reformulate the ordinal regression task as an image-language matching problem, which consists of a text encoder and an image encoder. The text encoder is used to extract language features with the fixed CLIP language model. As the parameters of the text encoder are fixed, the text is mapped to a well-learned language latent space, which encodes rich language priors.

To leverage the language priors with the text encoder, we treat the rank categories as words. Then we obtain a set of words for the rank categories $\{[r_{0}],[r_{1}],\ldots,[r_{C-1}]\}$. To obtain the language embedding for each rank, we construct a sentence for these ranks and send the entire sentence to the text encoder. For example, we can construct a sentence of “a person at the age of $[r_{j}]$” for the rank $[r_{j}]$. Then each word of this sentence is mapped to a 512-D word embedding vector.¹¹1Or several 512-D sub-word embeddings due to the usage of BPE [42] All the word embeddings are sent to the text encoder to obtain a text embedding. In this way, each rank category is mapped to a text embedding in the language latent space. We treat the mapped embedding as the prototype of its corresponding rank category. As these prototypes lie on the well-learned language manifold and encode the language concept, we name them as language prototypes. Formally, we use $\{\bm{p}_{j}\in\mathbb{R}^{d}|0\leq j<C-1\}$ to represent the $\ell_{2}$ normalized language prototypes, where $\bm{p}_{i}$ corresponds to the language prototype of the rank $r_{j}$ and $d$ is the feature dimension.

The input images are processed by an image encoder to extract image features. We can employ any popular vision backbones as the image encoder. While the parameters of the text encoder are fixed, we train the entire image encoder to align image features to the language latent space. For a batch of images $\bm{X}$, we denote the $\ell_{2}$ normalized image features as $\bm{I}=\{\bm{I}_{i}\in\mathbb{R}^{d}|0\leq i<B-1\}$, where $d$ is the feature dimension. As an image-language matching framework, we calculate the similarity scores between the two modalities through the inner product and obtain a similarity matrix $\bm{A}=[a_{i,j}]$:

We use image-text contrastive loss to optimize the network, which consists of an image-to-text loss and a text-to-image loss following the practice of CLIP [37]. For image-to-text loss, we first normalize the rows of the matrix $\bm{A}$ using a softmax layer to obtain the matrix $\bm{A}^{\prime}=[a^{\prime}_{i,j}]$:

where $T$ is the temperature parameter. We minimize the discrepancy between the matrix $\bm{A}^{\prime}$ and label $\bm{Y}$ using KL divergence. For text-to-image loss, there may be zero or multiple hits. Therefore we construct the label matrix $\bm{Y}^{\prime\prime}$ by normalizing the non-zero columns of $\bm{Y}$ [49]. Besides, we obtain a column-normalized matrix $\bm{A}^{\prime\prime}=[a^{\prime\prime}_{i,j}]$:

We also employ a KL divergence to minimize the discrepancy between the matrix $\bm{A}^{\prime\prime}$ and the label matrix $\bm{Y}^{\prime\prime}$. In the end, we have the image-text contrastive loss:

As illustrated in [37, 54], different text input, which is known as prompt, may give very different performances. Instead of designing the prompt context, we directly learn $m$ word embeddings $[E]_{0},[E]_{1},\ldots,[E]_{m-1}$ to serve as the prompt context, which is concatenated with the word embeddings of ranks and is sent to the text encoder.

To further strengthen the ordinality of our model and improve its performance, we propose to learn the rank embeddings to preserve the order of the language prototypes in the language latent space. The reason why we cannot directly apply CLIP to regression is that the human-built prompts from the rank labels lack ordinality (see Figure 3), which leads to degraded performance. Inspired by the arithmetic property in the word embeddings space [29, 35, 1], whose typical example is the following embedding arithmetic equation: “King $-$ Man $+$ Woman = Queen”, we choose to maintain the order of rank embeddings to preserve the order of the language prototypes. By explicitly constructing continuous rank embeddings, we can also improve the ordinality of the language prototypes (also see Figure 3). We regard this ordinality as the natural outcome of continuity.

We derive the ordinal rank embeddings from a set of base rank embeddings via interpolation. Suppose we have a set of base rank embeddings $\bm{R}^{\prime}=[\bm{r}^{\prime}_{0},\bm{r}^{\prime}_{1},\dots,\bm{r}^{\prime}_{C^{\prime}-1}]$, where $C^{\prime}$ denotes the number of the embeddings, and $\bm{R}^{\prime}\in\mathbb{R}^{512\times C\prime}$. Note that in practice, $C^{\prime}$ should be a relatively small number compared with $C$ (i.e., $C^{\prime}<<C$). Then the j-th rank embedding could be obtained by:

where $0\leq k<C^{\prime}$, and $\bm{W^{\prime}}=[w^{\prime}_{j,k}]\in\mathbb{R}^{C\times C^{\prime}}$ is the interpolation weights:

where $I(\cdot)$ is the interpolation function, and $|\cdot|$ outputs absolute value. We simply propose two interpolation approaches, which are linear interpolation: $I(w_{j,k})=1-\frac{w_{j,k}}{C-1}$ and inverse-proportion interpolation: $I(w_{j,k})=\frac{1}{w_{j,k}+\epsilon}$. We set $\epsilon$ to 1e-5 for numerical stability. Figure 2 presents the details of the rank prompt of OrdinalCLIP.

Although our method uses a huge language model to exploit the language concepts for training. We can only employ the learned language prototypes during testing. In this way, our method does not bring any additional computational and storage overhead compared with the linear head alternative.

Datasets: The task of facial age estimation is to predict the age of a given facial image. We test our method on the widely-used MORPH II [14] dataset and Adience [21] dataset. Morph II contains 55,134 portraits from 13,618 individuals. Each portrait image is labeled with an age value from 16 to 77. Popular evaluation protocol [39, 43, 23, 22] is adopted in our experiments, which only contains 5,492 images of Caucasian descent to remove the cross-race interference. For general regression, we select 80% of the images for training, and others for testing. Note the long-tailed distribution in the dataset. Besides, the distribution of the test set constructed by random sampling is consistent with that of the training data. Adience contains 26,580 Flickr photos from 2,284 subjects. The label is annotated with eight groups, which are 0-2, 4-6, 8-13, 15-20, 25-32, 38-43, 48-53, and over 60-year-old. We adopt the same five-fold cross-validation protocol as used in [26]. We adopted mean average error (MAE) to measure the absolute differences between the ground truth labels and the predicted ones. In addition, classification accuracy is adopted for Adience. For the detailed experimental settings please refer to Appendix.

Comparison with State-of-the-art methods: We report the results in Table 2. The state-of-the-art methods are usually designed especially for age estimation [50, 46]. DRFs [43] learned a Gaussian mixture model by extending the deep neural decision forests [18] for regression. AVDL [50] leveraged a meta-learning framework to learn an adaptive variance for label distribution. POE [22] proposed a general regression framework with uncertainty modeling and achieved superior performance. Compared with the advanced approaches, zero-shot CLIP only obtains an MAE of 14.45, implying the poor modeling of numerical and ordinal concepts. CoOp scores an MAE of 2.39, illustrating the effectiveness of the language priors. OrdinalCLIP achieves a superior MAE of 2.32 without bells and whistles, resulting from the explicit ordinal constraint. We found similar results in the Adience [26] dataset. Specifically, CoOp attains 60.6% for regression accuracy, which is better than the previous best. Our OrdinalCLIP outperforms previous state-of-the-arts with an accuracy of 61.2% and is on par with the previous best method under the MAE metric of 0.47.

Parameter Analysis: We first analyze the influence of different numbers of base rank embeddings and different interpolation types. With all parameter combinations, the models can all achieve comparable MAEs on MORPH II(Table 3). When varying the number of base rank embeddings, the results improve slightly until a certain sweet point. The best MAE of 2.32 is reported with 3 base ranks. For inverse-proportion one, the best MAE of 2.35 is attained with 7 base ranks. However, its overall performance is sub-optimal compared with the linear interpolation one. In the following experiments, we chose linear interpolation and 3 base ranks as the default hyperparameters.

Ordinality of Learned Language Prototypes: We validate the ordinality of the rank embeddings by measuring the similarity of their corresponding language prototypes. OrdinalCLIP makes predictions by measuring the similarity between the normalized image embedding and each normalized language prototype in the CLIP latent space. Ideally, the structure of the normalized language prototypes should preserve the ordinality - “lie on a line” in the CLIP latent space. This is aligned with human perception and judgment while facing ordinal regression tasks. We compute the similarity matrices followed by Equation 4 with both inputs being language prototypes. The derived matrices are then max-normalized by their maximum similarity. The ordinality score is the percentage of language prototype pairs that obey the ordinal property²²2i.e., $(\sum_{i=0,j=0,i<=j}^{N-2,N-2}\textbf{1}\{a^{\prime}_{i,j}>a^{\prime}_{i,j+1}\})/((N-1)\times N/2)$..

Figure 3 visualizes the ordinality under various situations. The CNN baseline (Figure 3(a)) has the least ordinality score of 49.13%; Half of the language prototype pairs violate the ordinal property and all the prototypes are distributed quite sparsely across the latent space. Figure 3(b) and Figure 3(c) are results computed directly from the CLIP model. Though the initialized context embeddings help to improve ordinality (from 49.71% to 54.84%), almost half of the language prototype pairs violate the ordinal properties. [37] found that CLIP models give degraded predictions facing abstract concepts like numbers. We attribute this to the loss of ordinality in the CLIP latent. Figure 3(d) shows that trained context embeddings from CoOp could further improve the ordinal property with an ordinality score of 59.92%. However, the learned language prototypes are still “twisted” in the CLIP space, leading to degraded predictions. Our OrdinalCLIP (Figure 3(e)) learns smoother language prototypes with a boosted ordinality score of 65.94%, which validates the effectiveness of the learned base rank and their interpolation. Note that the similarity matrices reflect the long-tail property of the dataset. Due to the insufficient samples for the older people, there is a “red blob” in the right-down corner. CLIP and CoOp cannot push the “older” labels away and leave them closer to the well-learned “younger” labels, which may lead to corrupted predictions. Our OrdinalCLIP reinforces the ordinality and as a result, outperforms the above two methods.

Ablation Study: To validate the effectiveness of OrdinalCLIP, we compare OrdinalCLIP with preceded CoOp [54] using different optimization and prompt context initialization strategies. There are four major observations in Table 4.

First, for both CoOP and OrdinalCLIP, solely optimizing either rank embeddings or context embeddings leads to sub-optimal performance, yet training them together results in optimal results. Note that CoOp only optimizes the context embeddings while leaving the rank embeddings untouched as they are initialized with the rank labels. However, after jointly training with the rank embeddings, the performance has been boosted from 2.44 to 2.39. For OrdinalCLIP, that is from 2.37 to 2.36 even with random base rank embeddings, resulting from the explicit ordinality from our method.

Second, the initialized rank embeddings are prone to be degraded, which is another observation validating that the CLIP latent space struggles to represent abstract concepts like numbers [37]. When only optimizing rank embeddings, CoOp obtains an MAE of 2.41 with initialized context prompts. In contrast, our OrdinalCLIP could achieve a better MAE of 2.38 even with all the context and base rank embeddings initialized randomly. We attribute this to the explicit ordinal modeling – the interpolation between the base ranks.

Third, initializing prompts with task-specific context could boost performance, no matter whether it is CoOp or OrdinalCLIP, or which kind of embeddings are optimized. For example, our OrdinalCLIP could achieve the best MAE of 2.32 when training both embeddings and initializing the context.

Last, OrdinalCLIP outperforms CoOp under all settings in the ablation, which further validates the effectiveness of the proposed learned base rank embeddings and their interpolation.

Few-Shot Learning: We validate the generalization ability of OrdinalCLIP for the few-shot regression learning. The full dataset is split into 80% for training and 20% for testing. We use the entire test set for validation and only choose 1/2/4/8/16/32/64 samples in the training set from each class of the labels for training. We report the results in Table 5. We observe the language models consistently outperform the baseline by a large margin. By incorporating the language prior, the MAE is improved by 2.95 compared with baseline under 1 shot setting for CoOp; Our OrdinalCLIP further improves the MAE to 4.94. For other shot settings, we also obverse the same performance gains, which shows the effectiveness of the proposed learned rank embeddings.

Distribution Shift: Another generalization experiment about data distribution shifts is conducted on the MORPH II database. We also use the entire test set of the general regression setting. For the training set, we randomly choose several rank labels, and then randomly discard a large portion of samples in those classes. Table 6 shows the comparison results. We observe that by incorporating language prior, CoOp improves performance by a large margin compared with the baseline. While OrdinalCLIP boosts performance furthermore, illustrating our OrdinalCLIP can better resist the distribution shift of dataset by the explicit ordinal constraint. Specifically, with 40 rank labels selected and 90% of their samples discarded (most severe distribution shift setting), CoOp obtains a 0.35 improvement of MAE compared with baseline, while OrdinalCLIP further attains a performance gain of 0.18.

Dataset. The image aesthetic dataset [41] contains 13,929 available Flickr photos of four categories, i.e, nature, animal, urban, and people. Five absolute rating scales are employed to evaluate the aesthetics quality: “unacceptable”, “flawed”, “ordinary”, “professional”, and “exceptional”. Each image is judged by at least five different examiners and uses the median rank as ground truth. The whole dataset is randomly split into three non-overlapped subsets following [41]. Five-fold cross-validation was used for fair comparisons. We reported the mean values for both MAE and classification accuracy. We detail the experimental settings in Appendix.

Results: Table 7 shows the comparison with the previous SOTA methods. Language priors assist the model to achieve better performance, while zero-shot CLIP still struggles to distinguish ordinal concepts. Our OrdinalCLIP attains an overall MAE of 0.280 and overall accuracy of 72.58%, outperforming the previous best method. The results across all nominal categories either outperform or are on par with previous state-of-the-art. Consistent improvements are observed across all categories compared with baseline and CoOp, showing the effectiveness of learned rank prompts with the ordinal property.

Dataset: The historical image dating dataset [30] is a benchmark for automatically predicting the decade of the historical colored image. There are five-decade categories in the database from the 1930s to the 1970s, where each category contains 265 images. For the general regression setting, the data of each category is divided into three parts: 210 for training, 5 for validation, and the rest for testing. Ten-fold cross-validation is adopted following [26, 30]. We report the mean and standard deviation for both MAE and accuracy metrics. Refer to Appendix for the detailed experimental settings.

Results: We compare our OrdinalCLIP with other state-of-the-art models using the full dataset. Table 8 demonstrates that CoOp improves the average accuracy by 9.34% compared with the baseline method, which again shows the importance of exploiting language priors. Zero-shot CLIP is still suffering from degraded results due to pool ordinal transferability. Our OrdinalCLIP further improves the accuracy by 4.45%. Compared with the previous best, OrdinalCLIP attains a competitive MAE of 0.67 and achieves a new state-of-the-art accuracy of 56.44% with only half of the standard deviation, which validates the effectiveness of the proposed learned rank embeddings.