Unveiling Backbone Effects in CLIP:
Exploring Representational Synergies and Variances

Large pre-trained models, also known as Foundation Models (FMs), are reshaping the landscape of learning, particularly in computer vision [39, 17, 24, 32] and natural language processing [9, 5, 44]. These models are trained through self-supervised and contrastive learning objectives on internet-scale data, eliminating the need for manual annotations [21, 42]. Contrastive Language Image Pre-training (CLIP) [39] stands as a pioneering contribution in the ongoing research endeavors within this domain, establishing itself as the state-of-the-art across diverse downstream tasks. The training objective is to learn to align text and image representations in an internet-scale dataset. This versatile framework is applicable to a wide range of commonly used architectures, facilitating the training of general image representation learning models. These representation learning models have significantly advanced the field by acting as backbones for other applications, producing state-of-the-art performance in various cross-modal alignment tasks, including but not limited to zero-shot classification [48, 29, 21], cross-modal retrieval [30, 31, 47], and the evaluation of machine-generated captions [27, 22].

Despite the extensive body of prior research on CLIP, there remains a notable gap when it comes to thoroughly examining these image backbones. While we find extensive work evaluating the backbones employed across various tasks to showcase the generalization capabilities of specific methods [14, 29, 49, 13], they typically exhibit general performance improvements when using larger models. This supports the prevailing perception that larger models inherently lead to better representations, implying that they not only learn the same patterns but also more of them. If that is true, increasing the size of the model alone is enough to improve generalization. However, even within the same family of backbones, architectural selections exhibit nuanced inductive biases, capturing distinct patterns by different representations [15]. Nevertheless, there is a conspicuous absence in thoroughly examining the synergies and variances among these backbones as learned by CLIP, particularly considering their shared training dataset and learning objective.

In light of these challenges, this paper conducts a comprehensive empirical investigation to unravel the intricacies of backbones within the CLIP framework. Our qualitative and quantitative evaluations reveal a notable “orthogonality” in the behaviour of these backbones. These backbones adeptly respond to diverse patterns in the input, manifested through different representations, and exhibit distinct confidence levels in their predictions. We observe that employing different normalizations on the representations obtained from these backbones results in varied performance, complementing previous studies [43, 50].

Based on these findings, we consider combining predictions from various backbones for the final predictions. To that end, we first consider a set of baseline methods, following [16]–motivated by calibration techniques [38]– and techniques used in ensemble methods [26, 10]. We illustrate that these methods do not consistently enhance generalization performance. Instead, we introduce a straightforward parametric approach, where a set of temperatures is learned to adjust the logits for each backbone. The inspiration for this temperature parameter stems from the observation of diverse performance outcomes when manipulating the logits in CLIP [43].

To determine these temperatures, we explore two approaches: (1) utilizing a genetic algorithm as a search strategy and identifying a dataset-specific value, and (2) employing a multi-layer perceptron (MLP) to learn and adjust them individually for each sample. We observe that MLP efficiently learns to predict these temperatures with few samples. This method allows for the seamless integration of information from different backbones, contributing to improved predictive outcomes.

In summary, our contributions are as follows:

• •

  We perform extensive experiments across eight datasets, revealing that the representations produced by CLIP backbones and their subsequent predictions exhibit a notable diversity rather than uniformity. For instance, we observe a slight majority, with just over half of the ImageNet test-set correct predictions aligning within the ViT family. The remaining instances only exhibit partial agreement with other backbones.

• •

  We underscore discerning the optimal backbone alone for each individual test instance can yield up to 13% improvement in zero-shot accuracy.

• •

  We demonstrate $L_{2}$ normalization of the representations generally outperforms the alternatives for zero-shot classification, complementing the previous findings of [50].

• •

  Leveraging these insights, we introduce a simple approach to aggregate predictions from multiple backbones, enhancing the performance of the best single model by up to 6.34%. Our neural network combination (NNC) approach demonstrates remarkable adaptability, requiring as few as a single instance to outperform the best of CLIP backbones. This performance boost is achieved solely by employing a scalar per backbone.

Vision Foundation Models have extended the paradigm of pre-training to encompass wide-ranging datasets, ranging from hundreds of millions to billions of images. This expansion was significantly influenced by the introduction of Vision Transformers (ViTs) [11], which have highlighted the feasibility of training Transformers [45] on such massive datasets within the field of computer vision. Subsequently, various large-scale pre-training methods have surfaced in the domain of computer vision [39, 47, 17]. Among these, a notable category of Vision Foundational Models is exemplified by CLIP [39, 42, 12], which specializes in aligning noisy image-text pairs extracted from web sources. The prominence of CLIP stems not only from its scalability but also from its capability to generate meaningful alignments with prompts that facilitate zero-shot classification [13, 49, 29]. There are multiple versions of CLIP trained on different datasets and backbones. Well-studied convolutional backbones like Residual Networks [18] (ResNet) and ViTs [11] with different architectures setup. The focus of our current work centers on studying the difference in image classification of each backbone used to train CLIP.

Model Ensembling Improving performance by combining the output of multiple models is a foundational technique in machine learning, with studies on their effectiveness dating back to at least three decades ago [10, 2, 4, 26] More recent work has shown that foundational ensembling techniques can also be combined with deep neural networks, leading to what is known as deep ensembles [26]. In this context, we find previous work showing that deep ensembles exhibit high accuracy under distribution shift [36], and that higher divergence in training methodology leads to uncorrelated errors and better ensemble accuracy In contrast to techniques utilizing the same model with varied random initializations to achieve orthogonal predictions, our approach involves combining predictions from pre-trained models with distinct backbones. These backbones inherently embody diverse biases and properties [34, 1, 19], facilitating the characterization of different properties associated with each label. By leveraging the inherent variations among these pre-trained backbones, our methodology provides a nuanced and effective means of capturing and utilizing diverse label-related features.

Backbone studies Goldblum et al. [14] undertake a comprehensive comparison involving a diverse array of widely utilized pretrained backbones and randomly initialized baselines. Their evaluation spans extensive downstream tasks, encompassing image classification across diverse domains such as natural, medical, and satellite images. Additionally, the study delves into tasks like object detection and segmentation, evaluates models for out-of-distribution generalization, and assesses performance in image retrieval. Compared to these study, our work focuses specifically on revealing the orthogonality in predictions among the backbones utilized in the CLIP framework for the task of image classification. We present a method that explores and leverages this orthogonality to combine predictions effectively, aiming to boost performance straightforwardly.

CLIP is trained using a self-supervised objective to align the output (i.e., representations) obtained from both language and visual encoders, which we denote as $\phi_{l}$ and $\phi_{v}$ respective. This alignment serves a critical purpose, as we can map textual or visual inputs using their corresponding neural encoders to the same semantic space, assuming the outputs are normalized.

One way to use these models is to consider the joint probability of image $x$ and its corresponding description in natural language $l$ is proportional to a compatibility score between them, which we compute as the inner product between the encoded inputs, as Equation 1 shows below.

To construct a probability distribution from these scores, we use the exponential function, thus allowing us to write the relationship shown below.

In Equation 2, ${\mathbb{X}}$ and ${\mathbb{L}}$ denote the visual and language spaces of possible inputs to the model, respectively. Then, given a downstream multi-class classification task with the label set ${\mathbb{Y}}\subset{\mathbb{L}}$ and $|{\mathbb{Y}}|=C$, we can write the following.

Based on these principles, to perform zero-shot image classification using CLIP, we follow previous work and generate prompts that encapsulate the labels of the target dataset (e.g., an image of {label}, which is a pet). Subsequently, we calculate the similarity between the image feature and these prompts, and the prompt exhibiting the highest similarity is then chosen as the label for the given image. Importantly, we note that while performing zero-shot classification in this fashion, normalization defined in Equation 3 is not required, which in practice means that the step is often not performed in order to reduce computational requirements.

Since our interest is to combine multiple CLIP backbones operating simultaneously in this fashion, our first key insight is to note that Equation 3 is essentially equivalent to computing a softmax over a vector of logit-like values $\bar{{\bm{z}}}\in{\mathbb{R}}^{C}$, whose entries are constructed using on the inner product between a given input $x$ and the set of labels ${\mathbb{Y}}$. Concretely, let $\bar{{z}}_{c}$ denote the cth entry in vector $\bar{{\bm{z}}}$, then we define the following.

We can use the equation above to rewrite Equation 3 simply as $p(y|x)=\text{softmax}(\bar{{\bm{z}}})$, meaning that we can now essentially equate the output of our zero-shot CLIP classification pipeline, to that of a regular classifier, and interpret the output as a probability distribution over the label space ${\mathbb{C}}$.

Since we are interested in combining backbones, we note that a key goal when constructing traditional model ensembles, a technique often used to combine different models, is to increase robustness by increasing the overall confidence of the predictions. In the supervised setting, confidence is generally measured as the probability associated with the predicted label. Though it is expected that a network should provide a calibrated confidence measure in addition to its prediction, it has been shown that deep neural nets are poorly confidence calibrated [16].

In order to shed light into this issue for the case of CLIP models, we first perform an in-depth study of the output of 5 different single-backbone models when in zero-shot classification in 8 benchmark datasets (please see Section 4 for details on backbones and datasets). Concretely, we propose to assess the “orthogonality” of zero-shot predictions by means of two distinct approaches. Firstly, we establish an Oracle prediction that is informed about the appropriate backbone to select for each image sample in the target dataset. While a more sophisticated combination of probabilities may yield enhanced performance, we consider this oracle prediction a pragmatic upper bound for evaluating the potential of combining these backbones. Secondly, we propose to employ Venn diagrams to visually analyze the discrepancies and overlaps in predictions, which allows us to qualitatively characterize the nature of the orthogonality across models.

At this point, it is also important to highlight that the features extracted from the image and language encoders must first inhabit the same embedding space for effective comparison between prompts and the corresponding image. While different methods for doing this exist, a prevalent technique for achieving this alignment involves normalizing the features using the $\textsc{L}_{2}$-Norm. However, Zhou et al. [50] recently argued that a discrepancy exists between the pre-training objective and the subsequent application in downstream tasks and proposed Distribution Normalization (DN), which consists of removing the mean of the data to each of the features,

where $\mu_{x}$ and $\mu_{y}$ are the means of a subset of the target dataset. We, therefore, also study the impact of different normalization techniques across our selected backbones and datasets, comparing against the option of not performing any normalization (UN).

Table 1 summarizes the results of our quantitative results. We first see that while the necessity for normalization for aligning the vision and language spaces is evident, determining the optimal normalization for each backbone and dataset proves less straightforward. In contrast to findings by Zhou et al. [50], our results indicate that, overall L2-Norm consistently demonstrates superior performance accuracy in our experimental setup. Despite the potential insights that could be derived from other properties of Distribution Normalizations [50].

Furthermore, within the ResNet family, it is observed that, for the Flowers dataset, the deeper model, ResNet-101, exhibits inferior performance compared to ResNet-50. Conversely, the ViT models consistently demonstrate an improvement in performance with increasing model size, and ViT-L-14 emerges as the superior performing backbone across all datasets by a significant margin.

Finally, we see that our Oracle consistently outperforms the best individual backbone. Specifically, we see that on FGVC, CUB, and Cars the oracle exhibits improvements of 20.38%, 19.14%, and 13.11%, respectively, for the L2 Normalization cases. It is important to notice that regardless of the normalization method used, a distinct orthogonality is evident when assessing the performance of the proposed Oracle.

In qualitative terms, as shown in Figure 1 via a Venn diagram with the correct predictions of each backbone on ImageNet-1k [8], we see that ViT-L-14 accurately predicts 37,766 images from the test set, with 21,593 correctly predicted across all five backbones (please refer to the Supplementary Material for results of other datasets). Leveraging the informed Oracle increases the overall correct predictions to 42,644 samples of 50,000. Notably, each backbone exhibits correct predictions that are unique to that specific backbone; RN50 and RN101 contribute 435 and 514 exclusive predictions, respectively. ViTs B-32, B-16, and L-14 present 558, 605, and 2,489 distinctive predictions, respectively. We highlight that predictions within each backbone family do not constitute a subset of the best model’s predictions in each family

Based on these empirical findings, we hypothesize that we can effectively improve the classification performance by learning to combine CLIP backbones. Thus, we propose an approach that is directly inspired by previous work on network calibration, but extended to multiple model combinations. Concretely, our framework can be regarded as a set of variations of Platt scaling [38]. In Platt scaling, the logits of a model are used as features for a logistic regression model, which is trained on the validation set to return probabilities. More concretely, given logit scores $z_{i}$ for an example $i$, Platt scaling learns scalar parameters $a,b\in{\mathbb{R}}$ and outputs $\hat{q}_{i}=\sigma(az_{i}+b)$ as the calibrated probability. Parameters $a$ and $b$ can be optimized using the NLL loss over the validation set, but during this time, the parameter of the original model are fixed.

In this work, we adapt the simplest extension of Platt scaling, also known as temperature scaling [16], to our backbone combination setting. In temperature scaling, a single scalar parameter $t>0$ is set for all classes of a given model. The new, calibrated confidence prediction is given by Equation 7, below, where $t$ is called the temperature, $z_{i}$ is the logits, for example $i$ returned by the uncalibrated model.

In the original setting, $t$ is optimized with respect to the NLL on the validation set aiming to reduce the overconfidence of the model on its predictions and produce more reliable predictions, but because the parameter $t$ does not change the maximum of the softmax function, the class prediction $\hat{y}_{i}$ remains unchanged, meaning that the performance of a given model remains the same.

Different from the above, in this work, we aim to jointly optimize a set of temperature parameters $t_{b}$ with $b\in[1,\ldots,B]$ for a set of $B$ CLIP backbones aiming to combine the predictions of the backbones to produce a final prediction that change the confidence of each backbone depending on the confidence of the others and the input. To the best of our knowledge, we are unaware of any prior use in the context of calibrating mixtures of probabilistic models such as the one proposed above. Thus, we aim to improve the classification performance by learning the temperatures $t_{b}$ that weigh the logit $z_{i}^{b}$ for a backbone $b$ and example $i$, using the cross-entropy loss as expressed in the following equation:

Following the original proposals, which rely on data to find the optimal calibration temperature $t$, we propose utilising two existing machine learning techniques to find the set of $B$ parameters that optimize the joint confidence (the temperatures for each backbone considered in the mix).

• •

  We utilize a genetic algorithm (GAC) to find the set of temperatures that better combine the backbones to improve the classification performance by regulating the confidence of each expert (backbone) when compared with the others. We use mutation and crossover operators to find the best temperatures that minimize the classification error.

• •

  We train a simple neural network combination (NNC) (a single-layer MLP) to predict the set of temperatures that best calibrate our backbone mixture. As input, this model receives the concatenated representations obtained by passing the images through the encoder $\phi_{v}$, for each backbone $b\in{\mathbb{B}}$. The neural net directly produces a vector of temperatures ${\bm{t}}\in{\mathbb{R}}^{B}$ and is trained using a cross-entropy loss.

For our experiments, we utilize OpenCLIP [20], an open source implementation of the original CLIP [40]. Our research is predominantly centered on the context of zero-shot learning, where we operate with the fundamental constraint of lacking access to labeled data. As a result, our initial investigation primarily revolves around evaluating the model’s performance in comparison to the baseline zero-shot CLIP paradigm. However, our study also encompasses experimental investigations into the differences in adapted CLIPs using linear probing and limited samples for combining the backbones.

To help characterize the improvements that can be achieved with our proposed approach, we consider common model ensembling techniques as baselines that are grouped into Non-Parametric and Parametric.

In the non-parametric, we consider four approaches commonly used in ensembles:

In this paper, we have undertaken a comprehensive analysis of various backbones within the CLIP framework, specifically focusing on the image classification task. Unlike broader studies that span various downstream tasks, such as those by Goldblum et al. [14], our emphasis lies in understanding and leveraging the unique contributions of each backbone within the CLIP context. Our research unveils a distinctive orthogonality in predictions between backbones of zero-shot CLIP and LinearProbe CLIP, presenting a valuable avenue for enhancing CLIP’s performance through a synergistic combination of these backbones. The introduction of an Oracle highlights this orthogonality, emphasizing the potential of mixing different backbones to optimize performance in image classification tasks. We propose an innovative approach that learns a set of temperatures, refining performance by appropriately weighting the logits from each backbone to yield an enhanced prediction. Notably, our proposed method introduces two efficient techniques, a genetic algorithm and a multi-layer perceptron (MLP), for learning these temperatures, with the MLP showcasing the ability to combine backbones effectively even with limited samples, e.g. one per class. This novel methodology provides a streamlined means of capitalizing on the individual strengths of diverse backbones, ultimately boosting predictive accuracy in the realm of image classification. Our findings contribute to a fine understanding of CLIP’s backbone behaviour and offer a practical strategy for optimizing its performance in real-world applications.