Improved Visual Fine-tuning with Natural Language Supervision

A generic visual pre-training paradigm includes both supervised and self-supervised approaches. Supervised pre-training requires a large number of labeled data and can learn rich semantic information for fine-tuning [30, 34, 42].

To eliminate the cost of labeling, self-supervised learning is developed to obtain pre-trained models from unlabeled data. Many pretext tasks were proposed for effective learning, e.g., instance discrimination that considers each instance as an individual class and optimizes random augmentations from the same instance [6, 18], cluster discrimination that explores the relationship between different instances [5, 32] and masked image modeling that leverages information within each image [16]. Moreover, [43] demonstrates that self-supervised pre-training improves supervised pre-training with strong data augmentations.

After pre-training, fine-tuning aims to facilitate the downstream tasks by transferring semantic information in the pre-trained models [21]. One main challenge for fine-tuning is from catastrophic forgetting that the fine-tuned model over-fits target data and becomes far away from the pre-trained model [9, 24, 25]. To mitigate the problem, [24] constrains the distance between the fine-tuned model and its pre-trained counterpart. [25] proposes a data-dependent strategy that limits the distance between representations of target data before and after fine-tuning.

Besides catastrophic forgetting, another challenge is from the bias in pre-trained models. Due to the gap between learning tasks and data for pre-training and fine-tuning, the bias in the pre-trained model will degenerate the performance on the target task [30]. Without any side information, the bias is hard to be eliminated. Recent work [26] shows that the bias can be reduced by fine-tuning the target data with a related subset selected from the pre-training data for the target task. In this work, we investigate another form of side information from text supervision. Compared with the information from pre-training data, the text supervision is easier to be accessed with a pre-trained text encoder.

First, we demonstrate that the bias in a pre-trained model will be preserved in the classifier $W$ after conventional fine-tuning. Moreover, even with a large learning rate solely for the classifier during fine-tuning, the obtained $W$ is still close to that implied by the pre-trained model.

Given fixed representations from a pre-trained model, the sub-problem of optimizing the classifier with cross entropy loss is convex and a global optimum can be obtained. Let $W^{0}=[\mathbf{w}_{1}^{0},\dots,\mathbf{w}_{C}^{0}]$ be the optimal solution as

where $\mathbf{x}_{i}^{0}=f_{0}(x_{i})$ is extracted from the pre-trained model $f_{0}(\cdot)$, $\theta_{0}$ is the parameter of $f_{0}(\cdot)$, and $\mathbf{w}_{j}^{0}$ denotes the proxy for the $j$-th class obtained from the pre-trained model.

After fine-tuning $T$ iterations with stochastic gradient descent (SGD), we can observe $(\theta^{T},W^{T})$ for the problem

where $W^{T}=[\mathbf{w}_{1}^{T},\dots,\mathbf{w}_{C}^{T})]$ and $\epsilon$ denotes the distance to the pre-trained backbone. A small $\epsilon$ is essential to avoid catastrophic forgetting in fine-tuning. We find that the proxies in $W^{T}$ are close to $W^{0}$ (i.e., the solution implied by the pre-trained model) as follows. All detailed proof of this work can be found in the appendix.

To reduce the potential bias existing in the pre-trained model, we introduce a reference distribution for the target task. Concretely, a distance constraint is added during the fine-tuning

where $W_{r}$ is a reference distribution of $W$ and $D(\cdot,\cdot)$ is a distance function, in which a squared Euclidean distance can be adopted as $D(W,W_{r})=\|W-W_{r}\|_{F}^{2}$.

By constraining the distance to a reference distribution, the bias in the pre-trained model can be reduced effectively. In addition, the regularization is defined with proxies independent from the backbone, where a small learning rate is applicable for the backbone to avoid catastrophic forgetting.

However, obtaining an appropriate reference without any side information is challenging. Inspired by the recent progress in visual representation learning with natural language supervision [33], we consider leveraging the class name as text information to generate the reference distribution. Compared with images, tokens in text is organized discretely and text encoder can be pre-trained on a sufficiently large corpus with rich context, which is ideal to complement discrimination tasks such as classification.

Let $Z=[\mathbf{z}_{1},\dots,\mathbf{z}_{C}]\in\mathbb{R}^{d_{z}\times C}$ denote the proxies extracted from a text encoder for classes, the problem with text supervision can be written as

which is equivalent to

The Euclidean distance requires that the text feature has the same dimension as the vision feature, i.e., $d_{z}=d$, while these dimensions can vary with different encoders. By applying a projection function $h(\mathbf{w}):\mathbb{R}^{d}\to\mathbb{R}^{d_{z}}$, proxies of classes from different modalities can be manually aligned and the optimization problem becomes

However, due to the inherent differences between modalities, matching representations directly will introduce text-specific noise to visual tasks, which can degenerate the performance on the target visual task. In addition, compared to matching proxies of different modalities, the relationship between classes from the text encoder can be more informative for the target task. Therefore, we consider transferring the pairwise similarity between proxies from the text reference distribution to the vision task.

Concretely, given an anchor class $j$, the distribution over all classes can be computed

where $P_{j}$ and $P^{\prime}_{j}$ denote the distribution defined by proxies from the vision encoder and text encoder, respectively. $\tilde{\mathbf{w}}$ and $\tilde{\mathbf{z}}$ have the unit norm for $\mathbf{w}$ and $\mathbf{z}$, while $\tau$ and $\tau^{\prime}$ are the temperature parameters for vision and text distributions, respectively. With the pairwise relations, we can constrain the KL-divergence between distributions as

To improve the efficiency of fine-tuning, the text encoder is fixed without fine-tuning. Therefore, representations of proxies $Z$ are only extracted once before fine-tuning the vision encoder. By eliminating the constant term from $P_{j}^{\prime}$ in KL-divergence, the problem can be simplified with the cross entropy constraint as

Compared with the problem in Eqn. 4, KL-divergence focuses on the similarity between classes, and thus can reduce the noise from the modality-specific information.

The class-level regularization in Eqn. 6 can be further improved by including the target task examples as a data-dependent regularization. First, if replacing the anchor class $j$ by an anchor example $x_{i}$, the revised instance-level distribution over vision classes becomes

We can show that the class-level distribution is an approximation for the instance-level distribution and optimizing the instance-level distribution can help capture the variance in real data better.

When the intra-class distribution is compact as $\|\mathbf{x}_{i}-\mathbf{w}_{y_{i}}\|_{2}\to 0$, the approximation becomes tight.

Unlike the class-level distribution, the instance-level distribution over text proxies is hard to compute due to the lack of corresponding text features for $x_{i}$. To mimic the data distribution in text space, we propose to optimize the cross entropy loss with the fixed text classifier for representation learning as

where $h^{\prime}(\mathbf{x}):\mathbb{R}^{d}\to\mathbb{R}^{d_{z}}$ is the projection head with the unit normalization to project vision representations to the text space. Compared with Eqn. 4, the text-specific information will be learned for projected examples, which will not introduce noisy patterns to vision proxies.

With the approximated text representations $\mathbf{x}^{\prime}_{i}=h^{\prime}(\mathbf{x}_{i})$, the distribution over fixed text proxies can be computed as

By optimizing the vision encoder and text projection simultaneously, the final objective of our proposed Text Supervised fine-tuning (TeS) can be cast as

where the former two terms fine-tune the network for the target vision task and the latter one is for the projection head to obtain approximated distribution from the text space. During the inference, the projection head will be discarded and only the vision network is adopted for evaluation.

For an extensive comparison, ResNet [20] and ViT [14], two prevalent but diverse visual backbones are included in experiments. For ResNet, ResNet-50 pre-trained by MoCo-v2 [8] and a supervised pre-trained ResNet-18 are adopted, while ViT-B/32 pre-trained by MAE [16] and CLIP [33] are applied for fine-tuning. Each model is fine-tuned with SGD [3] for 100 epochs. The batch size is 256 and the momentum is 0.9. The learning rate is searched in a range of 7 logarithmically-spaced values between $10^{-4}$ and $10^{-1}$ on a validation set. Weight decay is optional and if it is applied, the value is searched with the same setting between $10^{-6}$ and $10^{-3}$. The standard augmentations, i.e., random crop and random horizontal flipping, are applied as in standard fine-tuning pipelines. The parameter $\lambda_{V}$ in our method is fixed as 0.1 and shared by baseline methods with label smoothing, while tuning it may further improve our performance. The temperature $\tau^{\prime}=0.03$ and $\lambda_{T}$ is searched in $[0.1,1.5]$ with the validation set.

To obtain text supervision, two different language models are leveraged as the text encoder. First, we adopt the text encoder from CLIP [33] that is pre-trained with a visual encoder. Besides, we also include BERT [12] to explore the differences between the pure language model and the multi-modal language model. To extract features for each class name, the same prompt in [33], i.e., “a photo of a A” or “a photo of a A, a type of B” for fine-grained data, is applied as input for text encoder. The projection function $h^{\prime}$ consists of a 2-layer MLP suggested by the ablation experiment.

First, we compare different methods by fine-tuning a pre-trained ResNet. The comparison is mainly conducted on ResNet-50 as in Table 1, while the results on ResNet-18 are summarized in Table 2.

From Table 1, we can observe that conventional label smoothing methods cannot improve the performance over the baseline with a distinct margin. It is because that these methods are developed to avoid over-fitting in pre-training, while fine-tuning can mitigate over-fitting by leveraging the pre-trained model. Consequently, the bias in the pre-trained model is more critical for fine-tuning, which cannot be addressed by existing label smoothing techniques. With the instance-level reference distribution from the text encoder, our method TeS can effectively improve the performance over all data sets. By applying the text encoder pre-trained by CLIP, TeS outperforms CE by $1.34\%$ in average with ResNet-50. Since the average performance with ResNet-50 already achieves about $85\%$, the improvement is relatively large. Finally, both text encoders can help fine-tune from ResNet-50 while the one from CLIP is $0.61\%$ better than BERT. It shows that the text encoder pre-trained with vision data is more appropriate for supervising the fine-tuning of vision tasks. Therefore, the text encoder from CLIP will be adopted in the following experiments.

By further examining Table 2, we find that both supervised and unsupervised pre-trained models can benefit from TeS, which demonstrates that the proposed method is applicable for models pre-trained with different pretext tasks.

Then, we compare the proposed method with ViT as the pre-trained model. The provided projection head in CLIP vision encoder is kept as pre-trained parameters for projection. Table 3 and 13 show the results with different pre-trained ViTs.

Evidently, a similar phenomenon as that with ResNet can be observed. First, the proposed TeS can improve the average performance for diverse downstream tasks, while conventional methods are ineffective for fine-tuning. Second, TeS surpasses CE by $1.48\%$ when fine-tuning the ViT pre-trained by MAE, which shows the effectiveness of our method for different architectures. Note that the vision encoder pre-trained by CLIP already incorporates the text side information in contrastive pre-training, which makes the gain from our method less than that by MAE. Nevertheless, TeS can still improve the average performance by $0.78\%$ over CE and the repeated experiments in appendix confirm that TeS outperforms the best baseline significantly.

Third, fine-tuning vision encoders pre-trained by MAE and CLIP shows the similar average performance, while that for individual data sets varies. It demonstrates that different pre-training methods can encode diverse patterns, while the proposed method can obtain consistent improvement over various pre-trained models. Finally, compared with zero-shot transfer denoted as “ZS”, fine-tuning outperforms ZS by a large margin of $24.3\%$ and it suggests that fine-tuning is preferred when training examples are sufficient.

In addition to the above comparison with fine-tuning the whole data set, the proposed method is also evaluated in a challenging scenario, where each data set only contains $10\%$ randomly sampled training examples for few-shot learning. The minimal number of examples will be set to 10 for each class. Pre-trained ResNet-50 is applied as the vision encoder and Table 5 demonstrates the comparison.

With only 10% training examples, it is challenging for fine-tuning to reduce the bias in the pre-trained model when distribution shifts. With text supervision, our method can explicitly constrain the learned distribution close to the target task and achieve $4.36\%$ improvement on average. While our method demonstrates the superior performance on fine-tuning the whole data set, the experiment on few-shot learning implies that it can be more helpful when the number of target training examples is limited.

Finally, we investigate the proposed method for the data set with long-tailed distribution. Following [4], CIFAR-10 and CIFAR-100 with the imbalance ratio of $10$ and $100$ are included in the comparison. The performance of fine-tuning a pre-trained ResNet-50 is shown in Table 6.

With the reference distribution from text, the distribution of proxies from minor classes will not be overwhelmed by that from major classes. Thus, the performance of TeS surpasses the baseline CE with a clear margin. The improvement becomes larger when the imbalance ratio increases to $100$ and it shows that our method is not sensitive for long-tailed data.

To further demonstrate the proposed method for long-tailed data, we evaluate TeS on a challenging data set iNaturalist-2017 [36] that contains 5,089 categories in Table 7. Evidently, our method outperforms cross entropy baseline with a large margin of $6.19\%$ and it confirms the potential of text supervision for handling long-tailed data.

Since the proposed method is not optimized for class imbalance learning, combining it with state-of-the-art methods for this specific task [4] is left as future work.

In this subsection, we demonstrate the effect of different components in TeS by ablation study. Experiments are conducted with ResNet-50 on CIFAR-100.