Test-time Alignment-Enhanced Adapter for Vision-Language Models

Given a test image $x_{t}$, a global visual feature $f_{test}=E_{i}(x_{t})$ is obtained by CLIP’s visual encoder $E_{i}$. Similarly, the corresponding text features $\omega$ can be encoded using CLIP’s text encoder $E_{t}$. The text is composed of hand-crafted templates, one typical form being “a photo of [CLASS]” and a specific category name, such as “dog.” As a result, CLIP’s prediction $y$ for a test image is obtained by the matching score as follows:

$$P_{\mathrm{clip}}(f_{\mathrm{test}})=f_{\text{test}}\mathbf{\omega}^{T}.$$

(1)

To better adapt CLIP to TTA tasks, TDA [7] comprises two dynamic key-value caches, each storing a dynamic queue of few-shot test features as keys and their corresponding pseudo-labels as values. The first one is positive cache, which aims to collect high-quality few-shot pseudo labels $\hat{L_{p}}$ as positive values and the corresponding features $Q_{p}$ as keys. The second one is the negative cache, which aims to gather CLIP-generated image features to $Q_{n}$, and the corresponding negative pseudo labels to $\hat{L_{n}}$. Finally, the prediction of TDA can be formulated by combining the negative cache, the positive cache, and the pre-trained CLIP model as follows:

$$P_{\mathrm{TDA}}(f_{\mathrm{test}})=P_{\mathrm{clip}}(f_{\mathrm{test}})+P_{\mathrm{pos}}(f_{\mathrm{test}})+P_{\mathrm{neg}}(f_{\mathrm{test}}).$$

(2)

Although TDA has achieved great efficiency and effectiveness, solely adjusting classification logits based on the dynamic cache while keeping text features unchanged is not optimal.

This section presents a new approach called Test-time Alignment-Enhanced Adapter (TAEA) to adjust text features. And we employ an enhancement module to further enhance text-image alignment capability under the unsupervised condition of TTA. Fig. 1 shows an overview of our proposed method.

Adapter module. The adapter module introduces two steps. The first step involves retrieving test knowledge from the test image with attention mechanisms. This can obtain image features related to text features. The second step employs a gating network to aggregate hand-crafted text features with the image features obtained in the first step. After completing the above two steps, we can obtain the text features aligned with the image.

Specifically, given the test images, we can obtain the image features $F$ and the hand-crafted text features $\omega$ with the encoder of CLIP. The text features $\omega$ is composed of hand-crafted prompt and $N$-class names. Then, we match the test image features with text features using a single attention block. Thanks to the attention [8] mechanism, we can get better image features related to the text features by treating $\omega$ as the query and employing the test image features $F$ as both key and value. To reduce training overhead and achieve a lightweight adapter, following the previous works [12], we leverage MLP as the attention mechanism, which could be defined as:

$$\hat{\mathbf{F}}=F^{\top}\sigma((\omega W_{W}^{\top})(FW_{F}^{\top})^{\top}/\sqrt{D}),$$

(3)

where $\hat{\mathbf{F}}$ represents the image feature related to text feature. The $W_{F}$ and $W_{W}$ indicate the weights of MLP layers. The $\sigma$ denotes the Softmax function and $D$ is the scaling factor.

After obtaining the $\hat{\mathbf{F}}$, we introduce a learnable gate block $f(\cdot)$ to match $\hat{\mathbf{F}}$ and text features $\omega$, where $f(\cdot)$ only contains a single MLP layer. In this way, we can generate a modulation scalar to filter knowledge and control the ratio between text and image features in the test time [12]. Finally, we could fine-tune the text feature as follows:

$$\hat{\omega}=\omega+f(\omega)\odot\hat{\mathbf{F}},$$

(4)

where $\odot$ denotes Hadamard product. After training the gate block, $f(\cdot)$ can adjust the match ratio according to the $\omega$ and address distribution shift. The adapter could be defined as follows:

$$P_{\mathrm{adapter}}(f_{\mathrm{test}})=\gamma\frac{\hat{\omega}^{\top}f_{test}}{\|\hat{\omega}\|\|f_{test}\|},$$

(5)

where $f_{test}$ is the test image feature obtained by encoder and $\gamma$ is a hyper-parameter.

To train our adapter module, we select samples with lower entropy to generate a pseudo label $L_{a}$, a one-hot encoded vector of a categorical distribution, and store them during the testing period of the first $\lambda*N$ samples. The $N$ represents the total number of images in the test set, and $\lambda$ is a hyperparameter to determine the timing of training the adapter. After the model has tested $\lambda*N$ samples, we train the adapter module to update text features with these test samples. We use cross-entropy as the loss function.

The effectiveness of adjusting text features is theoretically guaranteed by non-local filters [16, 17, 18]. Intuitively, similar to the non-local filters, [12] states that an adapter based on gated attention could disregard some outlier samples while paying more attention to the samples related to the category description, resulting in robust feature representations.

Enhancement module. To mitigate the risk of prediction errors due to high entropy or being biased to certain predictions, we propose to adopt the negative cache from TDA [7] as the plug-and-play module for further enhancing the performance of our adapter. As stated by  [7], the negative cache is designed for negative learning. It aims to mitigate the negative impact of noisy pseudo-labels by introducing negative pseudo-labeling to identify class absence rather than presence.

In summary, with the help of the adapter and enhancement modules, our method can adjust hand-crafted text features to enhance the capability of text-image alignment. The overall formulation for computing classification results in our method is as follows:

$$P_{\mathrm{TAEA}}(f_{\mathrm{test}})=P_{\mathrm{clip}}(f_{\mathrm{test}})+P_{\mathrm{adapter}}(f_{\mathrm{test}})+P_{\mathrm{neg}}(f_{\mathrm{test}}).$$

(6)

Datasets. Consistent with prior works [6, 19], we conduct main experiments on two benchmarks: out-of-distribution (OOD) benchmark and cross-domain benchmark. To assess the robustness and generalization, OOD benchmark consists of one ID dataset ImageNet [20] and 4 out-of-distribution datasets derived from ImageNet: ImageNet-A [21], ImageNet-V2 [22], ImageNet-R [23], and ImageNet-S [24]. On the other hand, to evaluate the adaptation ability during testing on datasets with different domain distributions, the cross-domain benchmark consists of 10 diverse image classification datasets, each from a distinct domain with different classes: Pets [25], EuroSAT [26], Aircraft [27], Caltech101 [28], UCF101 [29], Cars [30], DTD [31], Flower102 [32], Food101 [33], and SUN397 [34].

Baselines. To evaluate the effectiveness of our method, we compare it with zero-shot methods, train-time adaptation methods, and test-time adaptation methods, including the recent state-of-the-art method. For the zero-shot method, We compared with public CLIP [2] results under ResNet-50 [35] and ViT-B/16 [36]. For the train-time adaptation methods, we compared with CoOp [37], CoCoOp [38], and Tip-Adapter [11]. For the test-time adaptation methods, we compared with TPT [5], and its improved version, DiffTPT [6]. We also compared against the challenging state-of-the-art method in this field: TDA [7]. All results of the methods compared in the table are obtained from the [7] paper.

Implementation Details. Same as prior works [5, 6, 7], we conducted experiments on out-of-distribution benchmark and cross-domain benchmark separately using ResNet-50 and ViT-B/16 backbones. Specifically, we use a batch size of 1, which means the test time adaptation is set for single-image scenarios. We set all database-related hyperparameters to be consistent with TDA. And we set $\lambda=0.25$, $\gamma=0.6$. For the training of the adapter module, We optimize the gated attention block with a batch size of 3 and use AdamW [19] optimizer with a learning rate of 0.001 and a cosine scheduler for 3 epochs. In all experiments, we evaluate a single NVIDIA Quadro RTX 6000 GPU.

Results on the OOD benchmark. The performances on the OOD benchmark are summarized in Table  2. Our method has surpassed the method that adjusts text features similarly and the current state-of-the-art method on average. Specifically, compared to the TPT method, which adjusts text features similarly, our method outperforms TPT on both ResNet-50 and ViT-B/16 architectures, improving all accuracy by 2.87% and 3.32% on average, respectively. Furthermore, compared to the current state-of-the-art method TDA, our method outperforms TDA on both ResNet-50 and ViT-B/16 architectures, improving all accuracy by 0.55% and 0.75% on average, respectively.

As shown in Table 3, to further evaluate the efficiency and effectiveness of our method, we conducted experiments on ImageNet validation dataset to test both accuracy and testing time. When compared to the current state-of-the-art method (TDA), although our approach requires additional testing time (requiring an additional 2min) due to minimal training requirements, we have achieved substantial improvements in accuracy (+3.82%). Compared to methods requiring similar training for adjusting text features, our approach dramatically reduces the testing time from 12h 50min by TPT and even more from 34h 45min by DiffTPT, down to just 18 minutes.

Results on the cross-domain benchmark. The performances on the cross-domain benchmark are summarized in Table 1. Our method not only surpasses the methods (i.e., TPT, DiffTPT) that adjust text features similarly but also exceeds the current state-of-the-art method TDA on average, reaching a new state-of-the-art. Specifically, compared to the TPT method, which adjusts text features similarly, our method outperforms TPT on both ResNet-50 and ViT-B/16 architectures, improving all accuracy by 6.03% and 4.93% on average, respectively. Furthermore, compared to the current state-of-the-art method TDA, our method outperforms TDA on both ResNet-50 and ViT-B/16 architectures, improving all accuracy by 2.66% and 2.5% on average, respectively.

To validate the effectiveness of our proposed method, we conducted ablation experiments on the ImageNet [20]. TAEA consists of the adapter module and the enhancement module from negative cache [7]. We first evaluated the effectiveness of using a single enhancement module and a single adapter module. As shown in Fig. 2(a), each module can outperform the original CLIP-ResNet-50. Our adapter module exceeds CLIP and surpasses enhancement module, demonstrating its ability to effectively adjust text features and enhance the alignment capability between text and image. Furthermore, combining the enhancement module with the adapter can further improve performance on ImageNet, which is our TAEA method.

Furthermore, we also conduct ablation studies on $\gamma$ in (5). $\gamma$ is a parameter used to control the weighting of the adjusted text features for classification. Specifically, We experimented with a challenging OOD benchmark using $\gamma=0$, 0.2, 0.4, 0.6, 0.8, 1.0. It’s important to note that $\gamma=0$ indicates that the model did not use the adapter to adjust the text, which means that the text features remain unchanged during the testing phase. In Fig. 2(b), we summarize the average results. The result of baseline is from [7]