Robust Noisy Correspondence Learning via Self-Drop and Dual-Weight

The purpose of confidence weight is to minimize the impact of a small number of noisy samples in $D_{p}$ while appropriately leveraging the vague samples within $D_{p}$. Following DSDMR [21], we fit the similarity of all pairs by a two-component Gaussian Mixture Model (GMM) [63, 64]:

where $\alpha_{k}$ denotes the mixture coefficient, and $\phi(x|\theta_{k})$ denotes the probability density of the $k^{th}$ component. The confidence probability $w_{con}^{i}$ of pair $i$ is calculated by:

where $\theta_{k}$ is the Gaussian component with the higher mean.

We utilize $w_{con}$ generated by the GMM to re-weight the data in $D_{p}$. After that, vague samples can contribute to model training with proper weight, while the negative impact of a small portion of noisy samples in $D_{p}$ is diminished.

NPC’s [22] performance on noise-free remarkably outperforms CLIP, demonstrating that not all clean samples are equally significant for VLP. Inspired by this observation, we introduce a significant weight into our approach.

It is a rational assumption that the model’s performance should be improved after training on clean samples compared to before the training. In other words, if the model’s performance is worse after training than before, we can regard training samples that are insignificant or even noisy. For convenience, we utilize the loss value of the model as a substitute for its performance. We evaluate the change in loss values before and after training with a memory bank $(MB)$ to calculate the significance of each sample. The memory bank is a reliable set of clean samples that correspond to each sample in the noisy dataset $D$.

To obtain $MB$, a clean dataset $D_{c}$ is sampled with a relatively high threshold $\beta$ (30):

Then, for each image-text pair $(I_{i},T_{i})$ in the dataset $D$, we assign two evaluation entries $(I_{i}^{I},T_{i}^{I})$ and $(I_{i}^{T},T_{i}^{T})$ to it, where $I_{i}^{I}$ shows the highest similarity with $I_{i}$ in $D_{c}$ and $T_{i}^{T}$ shows the highest similarity with $T_{i}$ in $D_{c}$. Specifically, the memory bank can be denoted as:

Samples in the partial dataset $D_{p}$ are evaluated to obtain significance weight by clean samples from the memory bank. As illustrated in Fig. 2, we initially create a siamese model by copying the parameters $\Theta_{b}$ of the base model, with the parameters of the siamese model represented as $\Theta_{s}$. We use the model’s loss $l^{i}$ on clean $D_{MB}$ to represent performance, where a lower loss indicates a higher significance of the sample. Formally, for the $i$-th data pair $(I_{i},T_{i})$ in $D_{p}$, we can compute its i2t loss and t2i loss of its evaluation entry in the current epoch $e$ as follows:

Then, we train the siamese model using data from $D_{p}$ weighted by $w_{con}$ and update the siamese model parameters $\Theta_{s}^{e}$ to $\Theta_{s}^{e+1}$. The change in performance after updating the siamese model parameters can be calculated by:

Finally, the significance of clean samples can be defined by:

The values of performance change $c_{i}$ are densely distributed around 1. Therefore, we aim to discretize the weights of significant and insignificant samples through a mapping function, such as the $tanh$ or the $sigmoid$ function. Samples that decrease performance after training ($c_{i}<1$) will be assigned with smaller weights, while those that improve performance ($c_{i}\geq 1$) are set the weights to 1.

NPC aims to estimate the negative impact of each sample and re-weights all samples before model training without sample selection. Conversely, our proposed SDD is established from the observation that a small number of clean samples is more valuable than a majority of noisy ones for fine-tuning VLP. Therefore, we perform sample selection to explicitly discard a large proportion of noisy data to enhance the data reliability.

Our significance weight is indeed motivated by NPC, indicating that not each sample contributes equally to model training. However, in our research, we discover that relying solely on a single weight (NPC’s method) is insufficient to eliminate the negative impact of noise (we will demonstrate it in experiment Fig. 4). Therefore, we further explore the weighting strategy and propose a dual-weight method to re-weight samples from both confidence and significance perspectives.

Following previous works, the proposed SDD was evaluated on three benchmark datasets, Flickr30K [5], MSCOCO [6], and CC120K [22]:

• •

  Flickr30K comprises 31,783 images with 5 annotated texts per image. Following previous works [8], 29,783, 1,000, and 1,000 images were used for training, validation, and testing, respectively.

• •

  MS-COCO encompasses 123,287 images with 5 annotated captions per image. Following previous works [8], we employed 113,287 images for training, with 5,000 images allocated for validation and 5,000 images reserved for testing.

• •

  CC120K is a subset of the web-crawled dataset Conceptual Captions [7], with about 3%-20% mismatched image-text pairs. CC120K consists of 120,851 images with a single caption for each. Following NPC [22], we utilized 118,851 images for training, 1,000 images for validation, and 1,000 images reserved for testing.

We evaluated SDD with the recall rate at K (R@K) which measures the proportion of relevant items found within the top K results of a ranked list. We took the image and text as queries, respectively, and calculated the corresponding results of R@1, R@5, and R@10, which were further summed to evaluate the overall performance, i.e., rSum. Additionally, we used the variance $(var)$ of rSum under different noise ratios to evaluate the approaches’ performance stability, with lower $var$ indicating higher stability.

We adopted the pre-trained CLIP [20] with ViT-B/32 as our backbone and trained SDD on a single RTX 3090 GPU. We employed a batch size of 128 and an AdamW [66] optimizer with a weight decay of 0.2. In all experiments, we trained the model for 5 epochs with a learning rate of $2e-7$. The hyperparameters $\alpha$ and $\beta$ were set to 20 and 30, respectively.

To verify the effectiveness of SDD, we conducted comparison experiments with 8 state-of-the-art methods, including NCR [8], BiCro [55], MSCN [56], CRCL [65], GSC [57], ESC [58], DSDMR [21], and NPC [22]. Given that the data in Flickr30K and MS-COCO are correctly paired, we introduced noisy correspondences by randomly shuffling the captions of 20%, 40%, and 60% of training images. Table I shows results on Flickr30K and MS-COCO under different noise ratios. For Flickr30K and MS-COCO 1K, we obtained results from ESC [58] and the publications introducing the respective models. For MS-COCO 5K, we used the data reported in the ESC as the results.

From the experimental results, it can be seen that the proposed SDD demonstrates significant improvements over all state-of-the-art methods. In the case of Flick30K, SDD shows advantages over the second-best method in the rSum score of recalls by 4.7%, 8.1%, and 7.9% under different noise ratios, respectively. For MS-COCO 1K, the rSum scores of SDD are 3.0%, 6.0%, and 7.1% higher than the second-best method. In Table II, we also demonstrate the superior performance of SDD on the MS-COCO full 5K dataset. SDD surpasses the second-best method in the rSum score of recalls by 9.8% and 12.8%.

Table III shows the results on CC120K under a real-world noisy correspondence scenario. The baseline results are derived from NPC [22] for convenience. It can be observed from Table III that SDD achieves the best performance with an overall score of 519.3%, surpassing the second-best method NPC by 1.7%. The results affirm SDD’s capability to manage not only simply simulated but also complex, real-world noisy correspondences.

To further investigate the performance stability of SDD, we present in Fig. 3 the rSum change curves of different methods on Flickr30K and MS-COCO 1K under varying noise ratios. The performance of each method is obtained from its original paper respectively. From Fig. 3, it can be observed that the proposed SDD considerably outperforms all methods across all noise ratios. SDD demonstrates relatively stable performance under different noise ratios, whereas the performance of other methods significantly decreases as the noise ratio increases. Furthermore, we calculate the variance of each method under different noise ratios to quantify the stability of the methods. Our model shows a significant stability gap compared with most other methods, with variances of only 4.54% and 4.25% on Flickr30K and MS-COCO 1K, respectively. Although DSDMR has a lower variance than SDD on Flickr30K, its performance is evidently worse than ours. These results validate the effectiveness of mitigating the negative effects of noise through self-drop and confidence weight $w_{con}$.

In order to demonstrate the noise-robustness of our approach, we further investigate the training process by conducting a progressive comparison. We selected NPC as a compared baseline method because it generally demonstrated the second-best performance on three datasets. The results of the NPC were reproduced by us with the same experimental setting as ours. In Fig. 4, we present the results of the R@1 average values for NPC and SDD on the Flickr30K and MS-COCO validation sets. The results indicate that our SDD demonstrates promising performance across all noise ratios and maintains relative stability throughout the training process. Conversely, although NPC initially exhibits notable performance, its effectiveness gradually declines due to the accumulation of errors. This phenomenon indicates that using a single weight for re-weighting (NPC’s strategy) is insufficient to address the challenges posed by noisy correspondence. Therefore, we opt to discard noisy samples using self-drop and further mitigate the impact of noise by dual-weight.

Since the CLIP ViT-B/32-based backbone inherently has advantages over other backbones, we also provide results of other methods adopting the ViT-B/32 backbone in Table IV, including VSE$\infty$ [67], PCME [68], PCME++ [69], PAU [70], NPC [22], for a fair comparison. The results of the models compared with SDD in this experiment are all from the paper of NPC [22]. Specifically, we conducted comparative experiments under 20% and 50% noise correspondence scenarios, and report the mean R@1 MS-COCO 1K and 5K, as well as rSum on 1K.

As demonstrated in Table IV, under 20% and 50% noise ratios, the rSum of SDD outperforms the previous best-performing method NPC by 2.9%, and 6.7%, respectively. Moreover, under 20% noise ratio, SDD surpasses NPC by 1.2% and 1.7% on MSCOCO 1K and 5K, respectively, and under 50% noise ratio, SDD surpasses NPC by 1.7% and 2.6% on MSCOCO 1K and 5K, respectively. It is worth noting that, as a representative of VLP, CLIP exhibits strong zero-shot capabilities but its performance significantly deteriorates in noisy scenarios. Specifically, when the noise ratio increases from 20% to 50%, CLIP’s rSum on MS-COCO 1K decreases by 21.2%. Compared to other CLIP-based methods, our work significantly reduces the risk of learning noise data with CLIP ViT-B/32-based backbone as much as possible, which demonstrates the value of our work.

We show the effect of each component in Table V. Specifically, we ablated the contributions of three key components of SDD, i.e., self-drop, confidence weight $w_{con}$, and significance weight $w_{sig}$. From Table V, we observe the following conclusions: 1) The full SDD could achieve the best overall performance, showing that all three components are important to improve the robustness against noisy correspondence. 2) The model’s performance significantly declines without confidence weight $w_{con}$ demonstrating that a small number of noisy samples can have a severe impact on the model.

The memory bank is used to assess the significance of samples, thus a reliable memory bank is a prerequisite for the superiority of our method. To verify the reliability of the memory bank, we visualize the data distribution of the clean dataset $D_{c}$ in Fig. 6 (a). We compare different choices of threshold $\beta$ on the Flickr30K dataset with low (20%) to high (60%) noise ratios, respectively. From Fig. 6 (a), it can be observed that the proportion of noisy samples in $D_{c}$ is larger when $\beta$ is small, but it can be negligible under a high threshold setting. Therefore, the samples in our memory bank are sufficiently reliable.

In addition, we also visualize the distribution of samples discarded by SDD in Fig. 6 (b) to verify the reliability of self-drop. Obviously, with the selection of an appropriate threshold, the self-drop strategy merely discards a small number of clean samples. At low noise ratios (20%), the self-drop strategy discards only about 1% of the clean samples, and at high noise ratios (60%), the number of clean samples discarded by the self-drop strategy is even lower than 0.2%. As we previously observed in Fig. 1 (b), discarding a small portion of clean samples does not significantly impact the model, whereas a few noisy samples can have a substantial negative effect. Therefore, it is sufficiently reasonable that SDD filters noisy correspondence samples by self-drop.

To further explore the influence of our method, we visualized the similarity distribution of clean and noisy pairs at different training stages of our SDD in Fig. 5 with the second-best baseline NPC for comparison. As training progresses, the similarity of clean samples of SDD rises higher while that of noisy data decreases, exhibiting a clear trend of separation. This phenomenon indicates that our approach less fits noisy samples and remarkably suppresses the negative effect caused by them. Meanwhile, one could observe that there is a clear distinction between noisy and clean samples in NPC’s distribution in the initial stage (Fig. 5 (a)). However, the overlap between noisy and clean samples increases in the following epochs (Fig. 5 (b)-(d)), indicating that NPC inevitably fits noisy data. Conversely, the noise samples and clean samples in SDD exhibit a more evident separation trend. This phenomenon accounts for the reason why SDD outperforms NPC across multiple datasets and demonstrates greater stability.

The selection of hyperparameters $\alpha$ and $\beta$ is crucial for SDD. The former determines the threshold for discarding noisy samples, while the latter sets the threshold for constructing the memory bank. As illustrated in Fig. 5 (a), the similarity of noisy samples is typically below 20, whereas that of clean samples is generally above 30. This phenomenon accounts for the hyperparameter setting $\alpha=20$ and $\beta=30$. In other words, $\alpha$ and $\beta$ are easy to adjust according to similarity distribution and will not complicate our approach.

We conducted the case study and show some examples intuitively. Fig. 7 (a)-(d) respectively illustrate the cases of our qua-partition: noisy, vague, clean yet insignificant, and clean and significant. We report similarity score $sim$ for the noisy sample and the confidence weight $w_{con}$ and significance weight $w_{sig}$ for others. In particular, noisy image-text pairs with low similarity are directly discarded, and thus their weights are not displayed. From Fig. 7 (a), we can see that the content of images has no relationship with their corresponding captions. Therefore, it is a cost-effective choice to discard such samples directly. Vague samples are tough to distinguish from noisy data according to image-text similarity and they are assigned proper confidence weights (Fig. 7 (b)). Clean samples share high $w_{con}$ while $w_{sig}$ balances their contribution to model training (Fig. 7 (c)-(d)).

Due to the complexity of vague samples, we present a case study about more vague samples in Fig. 8 for a comprehensive understanding of them. For the vague samples in Fig. 8, we leverage different colors to distinguish between correct (green) and incorrect (orange) correspondences. Specifically, vague mismatched pairs may exhibit subtle semantic misalignment, e.g., action and the scene in the second and third images. Moreover, complete misalignment and proper alignment are also very common in vague samples. Consequently, vague samples are located near the decision boundary [23], which presents a challenging issue to resolve in noisy correspondence. Therefore, we assign a smaller confidence weight to cautiously utilize these samples.