MAFA: Managing False Negatives for Vision-Language Pre-training

GRIT-VLP uses ITC, ITM with in-batch hard negative sampling, and MLM as the objectives proposed in ALBEF, except the utilization of the momentum encoder in the pre-training. However, it significantly extends ALBEF by implementing two key components as follows:
(a) GRouped mIni-baTch (GRIT) sampling aims to construct mini-batches containing highly similar example groups. This facilitates the selection of informative hard negative samples during in-batch hard negative sampling. To avoid excessive memory or computational overhead, the procedure of constructing grouped mini-batches for the next epoch is performed concurrently with the loss calculation at each epoch. For this, additional queues are used to collect and search for the most similar examples, one by one, in which the similarity is measured by the ITC scores. These queues serve as the search space and are significantly larger than the mini-batch size $B$. Thus, the size of the queue, denoted as search space $M$, controls the level of hardness in selecting hard negative samples.
(b) ITC with consistency loss attempts to address the issue of over-penalization in ITC that arises when GRIT is combined. Contrary to ALBEF, similar examples are gathered in the GRIT-sampled mini-batch. Thus, when one-hot labels are used for ITC, they result in equal penalization of all negatives, and it has been observed that the representations of similar samples may unintentionally drift apart. To mitigate this, GRIT-VLP incorporates soft pseudo-targets generated from the same pre-trained model as a mean of regularization.

As outlined in the Introduction, the issue of false negatives originates from missing positive connections within a paired dataset, a challenge that is computationally infeasible to address naively. To tackle this, ECM process is strategically designed to exclusively examine hard negatives with significantly higher likelihoods of being false negatives. Namely, as described in Figure 4 and Algorithm 1 in S.M, the training model selects the hardest negatives for each anchor based on ITC similarity in the GRIT-sampled mini-batches. Once hard negatives are selected, a separate pre-trained ITM model, Connection Discriminator (Con-D), is employed to determine whether these hard negatives are true (hard) negatives or false negatives. If Con-D assigns a probability to a candidate image-text pair of being positive higher than a threshold $\tau$ (which was set to $0.8$), that candidate is adopted as a new positive pair in the pre-training losses.

We note that ECM process can be seamlessly integrated with the training process of BLIP-family models (ALBEF [35], BLIP [36], BLIP-2 [37]) as well by adopting GRIT sampling and Con-D, given that ITC and ITM are used as their training objectives. Moreover, due to the inherent randomness of mini-batches, Con-D encounters a variety of hard negatives in each batch and epoch, which enables ECM to create diverse positive connections during training.

Now, we will elaborate on the details of the three pre-training losses (ITC, ITM, and MLM) utilized in our model, and then explain how false negatives identified by ECM are integrated into these losses. Briefly, for ITC and ITM, the labels for identified false negatives are revised from negatives to positives. Moreover, these new positives are additionally used as inputs for MLM.

To incorporate missing positive connections identified by Con-D, the one-hot label $\bm{y}$ is adjusted to $\tilde{\bm{y}}^{\text{ITC}}$. For example, for an anchor image $i$, a single text $t_{k}$ is picked by in-batch hard negative sampling. Then, if $t_{k}$ is recognized as a new positive by Con-D, the $k$-th element of the one-hot vector $\bm{y}^{\text{I2T}}(i)$ changes from $0$ to $0.5$. Simultaneously, the original label value of $1$ in $\bm{y}^{\text{I2T}}(i)$ becomes $0.5$, ensuring that the sum remains $1$. If a positive connection is not newly established, the label remains unchanged. The above process is applied identically for text $t$ and image $i_{k}$. Now, we denote the new ITC loss equipped with $\tilde{\bm{y}}^{\text{ITC}}$ as $\mathcal{L}_{\text{ITC}}^{\textit{ECM}}$.

To overcome the challenge of false negatives in ITC under GRIT sampling, we additionally introduce a computation-free approach named S-ITC, which employs label smoothing to contrastive loss, which has not been extensively explored in VLP. Specifically, we take the following loss form:

in which $\alpha$ represents a mixing parameter, and $\bm{1}$ denotes all-one vector.

We emphasize that label smoothing has not been widely adopted in typical VLP settings since it has not been very effective. As we show in Table 6 (Section 5), performance is significantly degraded when S-ITC is applied under the random sampling scenario. This decline is largely due to the detrimental effect of providing soft labels for the examples in the randomly sampled batch where true negatives are prevalent. In contrast, under GRIT sampling where each mini-batch is predominantly composed of samples that are likely to be false negatives, we observe that S-ITC, which ensures relatively high soft labels for all negatives, becomes highly effective.

There also have been other attempts to address the issue of false negatives in ITC, such as momentum distillation [35] and consistency loss [3]. Here, we explain only I2T-related terms for simplicity; T2I-related terms are similarly computed. Momentum distillation replaces $\bm{y}_{t}^{\text{I2T}}(i)$ by $\bm{y}_{t}^{\text{MD}}(i)=(1-\alpha)\bm{y}_{t}^{\text{I2T}}(i)+\alpha sg[\tilde{\bm{p}}_{t}^{\text{I2T}}(i)]$, where $sg[\cdot]$ is the stop gradient operator, and $\tilde{\bm{p}}$ denotes the probability obtained from the momentum encoder. Here, $N$ is equal to batch-size $B+Q$ since the model is accompanied by a queue of size $Q$ that stores embeddings to provide additional negatives. However, this approach suffers from inefficiency due to the additional forwarding of the momentum model, and it results in increasing the model size. In consistency loss, $\bm{y}_{t}^{\text{I2T}}(i)$ is substituted with $\bm{y}_{t}^{\text{CS}}(i)=(1-\alpha)\bm{y}_{t}^{\text{I2T}}(i)+\alpha sg[\bm{p}_{i}^{\text{T2I}}(t)]$, where $\bm{p}_{i}^{\text{T2I}}(t)$ is computed by the model itself. Here, $N$ is the same as $B$ since it does not involve a queue.

However, as shown in Table 6 (in Section 5), the effectiveness of momentum distillation and consistency loss is limited. To explore the reason behind this, we examine the soft labels from the above methods in the GRIT sampling scenario as reported in Table 2, aiming to uncover the distribution shapes of the soft labels. The values in the table are obtained through the following process: the soft labels are sorted in descending order, and then averaged across all samples. Further details on the computation process are described in S.M. We observe that momentum distillation continues to assign almost zero labels to negative samples, which are likely to be false negatives under GRIT sampling. This result may stem from the large number of negatives in the queue, which prevents each negative sample from receiving non-negligible labels. On the other hand, consistency loss assigns comparatively higher soft labels (0.0178) than momentum distillation (0.0009) but overly concentrates on a few pairs, resulting in negligible labels for most negatives. In S.M, we provide an analysis that this phenomenon cannot be resolved by merely tuning $\alpha$.

Given that both consistency loss and the momentum distillation fail to achieve the intended objective of assigning non-negligible soft labels to the majority of negatives, we argue that S-ITC, which explicitly assigns higher soft labels to all negatives, can be a simple but effective solution. In S.M, we include an analysis of its robustness to $\alpha$.

Consequently, as illustrated in Figure 3 and Algorithm 1 in S.M, we adopt the following pre-training objective:

where $\mathcal{L}_{\text{S-ITC}}^{\textit{ECM}}$ represents the integrated ITC loss of $\mathcal{L}_{\text{ITC}}^{\textit{ECM}}$ and $\mathcal{L}_{\text{S-ITC}}$, which adopts the target labels as $(1-\alpha)\tilde{\bm{y}}^{\text{ITC}}+\frac{\alpha}{N}\bm{1}$.

During our training process, we utilize four datasets (MS-COCO [40], Visual Genome [32], Conceptual Captions [54], and SBU Captions [47]) with a total of 4M unique images (5M image-text pairs), as proposed by ALBEF [35] and UNITER [6]. We refer to this collective dataset as the “4M-Noisy” dataset due to a significant number of captions that offer either incomplete or incorrect descriptions, which can be seen as false positives. To analyze the impact of our approach in handling false negatives relative to the effect of removing false positives, we construct an additional same-sized training set named “4M-Clean” which is composed of clean image-text pairs, refined by the BLIP captioner [36]. Note that all the models are pre-trained with the “4M-Noisy” unless specifically stated as “4M-Clean” in our results table below. Further details on constructing the 4M-Clean dataset are in S.M.

Following ALBEF, we adopt our image encoder as a 12-layer Vision Transformer [13] with 86 million parameters, pre-trained on ImageNet-1k [59]. Both the text and multi-modal encoders utilize a 6-layer Transformer [60], initializing the former with the first 6 layers and the latter with the last 6 layers of BERT-base model (123.7M parameters) [11]. We use the same data augmentation method used in ALBEF and train our model for 20 epochs using 4 NVIDIA A100 GPUs, but excluding the momentum encoder in ALBEF. For Con-D, we use the exact same model architecture as the training model. Unless otherwise noted, we set $B=96$ and $M=4800$ for GRIT sampling, and for all other hyper-parameter settings, we follow GRIT-VLP [3]. More details on the dataset, training, and hyperparameters are in S.M.

After completing the pre-training phase, we proceed to fine-tune our model on three downstream vision and language tasks: image-text retrieval (IRTR) [40], visual question answering (VQA) [19], and natural language for visual reasoning (NLVR2) [57]. For IRTR, we utilize the MS-COCO [40] and Flickr30K (F30K) [48] datasets, with F30K being re-splitted according to [30]. Following BLIP [36], we exclude the SNLI-VE dataset [64] due to reported noise in the data. Our fine-tuning and evaluation process mostly follows that of GRIT-VLP. More details of downstream tasks are in S.M.

In Table 3, we observe that our approach consistently outperforms other baselines in multiple downstream tasks (IRTR, VQA, NLVR2). Notably, MAFA even surpasses ALBEF (14M) and competes with BLIP (14M) on certain metrics, despite being trained on a significantly smaller dataset. Specifically, MAFA achieves significant improvements over GRIT-VLP, with a substantial margin of $+1.4\%$ IR/R@1, $+1.6\%$ TR/R@1 on MS-COCO and $+1.1\%$ on NLVR2 dev. These results clearly show the significance of addressing false negatives when leveraging hard negative mining. Additionally, we believe that the comparison between BLIP (4M-Clean) and our MAFA shows that the effectiveness of managing false negatives may surpass the impact of mitigating false positives. Furthermore, the enhanced performance of MAFA (4M-Clean) over MAFA shows the synergistic effect of addressing both false positives and negatives.

Table 4 presents the effectiveness of two proposed components: efficient connection mining (ECM) and smoothed ITC (S-ITC). Here, all model variants adopt GRIT sampling, with row 1 representing the original GRIT-VLP model. The results clearly demonstrate that applying either the S-ITC (row 3) or the ECM (row 2) individually leads to performance improvements compared to a model that does not consider false negatives (row 1). By combining both S-ITC and ECM in our final model (row 4), we observe significant performance enhancements on the 4M-Noisy dataset. This tendency is validated again in the 4M-Clean dataset, confirming the consistent effectiveness of the proposed components (row 6). Moreover, by comparing the performance gap between MAFA trained on the noisy dataset (row 4) and GRIT-VLP trained on the clean dataset (row 5), we reaffirm that addressing false negatives outweighs the impact of handling false positives. Beyond the 4M dataset, we present additional results across a broader range of data scales (1M, 2M, and 14M) in S.M, demonstrating the robustness of MAFA with respect to data scale variations.

Table 5 provides an additional comparative analysis on the effectiveness of MAFA, based on whether GRIT sampling and ECM are either applied or not. Here, row 1 denotes the ALBEF model without momentum distillation. Since S-ITC is ineffective under random sampling (as we show in Table 6 below), S-ITC is excluded when GRIT sampling is not utilized (row2). We observe that MAFA enhances the performance for both random and GRIT sampling. However, the effect of MAFA is much more vivid for GRIT sampling (row 4, 5), underscoring the critical role of managing false negatives in hard negative sampling. Moreover, our experiments reveal that converting false negatives into additional positives (row 5) is considerably more beneficial than merely removing them (row 4), which highlights the effect of leveraging new positive connections constructed by the model within the dataset itself.

Furthermore, in Table 6, we provide a comparative analysis on S-ITC, which supports our discussion in Section 4.2; S-ITC brings out a unique synergy only when combined with GRIT-sampling. Namely, in random sampling, we observe that S-ITC rather detrimentally affects performance (row 2). Conversely, under GRIT sampling, we verify that assigning relatively high nonzero labels to most negatives enhances performance. Namely, consistency loss (row 4), which assigns relatively higher soft labels to samples in a batch, outperforms momentum distillation (row 5). S-ITC significantly outperforms the other two variants, which highlights the importance of assigning substantial labels to the majority of negatives, rather than just a few.

In Tables 7 and 8, we demonstrate that our MAFA can be successfully integrated with the recent BLIP-2 [37], which is quite a successful vision-language pre-training framework. As described in S.M., the stage-1 of BLIP-2, which adopts ITC, ITM, and (auto-regressive) LM losses as objectives, closely resembles the pre-training procedures of both BLIP and ALBEF. Thus, MAFA can be effortlessly incorporated into stage-1 of BLIP-2, following the identical way described in Section 4.

In Table 7, we observe the performance of BLIP-2+GRIT is significantly degraded (row 2), which indicates that solely applying GRIT sampling leads to a failure of learning. We believe this performance degradation primarily stems from more frequent occurrences of false negatives in BLIP-2. In BLIP-2, due to the significantly enhanced capacity of the model, GRIT sampling, which mines hard negatives based on contrastive similarities calculated from the training model, includes a substantially higher number of false negatives in each batch. The integration of MAFA with BLIP-2 leads to enhanced performance, highlighting the importance of managing false negatives to increase the stability of the training process.

We further explore whether the integration of MAFA in stage-1 leads to improved generative learning capabilities after additional stage-2 training where the model is connected to a frozen LLM and pre-trained only with LM loss. We evaluate the performance of models in various zero-shot visual question answering benchmark datasets including GQA [25], OKVQA [44], and VQA [19]. Moreover, we assess the zero-shot image captioning ability on the Karpathy test split of MS-COCO [40]. Table 8 shows that MAFA significantly improves zero-shot performance across various VQA and image captioning tasks. This result not only underscores the compatibility of MAFA with BLIP-2 but also emphasizes that the exclusive integration of MAFA in stage-1 is also beneficial in generative learning capability (stage-2) as well. More detailed results, including those from fine-tuned image captioning and an analysis on how extra positive examples from ECM influence the BLIP-2 stage-2 performance, are provided in S.M.