SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels

The architecture overview of SemVLP is shown in Figure 2. Inspired by the idea of sharing the encoder and decoder in Transformers for neural machine translation (Xia et al., 2019), we base the architecture of SemVLP on a shared bidirectional Transformer encoder, where a pluggable cross-modal attention module is further used to enhance high-level semantic alignment. By sharing the model parameters and adjusting the input format, SemVLP can be flexible to switch between single-stream and two-stream pre-training architectures, with the input text and image encoded in different semantic levels. In this way, we cast both the mainstream pre-training architectures into a more compact one in that there is only one copy of parameter set, which is applicable to both the low-level and high-level semantic alignment with much less parameter cost. We iteratively pre-train on the two settings towards better understanding of the real-world image-text pairs.

Given the embeddings of the words for the sentence $\{e_{i}\}_{i=1}^{m}$ and the image regions $\{o^{\prime}_{j}\}_{j=1}^{n}$, the full encoder is a stacked model with $L$ blocks, where the $l^{\prime}$th block consists of a self-attention module $\varphi_{S}^{l}$, a nonlinear feed forward network $\varphi_{F}^{l}$ and a pluggable cross-modal attention module $\varphi_{C}^{l}$, where superscript $l$ represents the layer index. Both the self-attention and cross-modal attention modules are based on the multi-head attention (Vaswani et al., 2017), where the feed forward network (FFN) consists of an intermediate layer and an output layer as in BERT (Devlin et al., 2018). In both the single-stream and two-stream modes, the self-attention module $\varphi_{S}^{l}$ and feed forward network $\varphi_{F}^{l}$ are shared and tied to a single one copy of parameter set. The cross-modal attention module $\varphi_{C}^{l}$ is additionally used in two-stream mode to enhance the high-level semantic alignment.

Pre-training Data  We use the same in-domain data as in LXMERT (Tan and Bansal, 2019) for pre-training. It consists of the image caption data from MS COCO (Lin et al., 2014), Visual Genome (Krishna et al., 2017), and image question answering data from VQA v2.0 (Antol et al., 2015), GQA balanced version (Hudson and Manning, 2019b) and VG-QA (Zhu et al., 2016). The total amount of the dataset is 9.18M image-and-sentence pairs on 180K distinct images. Besides, we also use additional out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model pre-training, which consists of about 4M image-text pairs on 4M images.

Implementation Details  The maximum sequence length for the sentence is set as 20. We use Faster R-CNN (Ren et al., 2015) (with ResNet-101 backbone (He et al., 2016)) pre-trained on Visual Genome dataset (Krishna et al., 2017) to detect the objects and extract the region features. We consistently keep 100 objects for each image to maximize the pre-training compute utilization by avoiding padding. For the model architecture, we pre-train a 12-layer SemVLP-base model with hidden size of 768, where we initialize it with the parameters from StructBERT base model (Wang et al., 2019). We set $L_{s}=6$, which obtains the best performances on the development set of the downstream tasks, at a proper semantic level for cross-modal fusion ¹¹1We only introduce the cross-modal attention from text space to image space due to the superior performance in our framework, where modeling of vision modality is emphasized.. We train SemVLP model with a total batch size of 256 for 40 epochs on 4 V100 GPUs. The Adam optimizer with initial learning rate of 1e-4 and a learning rate linear decay schedule is utilized.

We compare SemVLP model against other state-of-the-art single-stream and two-stream cross-modal pre-training models of the comparable model size ²²2Most of the compared models have similar model size as 12-layer BERT-base. on the following downstream tasks. The details on these tasks and fine-tuning configurations can be found in the supplementary material.

• $\bullet$

  VQA v2.0 Antol et al. (2015): A visual question answering task/dataset that asks a model natural language questions on a given image.

• $\bullet$

  Image-Text Retrieval: We test on the popular Flickr30K dataset (Young et al., 2014).

• $\bullet$

  NLVR2 Suhr et al. (2018): A visual reasoning task that aims to determine whether a natural language statement is true about a pair of images.

• $\bullet$

  GQA 2019 Hudson and Manning (2019b): An image question answering task/dataset that emphasizes on the reasoning capability of a model to answer a question.

The results on the four downstream V+L tasks are shown in Table 1,2,3 respectively. We can see that: (1) Among all the VLP models of similar size to BERT base, SemVLP consistently outperforms other strong single-stream and two-stream VLP methods (e.g., UNITER (Chen et al., 2019b), OSCAR (Li et al., 2020) and 12-in-1 (Lu et al., 2020), ERNIE-ViL (Yu et al., 2020)) on all the examined tasks, which validates the effectiveness of SemVLP on combining single-stream and two-stream architectures to align semantics at multiple levels. (2) With much less parameters, the single-stream architecture can achieve comparable performance to the two-stream architecture, which is more parameter efficient. The proposed SemVLP model can be easily adapted to either architecture according to the typical scenario. By sharing parameters, SemVLP can also be parameter efficient while keeping superior performance. It is partially because SemVLP is pre-trained to align cross-modal semantics at multiple semantic levels, which makes the learning of semantic alignments more robust toward the diverse image-text pairs.

To validate the effectiveness of aligning cross-modal semantics at multiple levels, we conduct in-depth analysis on pre-training at different semantic levels with various architectures.

Analysis on Various Pre-training Architectures  We first examine the importance of pre-training at multiple semantic levels by conducting ablation study on the pre-training fashions. Specifically, we pre-train the SemVLP model with only one type of model architecture each time and test the performance on the downstream tasks. All the pre-training settings are kept the same as in the original SemVLP pre-training. As shown in Table 4, both the two pre-training fashions play important roles in obtaining the full SemVLP model, and removing each task will consistently decrease the final downstream task performance. The single-stream architecture is used to align fair-grained feature-level semantics, while the two-stream architecture helps align semantics at a higher-level semantic space. By iterative training with a shared set of Transformer parameters, the proposed SemVLP model can take the advantage of both the single-stream architecture and two-stream architecture towards more robust vision-language pre-training.

Analysis on Different High-level Semantic Alignments  There are many different ways for high-level semantic alignment, now we further analyze the advantage of our two-stream architecture and the specific “point” to conduct modality fusion with the cross-modal attention module. Therefore, we pre-train SemVLP with only high-level semantic alignment and examine in which layer to introduce the cross-modal attention module by setting different $L_{s}$. The pre-training details are kept the same as in the original SemVLP pre-training. We test the performance on VQA and NLVR2 tasks, and the results are shown in Figure 3. We can see that by introducing the cross-modal attention module properly, the two-stream mode of SemVLP method obtains significantly better performance than the previous two-stream model LXMERT. The best performance is obtained when $L_{s}=6$, where the separated image/text encoding and cross-modal attention is equally emphasized. It again demonstrates the importance of aligning cross-modal representations at a proper semantic level.

After SemVLP is fully pre-trained at multiple semantic levels, we further examine what fine-tuning architecture/mode is more appropriate for the downstream tasks. We keep the fine-tuning setting as the same with the original setting in the supplementary material, and examine the performance using different architectures. Table 5 shows that on all the examined tasks, the single-stream fine-tuning architecture gives better performance than the two-stream one does. This is due to the fact that: (1) by learning cross-modal fusion from a fine-grained feature-level, the single-stream mode can capture full association across modalities from more basic semantic granularity with powerful self-attention mechanism, which originates from the success of BERT. (2) Our design of SemVLP shows more favor of the single Transformer encoder, which is well regularized when pre-training with both training modes via parameter sharing. The pre-training strategy tends to enhance the high-level semantic alignment into single-stream training process.

The motivation behind SemVLP is to align cross-modal semantics at multiple levels by taking advantages of both single-stream and two-stream architectures. As stated above, single-stream and two-stream models are good at feature-level and high-level semantic alignments, respectively. To verify this, we visualize the attention map on the same layer and head of both the single-stream model and SemVLP for image objects and its associated description, as shown in Figure 3. A darker color indicates a higher attention weight. Take the image on the left of Figure 3 as an example. In order to align the objects and the description, a model needs to capture the high-level text semantics: “Two people” are standing and playing games, and “a person” is sitting on the couch. For Object 1 sitting alone, the single-stream model mis-attends to “people” in the description. SemVLP, on the other hand, properly pays high attention to “person”. For Object 2 standing alongside another, our SemVLP attends to “people” correctly, while the single stream model fails to do so. This example shows that the single-stream cannot differentiate between the semantics of the “person” and “people” at the high level, which leads to the false alignments. The same pattern can be observed from the image on the right of Figure 3.