Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignment

Existing works can be categorized into two types: the independent-embedding approaches and the interactive-embedding approaches. The former approaches [11, 10, 23, 45, 30] aim to project the images and texts into a common semantic space, so image-text pairs can be compared directly via simple distance metrics. The architecture of mainstream independent-embedding approaches is an independent-embedding learning structure consisting of an image encoder and a text encoder, and they adopt ranking loss [10] for metric learning. Though these approaches have achieved some promising performance, they are still limited since they cannot conduct interactive encoding process and thus fail to provide fine-grained alignment between images and texts.

Interactive-embedding approaches [18, 21, 25, 5, 16, 31] aim to learn fine-grained image-text matching by complex object and word interactions with cross-modal attention mechanism. Lee et al. [21] compute the similarities between regions and words, and only count the region-word pairs with high relevance. Some works [25, 5, 16] propose hierarchical interaction methods for progressively extracting the complicated correspondence. Recently, Qu et al. [31] propose a dynamic router with the capability to choose the different interactive mode for each image-text pair and achieves state-of-the-art performance. Nevertheless, the quadratic computational complexity takes unavoidable computational cost for retrieval. In this paper, we propose a model-agnostic module with multi-level self-supervised learning strategy for independent-embedding models to learn the fine-grained semantic correspondences, instead of time-consuming attention mechanisms. Thus, the independent-embedding backbones achieve superior retrieval accuracy without sacrificing efficiency.

To strike a balance between retrieval efficiency and accuracy, there are almost three types of approaches that have been proposed recently: late-interaction approaches [28, 26], two-stage approaches [12, 29], and intra-interactive embedding approaches [34, 17]. The late-interaction approaches retain the independent encoding architecture and perform lightweight token-wise interactions only in the late scoring stage [28, 26]. Their retrieval speed is still slower than independent-embedding methods since the independent-embedding methods only require the global embedding for similarity computation. Secondly, two-stage approaches [12, 29] first adopt independent-embedding models for coarse-level retrieval and then utilize interactive-embedding models for finer retrieval to trade-off between efficiency and accuracy. But they are still slower than independent-embedding models due to the existence of the re-rank stage. Lastly, the intra-interactive embedding approaches [34, 17] take two independent encoders but they require large-scale image-text pairs for training and stack a few Transformer blocks [37] to build stronger encoders. For example, ALIGN [17] leverages two transformers encoders with 400M parameters and 1.8B image-text pairs for training. Though they achieve surprising performance, the inevitable huge computation cost in training and deploying such massive-scale models limit their development. Contrast to these three types of trade-off methods, without sacrificing the retrieval time or requiring extra training data, our proposed module improves the accuracy by enhancing image and text representations of fine-grained information.

Self-supervised learning [6, 35, 46] aims at learning features without manual annotations. Recent approaches based on contrastive learning have achieved remarkable progress in visual domain. Current contrastive learning can be divided into two groups: individual-based contrastive learning [6, 13] and cluster-based contrastive learning [4, 3]. Individual-based contrastive learning [6, 13] considers each image in a dataset as its own class [4], and brings the embedding of different views from the same image closer and push embeddings from different images far apart using instance-level contrastive loss. This approach introduces the individual-level discrimination but requires a large batch size for negatives storage. Cluster-based contrastive learning [4, 3] encourages the image embeddings to be closer to their assigned prototypes obtained by clustering algorithm, and far from negative prototypes. This approach introduces the group-level discrimination between instances. However, current works only construct image-level representation learning and lack of fine-grained information learning such as the object information in images.

To learn local information, some researchers [41, 22] construct local level contrastive learning to learn visual pixel-level semantic information. Moreover, some works [14, 2] maximize global-local mutual information and aim to learn the shared context information across patches/tokens. Here, we aim to learn the fine-grained correspondences between images and texts without detail annotations, and our word-object alignment learning and context-level alignment learning were inspired by these local contrastive learning works. Differently, the contrastive reasoning should be performed across modalities instead of cross-image views, and thus the learning process is not fully symmetric since the local semantic information involved in image-text pairs is not exactly equivalent to that of two augmentations of an image.

Image Embedding Extraction. For each input image $I$, recent works [21, 23] usually employ an off-the-shelf object detection model, such as Faster R-CNN [33], to detect $M$ objects $O=\left\{o_{i}\right\}^{M}_{i=1}$, where each object $o_{i}$ is represented by an object feature embedding $\bm{o}_{i}\in\mathbb{R}^{d_{o}}$. Then a linear projection is utilized to transform $\bm{o}_{i}$ into a $h$-dimensional embedding. Then the embedding of the image is represented by a set of object embeddings $\bm{V}^{l}=\left\{\bm{v}_{i}^{l}\right\}^{M}_{i=1}$. We name this encoding process as visual object encoding stage. Then different works utilize various approaches such as GCN [23] or Transformers [23] to model the relationships between objects and obtain contextualized object embeddings, named as visual context encoding stage. Here, we omit the computation details and simplify them as the visual context encoder, and the output of the context encoder is denoted as visual context embeddings $\bm{V}^{c}=\left\{\bm{v}^{c}_{i}\right\}^{M}_{i=1}$. Finally, the visual global embedding $\bm{V}^{g}$ is obtained by integrating the context embeddings $\bm{V}^{c}$, which is named as visual embedding aggregation stage.

Text Embedding Extraction. For each input text $T$, the word-level embeddings $\bm{W}=\left\{\bm{w}_{i}\right\}^{D}_{i=1}$ is generally obtanined from the pre-trained word embeddings [23, 21], such as BERT [8]. Then an embedding layer is also employed to transform $\bm{w}_{i}$ to a $h$-dimensional vector. The word embedding set is denoted as $\bm{T}^{l}=\left\{\bm{t}_{i}^{l}\right\}^{D}_{i=1}$. We name this encoding process as the textual word encoding stage. Then different works utilize different approaches, such as GRU [21, 23] and Transformer [30, 17], to capture the contextualized word embeddings, named as the textual context encoding stage. We denote the output of this stage as textual context embeddings $\bm{T}^{c}=\left\{\bm{t}^{c}_{i}\right\}^{D}_{i=1}$, where $D$ denotes the number of the words in the sentence. The final global embedding $\bm{T}^{g}$ is obtained by aggregating the context embeddings $\bm{T}^{c}$, named as texual embedding aggregation stage.

Local Concept Alignment (LCA) sub-module is injected into the object and word encoding layer of the baseline model and aims to force the consistency between the visual and textual concept embeddings, as presented in Figure 2 (b). We regard the object regions as the visual concepts and the whole words in the sentences as textual concepts to make full use of information in the whole sentence. Since there are no word-object pair annotations, a word-object correspondence discovery strategy is first proposed to build pseudo word-object correspondences in LCA. Then a cluster-based fine-grained alignment is designed to force the consistency between the selected matched word-object pair by comparing their cluster assignments. Though there are no annotations of the word-object pairs, our concept alignment is under the constraint of global image-text matching supervision.

Word-Object Correspondence Discovery. We take the visual object embeddings $\bm{V}^{l}$ and word embeddings $\bm{T}^{l}$ as the input of the LCA sub-module. Since there are no word-object annotations, we first estimate the word-object correspondences from image-text pairs. Specifically, we compute the cosine similarities between the words and objects and choose the most similar object as the correspondence for each word. Notably, since the texts of an image is the linguistic expression of the image itself [48, 24], each word is able to find at least one matched object region given an image-text pair. We cannot guarantee each object in the image has corresponding words in the text. Therefore, we align each word to an object rather than the inverse direction.

Formally, given the $i$-th word embedding $\bm{t}_{i}^{l}$, we compute its cosine similarities with the region embeddings $\bm{V}^{l}$ and apply the argmax operation to select the most matched object region:

where $\bm{v}_{j^{+}}$ is the selected matched image object region for $\bm{t}_{i}$ and ${\rm cos}(\cdot,\cdot)$ represents the cosine similarity function. We utilize the whole word-object pairs obtained by all words in a batch as pseudo-supervised alignment information for local concept-level alignment.

Cluster-based Fine-grained Alignment. Cluster-based fine-grained alignment aims to enforce the consistency between each word embedding and the matched object embedding selected by the word-object correspondence discovery strategy. Inspired by the recent success of self-supervised learning [6, 35], we treat the selected matched word-object pair as two different views depicting the same conceptual semantic and utilize cluster-based contrastive learning [4, 3] to align their embeddings across different modalities. The basic idea lies in that we cluster object region embeddings to form a concept dictionary, where each cluster center represents a concept. The embeddings of the cluster centers are more stable and representative to represent concepts compared with the object region embeddings in each image. Each region embedding and word embedding are then mapped to the concept dictionary and represented by concept assignments. In this way, we represent regions and words at the semantic-consistent concept level instead of the expression-variant instance level. After that, we utilize cross-prediction to align the region and word embeddings. Notably, there is an alternative solution to align the embeddings, named as individual-based contrastive learning [6, 13]. However, individual-based contrastive learning requires a large number of paired negative-positive samples, and it is challenging to select the appropriate negative samples without ground-truth word-object annotations. Thus, to efficiently and effectively align the word-object embeddings, we adopt cluster-based contrastive learning without requiring numerous computations. It is an alternative to contrasting multiple views by comparing their cluster assignments instead of their instance-level embeddings [4].

The cluster-based fine-grained alignment process contains two main steps: (1) We first perform concept codebook construction to obtain the concept assignments, where the concept codebook is a collection of learnable prototypes based on object region embeddings. (2) Then cross-prediction learning is conducted from word concept assignments to object concept assignments to align the word-object embeddings.

Step 1: concept codebook construction. To learn the concept codebook and let the learning process of cluster centers be synchronous with the whole network, we use the online clustering method as in [4]. Following [4], we define the trainable concept codebook $\bm{C}=\left\{\bm{c}_{i}\right\}^{K}_{i=1}$, where $K$ represents the number of concept centers and each center $\bm{c}_{i}$ is a $h$-dimensional vector as visual and textual concept embeddings.

Step 2: cross-prediction learning. We use cross-entropy loss between object and word concept assignments to set up this cross-prediction problem. Firstly, given the concept codebook, the visual concept assignment $\bm{v}_{i}^{P}\in\mathbb{R}^{K}$ for the object embedding $\bm{v}_{i}^{l}$ can be obtained by mapping $\bm{v}_{i}^{l}$ to the concept codebook. Each scalar $\bm{p}_{i}^{k}$ in $\bm{v}_{i}^{P}$ represents the probability that $\bm{v}_{i}$ belongs to the $k$-th prototype $\bm{c}_{k}$ and is obtained by taking a softmax of the cosine similarity of $\bm{v}_{i}^{l}$ and $\bm{c}_{k}$ as follows:

where $\tau_{1}$ is a temperature parameter. Similarly, the word assignment is computed by mapping each word embedding $\bm{t}_{i}^{l}$ to the concept codebook, and each scalar of the probability that $\bm{t}_{i}^{l}$ belongs to the $k$-th prototype, denoted as $\bm{q}_{i}^{k}$.

Then cross-prediction from word concept assignments to the corresponding object concept assignments is performed to align the selected word-object pair. The prediction problem is optimized by the cross-entropy loss as follows:

where $\bm{p}_{j^{+}}^{k}$ is the object concept assignment of the matched visual object embedding $\bm{v}_{i^{+}}^{l}$. Finally, we aggregate the loss values in Equation 3 over all the words $D$ and result in the following loss function to optimize the feature encoders:

Since semantically similar concepts have different semantics in different contexts, Context Relation Alignment (CRA) sub-module is further proposed to capture context-level semantic correspondences in the context embedding layer, as presented in Figure 2 (c). CRA first performs shared context enhancement to capture the shared context-level information and suppress the irrelevant information between the images and texts. Shared context alignment is then conducted to achieve contextual-level alignment.

Shared Context Enhancement. Given the visual context embeddings $\bm{V}^{c}=\left\{\bm{v}^{c}_{i}\right\}^{M}_{i=1}$ and text context embeddings $\bm{T}^{c}=\left\{\bm{t}^{c}_{i}\right\}^{D}_{i=1}$, we acquire the global context embedding $\bm{v}^{c}_{g}$ and $\bm{t}^{c}_{g}$ for contextual relation alignment by adopting the average-pooling over $\bm{V}^{c}$ and $\bm{T}^{c}$ respectively. Here, we design two symmetric contrastive mechanism to learn the shared context information from visual perspective and textual perspective: contrasting between the visual global context embedding and the textual local context embeddings, denoted as V-global/T-local and contrasting between the textual global context embedding and the visual local context embeddings, denoted as T-global/V-local. Under the global supervision from one modality, the relevant local semantics of objects and relationships in the other modality will be strengthened while irrelevant semantics will be weakened.

Formally, a fully connected layer is first utilized to map the global context embedding into the corresponding local context space as the global supervision:

where $\sigma$ is the ReLU activation and BN is the batch normalization. $\mathbf{W}_{1}$, $\mathbf{W}_{2}$, $b_{1}$ and $b_{2}$ are learnable parameters. Meanwhile, the batch normalization and ReLU activation operations are employed on the local contextual embeddings ${\bm{v}_{i}^{c}},{\bm{t}_{i}^{c}}$, and they are then denoted as ${\bm{v}_{i}^{*}},{\bm{t}_{i}^{*}}$ in contrastive learning.

Given the global context and local context embeddings, the T-global/V-local contrastive learning and V-global/T-local contrastive learning is conducted in parallel. For T-global/V-local contrastive learning, we consider the textual global supervision embedding $\bm{t}_{s}^{c}$ and the visual local contextual embeddings $\bm{v}_{i}^{*}$ that come from an image-text pair as a positive sample, while the embeddings from unpaired image-text pairs as negative samples. Then we define the T-global/V-local contrastive loss as:

where $\tau_{2}$ is a temperature hyper-parameter and $M$ is the number of visual regions. $\left\{\tilde{\bm{v}}_{j}^{*}\right\}^{N}_{j=1}$ is a set of negative visual local embeddings. $N$ is the number of negatives. Similarly, the V-global/T-local contrastive loss is defined as:

where $\left\{\tilde{\bm{t}}_{j}^{*}\right\}^{N}_{j=1}$ is negative textually local context embeddings. $D$ is the number of textual words. We select the $N$ negative local embeddings that are most similar to the global embedding of the other modality ranked by cosine similarity. As a result, these samples are hard negative samples which are beneficial for learning high-quality representations of the anchor sample [47]. The loss for shared context enhancement is defined as:

Shared Context Alignment. Shared context alignment aims to perform contextual level alignment based on the enhanced global context embedding and local context embeddings. Specifically, both of the V-global/T-local and the T-global/V-local contrastive learning enhance the global context embeddings $\bm{v}^{c}_{g}$ and $\bm{t}^{c}_{g}$ from local and global perspectives and introduce a paired final global context embeddings for context alignment. Taking T-global/V-local contrastive learning as an example, it enables the encoder to filter out irrelevant local contextual information which is not aligned with global contextual information of the other modality. We further use average-pooling on visual enhanced local context embeddings $\left\{\bm{v}_{i}^{*}\right\}_{i=1}^{M}$ to form the final visual context embedding, denoted as the visual local-aggregated context embedding $\bm{v}_{g}^{*}$. Given the text global context embedding $\bm{t}_{g}^{c}$ and its enhanced global context embedding $\bm{t}_{s}^{c}$, we perform the fusion operation as follows:

where $\bm{t}_{f}^{c}$ denotes the textual global-fused context embedding obtained by the text global context information $\bm{t}_{g}^{c}$ and its enhanced global context information $\bm{t}_{s}^{c}$ from visual modality. $g$ is a gating value to adaptively balance the importance of $\bm{t}_{g}^{c}$ and $\bm{t}_{s}^{c}$. $[\cdot,\cdot]$ means the concatenation operation.

Similarly, as for V-global/T-local contrastive learning, we obtain the textual local-aggregated context embedding $\bm{t}_{g}^{*}$ based on $\left\{\bm{t}_{i}^{*}\right\}_{i=1}^{D}$. By fusing $\bm{v}_{g}^{c}$ and $\bm{v}_{s}^{c}$, we get the visual global-fused context embedding $\bm{v}_{f}^{c}$ similar to Equation 9.

Based on these final global context embeddings, we obtain the contextual-level matching score for a given image-text pair ($I$, $T$), which is defined as:

Then we use the hinge-based triplet ranking loss [10] to enforce the contextual similarity of matched image-text pair to be higher than unmatched ones for contextual alignment:

where $\widetilde{T}=\operatorname{argmax}_{d\neq T}s(I,d)$ and $\widetilde{I}=\operatorname{argmax}_{j\neq I}s(j,T)$ are the hardest negatives in a mini-batch for a positive pair ($I$, $T$), and $\alpha$ is the margin parameter. Taking the loss objective for shared context enhancement in Equation 8 together, we define the total loss objective in CRA sub-module as:

To prove the effectiveness of our module SelfAlign, we case study on two typical independent-embedding models, the widely compared model VSRN [23] and the state-of-the-art independent-embedding model CAMERA [30].

VSRN with SelfAlign. For image encoding, VSRN [23] uses the object embeddings from an object detection model as visual inputs, followed by an FC layer and a graph convolution network with four layers to perform object-relation reasoning. Finally, VSRN utilizes a GRU unit to obtain the visual global embedding. For text encoding, VSRN exploits a word embedding layer to encode word-level embeddings and adopts an LSTM to obtain the contextualized word embeddings. Finally, VSRN utilizes the final hidden state of the LSTM as the textual global embedding. For the loss function, VSRN utilizes the sum of a matching loss on global embeddings via triplet ranking loss and a generation objective from images to texts to jointly align the images and texts. Here, we denote the loss function of VSRN as $\mathcal{L}_{base}$. We take the outputs of the FC layer in the image encoder and the outputs of word-embedding layer in the text encoder into LCA sub-module. And we regard the outputs of the last GCN layer and the LSTM as the input of the CRA sub-module.

CAMERA with SelfAlign. For the image encoder, CAMERA concatenates the object embeddings and the position embeddings of each object as visual inputs and an FC layer is followed to obtain the visual object embeddings. The word embedding inputs are obtained from the output of pre-trained BERT [8]. Then CAMERA adopts a self-attention layer to perform relation reasoning for image and text respectively. For the loss function, CAMERA adopts the sum of a triplet ranking loss on global embeddings and a diversity regularization loss for cross-modal alignment and the summarization of multi-view descriptions. Here, we also denote the loss function of CAMERA as $\mathcal{L}_{base}$. And we take the outputs of the FC layer as LCA sub-module inputs, taking the outputs of the self-attention layer as the inputs of the CRA sub-module.

The final training loss for baseline models with SelfAlign is defined as:

where $\mathcal{L}_{base}$ means the loss objectives of the baseline model. $\mathcal{L}_{LCA}$ and $\mathcal{L}_{CRA}$ are defined in Equation 4 and Equation 12.

In the inference stage, the similarity for each image-text pair of baseline with SelfAlign is computed as follows:

where $S_{b}(I,T)$ is the similarity score evaluated by the baseline model, and $S_{c}(I,T)$ denotes our proposed contextual-level similarity score according to Equation 10. It is noteworthy that since our module keeps the advantage of the independent encoding framework of the independent-embedding models, the queried image or text embeddings can pre-computed offline before the inference stage.

To verify the effectiveness of SelfAlign, we compare our results with the state-of-the-art models on Flickr30k and MS-COCO in Table I. This table is split into three blocks, from top to bottom, representing independent-embedding models, interactive-embedding models, and our models (i.e. VSRN with SelfAlign and CAMERA with SelfAlign), respectively. Notably, for a fair comparison with the interactive-embedding models, following [23], we achieve our ensemble models by averaging the predicted similarity scores of the two different models obtained by utilizing different seeds for training.

From the comparison between the first block and last block in Table I, we conclude that our module SelfAlign can remarkably improve baseline independent-embedding models on all the metrics, which proves the effectiveness of learning fine-grained correspondences of word-object and global-to-local. Specifically, SelfAlign consistently improves the performance of VSRN and CAMERA by 11.2% and 6.6% in terms of R@sum on Flickr30k, MS-COCO 1K, and MS-COCO 5K. For the strongest baseline model CAMERA, SelfAlign also achieves 9.1%, 4.2%, and 6.6% boosts in terms of R@sum respectively and achieves a new state-of-the-art performance in independent-embedding models. It is worth noting that SelfAlign helps the strongest baseline model CAMERA achieve 1.1%$\sim$3.1% boost in terms of R@1 on both retrieval tasks. We also find the performance improvements of R@1 are almost superior than the improvements of R@5/10, indicating that SelfAlign improves the capability of the baseline model to capture fine-grained discrimination on similar images or texts.

From the comparison between the second block and the last block, we conclude that with the help of SelfAlign rather than cross-attention mechanisms, the performance gap between the interactive-embedding models and the independent-embedding models is reduced by a large margin. Specifically, CAMERA with SelfAlign model outperforms the second-best interactive-embedding model ADAPT [44] by 6.7%, 4.8% in terms of R@sum on Flickr30k, MS-COCO 1K. Compared to the-state-of-art interactive-embedding model DIME [31], CAMERA with SelfAlign also achieves comparable performance in terms of R@1 on text retrieval on Flickr30K.

To verify the advantage of SelfAlign in keeping the efficiency of independent-embedding models, we construct retrieval latency comparison in the inference phase on baseline models, baseline with SelfAlign models and typical interactive-embedding models. Since the retrieval latency is composed of feature encoding latency and scoring latency, we construct the comparison from these two aspects as shown in Table II and Table III respectively. Since the interactions between the object and word can occur in the feature encoding stage only [43, 49, 44, 31], the scoring stage only [18, 15, 21, 42] or both stages [5, 40], we select representative interactive-embedding models from the above three kinds of methods with comparable accuracy as ours for retrieval latency comparison, e.g., DIME [31], PFAN [42], IMRAM [5].

Encoding Latency Comparison. Table II shows the online encoding latency given per query. Besides, Table II also reports the model parameter size which mainly affects the encoding time. Though SelfAlign adds about 36M and 27M parameters and 1$\sim$3ms latency to the baselines, VSRN and CAMERA respectively, the encoding latency of the baseline with SelfAlign models still keeps invariant to the number of candidates, which is much lower than the accuracy comparable interactive-embedding models. This is because SelfAlign preserves the independent feature encoding architecture and does not add any cross-modal interactions in the encoding stage. Therefore, the query embedding can be encoded independently without the interactions with the queried candidates, and the embeddings of the queried candidates can be pre-computed offline without the comsuming of online encoding latency. In contrast, those models performing cross-modal interactions during encoding stage, like IMRAM and DIME, need to encode each text-image pair, resulting in the encoding latency is linearly related to the number of candidates.

Scoring Latency Comparison. Table III shows the scoring latency of per query on 100K candidates. The scoring time is linear with the number of interactions between image embeddings and text embeddings. The results show that SelfAlign increases about 0.3ms latency to baseline models while both baseline models with SelfAlign are still 1000 times faster than the model PFAN and IMRAM, which have cross-modal token-wise interactions during the scoring stage. This is because SelfAlign still keeps the baseline model to perform the computation of similarity scores by using simple similarity calculation operations like dot product.

To prove the good balance between accuracy and efficiency for image-text retrieval of our module, we visualize the accuracy and efficiency jointly on Flickr30K and MS-COCO 5K in Figure 3. Specifically, the trade-off models include VisualSparta [28], ISERI_inflate [26], ISERI_fast [26], LightDOT [34]. VisualSparta [28], ISERI_inflate [26] are late-interaction trade-off models, where there are no cross-modal token-wise interactions in the feature encoding stage but in the scoring stage. LightDOT [34] is a pre-trained re-ranking based method, which utilizes a pre-trained independent-embedding model to coarsely rank in the first stage and then utilizes another pre-trained interactive-embedding model UNITER [7] to finely rank in the second stage. ISERI_fast [26] is also an independent-embedding model but is obtained by performing knowledge distillation twice under the extra model supervision of ISERI_inflate [26]. This enables ISERI_fast to obtain more accurate fine-grained information under the supervision of the teacher model ISERI_inflate compared with our models.

In Figure 3, the x-axis represents the retrieval complexity referring to the average number of cross-modal interactions per candidate when given a query to retrieve the ground-truth from 1000 candidates. The number of cross-modal interactions is the sum of the number of interactions in both of the feature encoding stage and the scoring stage. The y-axis denotes the results of R@1 in the image retrieval task. The red dots represent our models. As shown in Figure 3, SelfAlign enables both baseline models improving the accuracy without increasing retrieval complexity while other trade-off models sacrifice retrieval efficiency for cross-modal token-wise interactions. SelfAlign explores the cross-modal semantic correspondences and improves the retrieval efficiency by achieving fine-grained alignment matching fine-grained correspondences in the early feature encoding stage.

We conduct ablation study to evaluate the effectiveness of essential components in SelfAlign. The results on Flickr30k are shown in Table IV. We use CAMERA+SelfAlign as the full model for all the following variants:

• •

  w/o LCA (model ‘#1’): this model removes the Local Concept Alignment sub-module in SelfAlign.

• •

  w/o CRA (model ‘#2’): this model removes the Contextual Relation Alignment sub-module in SelfAlign.

• •

  w/ Concept O2W (model ‘#3’): this model changes the direction of concept alignment learning in LCA sub-module from word-object to object-word. Specifically, we choose the most similar word as the correspondence for each object and perform clustering on word embeddings for fine-grained alignment learning.

• •

  w/ Concept Dual (model ‘#4’): this model preserves these two directions of concept alignment learning.

• •

  w/ noun+adj+verb (model ‘#5’): this model performs concept alignment learning only with nouns, verbs and adjectives instead of all the words.

• •

  w/ noun (model ‘#6’): this model performs concept alignment learning only with nouns.

• •

  w/o $\mathcal{L}_{cs}$ (model ‘#7’): this model removes the global-to-local contrastive loss $\mathcal{L}_{cs}$ in Equation 8 for shared context enhancement in CRA sub-module.

• •

  w/o $\mathcal{L}_{ca}$ (model ‘#8’): this model removes that the contextual alignment loss $\mathcal{L}_{ca}$ in Equation 11 for shared context alignment learning in CRA sub-module.

• •

  w/o $\mathcal{L}_{tg}$ Atd Align (model ‘#9’): this model removes T-global/V-local contrastive learning attended context alignment. There is no projection for $\bm{t}_{g}^{c}$ in Equation 5, $\mathcal{L}_{tg}$ in Equation 6, text fusion in Equation 9 and the similarity computation in the first term of Equation 10.

• •

  w/o $\mathcal{L}_{vg}$ Atd Align (model ‘#10’): this model removes V-global/T-local contrastive learning attended context alignment, which is symmetrical with model ‘#9’.

Models in the first block are designed to evaluate the contribution of each sub-module in SelfAlign. We observe that the R@sum value of models ‘#1-#2’ all significantly decrease by 5.3% and 6.3%, but they still outperform the baseline model ‘#11’. It shows that both alignment sub-modules are effective for baseline to extract different levels of fine-grained correspondences and the combination of them can further improve retrieval accuracy. It proves the effectiveness and complementarity of concept level and contextual level alignment information learned in LCA and CRA sub-module.

Models in the second block evaluate the influence of the single direction and the effectiveness of utilizing all words to perform word-object alignment learning in LCA. The performance of either model ‘#3’ or model ‘#4’ decreases slightly by 2.7% and 2.4% respectively, which demonstrates word-to-object alignment captures more accurate concept alignment information. Besides, The performance of either model ‘#5’ or model ‘#6’ decreases by 5.0% and 6.1%, which demonstrates the effectiveness of making the most use of text information via utilizing all words to align regions in LCA sub-module.

Models in the third block evaluates the influence of the key components in the CRA sub-module. The performance of model ‘#7’ and model ‘#8’ decreases remarkably compared with the full model. The results show that both shared context enhancement by the global-to-local contrastive loss and shared context alignment learning are essential for contextual alignment. The performance of model ‘#8’ is decreased to the results as the baseline model ‘#11’ in terms of R@sum, which indicates that CRA sub-module only with global-to-local constrastive learning objective $L_{cs}$ is not only ineffective in context-level alignment but also damaging to the concept-level alignment in the LCA sub-module. The reason is that only with $\mathcal{L}_{cs}$ in the model ‘#8’ takes the local contextual embeddings from unmatched image-text pairs as the negative samples according to Equation $L_{cs}$ to learn the shared context information in the image-text pair, which ignores that these local contextual embeddings could mainly include the correct concept information in the concept alignment of the LCA sub-module. This conflict makes both the LCA and CRA sub-module ineffective. Compared to model ‘#2’, the performance of model ‘#9’ and model ‘#10’ improves slightly even when the models only preserve half part of CRA sub-module. These results verify the effectiveness of preserving the joint learning mode of global-to-local contrastive loss. Moreover, compared to the full model, model ‘#9’ and model ‘#10’ lead to the performance degradation, which demonstrates that these two context-level alignment are effective and capture complementary contextual information from vision global supervision and text global supervision.

Moreover, we observe that the LCA sub-module does not take extra parameters for the baseline model as it only involves in $\mathcal{L}_{LCA}$ defined in Equation 4 for local concept alignment learning. The CRA sub-module causes the increments of parameters and retrieval latency for baseline model CAMERA by 27.3M and 0.3ms, respectively, which is due to the feature linear projections defined in Equation 5 and feature fusion based on the gate mechanism defined in Equation 9.

Alignment visualization of the concept representation. We utilize t-SNE [36] to visualize region and word embeddings of some frequent concepts for intuitively analyzing the concept-level alignment, as shown in Figure 5. The embeddings are obtained by CAMERA model and CAMERA+SelfAlign model. We conclude that SelfAlign pulls the distance of same semantic concepts of different modalities. For example, in Figure 5 (b), the concept ‘water’, ‘dirt’, and ‘shorts’ are densely clustered by CAMERA+SelfAlign while CAMERA can not.

Alignment visualization in LCA and CRA sub-module. The detailed alignment process is interpretable by visualizing the conceptual and contextual similarity scores at LCA sub-module and CRA sub-module. The results are shown in Figure 4. Specifically, given an image-text pair in the inference phase, the most similar region for each word is identified in LCA sub-module. The global to local similarity map is computed in CRA sub-module, which supports the explicit visualization and reveals the alignment process. From the first four columns, we observe that the words align to the proper regions with the highest similarity. For example, in the first example, our model finds appropriately matched regions on nouns like ‘toddler’, ‘food’, ‘bowel’, as well as the action word like ‘mixed’. In the CRA sub-module, we observe that semantically matched image regions align well to the text global context embedding, while irrevevant image regions are suppressed. These examples prove that SelfAlign learns the concept-level word-object correspondences in LCA sub-module and composites the essential regions to understand the global context information in CRA.

Qualitative retrieval results analysis. The qualitative results from text-to-image retrieval and the image-to-text retrieval on Flickr30K are illustrated in Figure 6 and Figure 7, respectively. From both retrieved results, it’s clear that CAMERA+SelfAlign retrieves the correct candidates to a more forward position. Moreover, for those candidates with similar scenes, SelfAlign enables the baseline model to distinguish the fine-grained discrimination among the candidates well, which verifies that SelfAlign is effective in fine-grained cross-modal information retrieval.