HGAN: Hierarchical Graph Alignment Network for Image-Text Retrieval

Image-text retrieval aims at calculating the similarity between the images and texts, which can be mainly divided into three categories, global matching, regional matching and multi-level matching methods Global matching methods normally embed the global features of images and texts into a common embedding space, and then employ a ranking loss calculating the distance between image-text pair. Kiros et al. [19] utilize the convolution neural network (CNN) for image encoding as well as the recurrent neural network (RNN) for text encoding, and Faghri et al. [20] employ the ResNet152 and GRU modules to encode images and texts respectively. Regional matching methods cast lights on the fine-grained alignment of the regions of image and the words of sentence. Qu at ul.[21] adopt pyramid dilated convolution to obtain a multi-view representation from image region features, using the viewpoint that best matches the text feature to measure similarity. Chen et al. [22] design a generalized pooling operator to replace average pooling and max pooling, which can adaptively aggregate visual and language features to improve the performance of cross-modal retrieval. Multi-level matching methods consider multi-granularity feature alignment of image and text, or matching of global feature, region feature, and other intermediate feature. The second is similar to the ontology depicted in [23], whose hierarchical levels map different granularity of features. Qi et al. [24] consider not only the global and local alignments but also the relation alignment across images and texts, which can learn more precise cross-modal relevance. Zeng et al. [25] propose a multi-layer graph convolutional network with object-level, object-relational-level, and higher-level learning sub-networks to learn hierarchical semantic correspondences by both local and global alignment. Huang et al. [26] propose a bi-directional spatial-semantic attention network, which uses the word-to-regions relation to deduce the most relevant image regions, and employ the visual object to words relation to infer the close words for visual objects in the images. Recently, Diao et al. [27] use similarity graph reasoning and similarity attention filtration module to reason about the relationship between global-local alignment information, focusing on more informative alignments and have achieved SOTA performance.

Graph convolution network (GCN) has been successfully applied into many tasks in the cross-modal fields such as cross-modal retrieval[28, 29], image captioning [30, 31], visual question answering [32, 33] and visual entailment [34, 35]. Rather than merely focus on the similarities in content, GCN can discover the potential semantic relationships among different modalities and integrate the information of neighbor nodes. As a result, GCN is adopted into the field of cross-modal retrieval to learn the representations across different modalities, as it will be more accurate when learning local and stationary features on graphs. Yu et al. [36] adopt GCN to enhance the representation of text features, combining the strengths of structural information with semantic information. Li et al. [11] utilize GCN in feature reasoning to generate image features with semantic relationship between image regions, as response of each node is computed based on its neighbour nodes. Yao et al. [37] utilize a GCN-based encoder to refine the representations of each region proposed on objects with the learned region-level features. Yang et al. [38] delve into the details of improving semantic navigation using GCN by incorporating prior knowledge and updating it dynamically as the agent receives the current environment information. Wen et al. [39] design the dual semantic relation module through the graph attention network (GAT) [40], which aims to enhance the regional and global relations for more accurate visual and text representations. Liu et al. [41] design the graph structure for both the image and text to perform node-level and structure-level matching.

Since there exist abundant semantic information in images, we try to learn image representations by jointly considering the global and local features. Different feature encoders are employed to obtain the global and local features respectively, and then a unified image representation is obtained by using concatenate operation to construct the image feature graph.

The traditional text feature representation methods use the RNN-based models such as LSTM [46] or GRU [47], and treat the output of the network as the sentence feature. The language representation model BERT [6] uses the self-attention based transformer structure, which is accomplished at learning semantic relationships. The model has powerful feature extraction capabilities to generate deep bidirectional linguistic representations for word tokens. In this paper, we employ the BERT as the text encoder. First, the sentence is tokenized by WordPiece, and then the features of the word are extracted through the BERT model. Therefore, the output of the text encoder can be represented as $S=\{{{s}_{1}},{{s}_{2}},...,{{s}_{l}}\},{{s}_{i}}\in{{R}^{{{D}^{1}}}}$, where ${{D}^{1}}$ represents the dimension, and $l$ is the maximum number of words in the sentence. Then we use a fully connection layer to project ${{t}_{i}}$ into the $D$-dimensional embeddings:

where ${{T}_{S}}$ is the feature of the texts. ${{W}_{s}}\in{{R}^{D\times{{D}^{1}}}}$ represents the weight matrix, and ${{b}_{s}}$ represents the bias. The text feature can be represented as ${{T}_{S}}=\{t_{S}^{1},t_{S}^{2},...,t_{S}^{l}\},t_{S}^{i}\in{{R}^{{{D}^{S}}}}$. Then in order to build the text feature graph, we define the relationship ${E}_{S}$ between $t_{S}^{i}$ and $t_{S}^{j}$:

where $\odot$ represents the element product. Finally, the features $t_{S}^{i}$ and $t_{S}^{j}$ are regarded as the nodes, and the relationship ${{E}_{S}}$ between them is regarded as the edge to construct the text feature graph ${{G}_{T}}=({{T}_{S}},{{E}_{S}})$.

In this section, we introduce the designed multi-granularity feature aggregation and rearrangement (MFAR) module, which filters out the noisy parts of the global-local semantic alignment and retains the useful parts to accomplish multi-granularity feature fusion. For image and text feature graphs, we both use the MFAR module to conduct feature aggregation and rearrangement. Here the image modality is taken as an example to illustrate the details of MFAR module.

To align the global and local information simultaneously, we set three-level cosine similarity functions in the proposed model. ${{S}_{1}}$ calculates the similarity of image region features $V_{L}$ and text features $T_{S}$ and ${{S}_{2}}$ computes the similarity of image unified features $V_{U}$ and text features $T_{S}$. The similarity of image and text features $V$ and $T$ refined by the MFAR module are calculated by ${{S}_{3}}$. $S$ is the final similarity of image and text in the multi-granularity shared space, that is, the sum of the above three similarity functions:

where $(I,T)$ is the matched positive pair of image and text. And the formulations for the three cosine similarity functions are specified as follows:

Then, a bidirectional hinge-based triplet ranking loss [8, 20] is adopted to make the matched image-text pairs have higher similarity scores than unmatched ones.

where $d$ denotes the margin parameter, and ${{\left[x\right]}_{+}}\equiv\max(x,0)$. $I^{\prime}=\arg{{\max}_{X\neq I}}S(X,T)$ and $T^{\prime}=\arg{{\max}_{Y\neq T}}S(I,Y)$ denote the hardest negatives corresponding to the positive pair $(I,T)$.

We employ two commonly-used image-text retrieval datasets, MS-COCO (1K and 5K) [58] and Flickr30K [59], to evaluate our model.

The MS-COCO is a large-scale benchmark dataset used for image recognition, segmentation and retrieval. It is composed of 123,287 images, each with 5 corresponding captions. Following the experiment settings in [8, 11], we evaluate our method on 1K and 5K test images respectively. Specifically, the train, validation and test splits contain 113,287, 5000 and 5000 images. The 1K branch refers to the results are reported by averaging from 5 folds of 1K test images and the 5K branch refers to testing on the full 5K test images directly.

The Flickr30K dataset contains 31,783 images. Each image is paired with 5 corresponding captions. Following the split method in [20] about the Flickr30K dataset, we evaluate the performance of our model using 29,000 images for training, 1,000 images for validation, and the remaining 1,000 ones for testing.

For both image-to-text retrieval and text-to-image retrieval tasks, we report the results with the standard metrics, including R@K (Recall@K, K=1, 5, 10) and Rsum. R@K is defined as the proportion of correct image or text being retrieved among top K results, and Rsum is the sum of six R@K value to evaluate performance comprehensively.

In order to prove the effectiveness of the proposed HGAN method, we choose the models shown in Table I as the baseline models. In the Table I, we introduce the main works in the field of the image-text retrieval, including the publication year, the pros and cons, and the main content of each baseline model.

In this section, we describe the software and hardware configuration of the experiments in detail. Our model is evaluated in pytorch-1.7.1 with python wrapper and a machine with Intel Xeon Gold 6226R CPU, 64GB RAM, 1T SSD and NVIDIA Tesla A100 GPU. Specifically, for dataset Flickr30K and MS-COCO, the model is trained for 12 and 20 epochs with the adaptive moment estimation optimizer (Adam) [60], respectively. The batchsize is set to 256 and 320 for Flickr30K and MS-COCO datasets, respectively. The warmup is set to 0.1 and the learning rate is set to 0.0002 with a decay rate of 0.1 every 6 epochs. For images, the feature dimension ${{D}^{0}}$ is set to 2048 for both global and local features. The basic version of the pre-trained BERT [6] is used to extract text features, which includes 12 layers, 12 heads, 768 hidden units, and 110M parameters, to get the text embeddings with ${{D}^{1}=768}$. The image-text shared space dimension $D$ is set to 1024. In our model, we set the number of MFAR layers to 2 and 4 for the Flickr30K and MS-COCO datasets, respectively.

In this section, we show the experimental results on the MS-COCO (1K and 5K) and Flickr30K datasets in Table II and Table III. For the sake of fairness, we directly cite the simulation results of the baseline models in their corresponding original papers. The bold indicates the optimal results, and the underline indicates the suboptimal results. Overall, our model achieves the state-of-the-art retrieval results on the Flickr30K and MS-COCO 1K and 5K datasets.

Table II shows the performance of each model on the MS-COCO dataset. CSCC [57] has the optimal performance on the 1K test set, probably because it considers the syntactic alignment in addition to the fine-grained alignment, which is similar to our model. SGRAF [27] has the best performance on the 5K test set, which utilizes a vector-based similarity representation method to deduce more accurate matching score of images and texts through the fine-grained alignment. Overall, our HGAN model shows the best performance according to the Rsum metric. As for the 1K test set of MS-COCO, our model reaches 81.1% R@1 score and 67.4% R@1 score on image-to-text retrieval and text-to-image retrieval respectively, both outperforming other state-of-the-art methods. For the 5K test set, image-to-text retrieval and text-to-image retrieval achieve the best R@1 of 60.0% and 45.4% respectively, with 2.2% and 3.5% improvement over the SGRAF model. The performance of R@5 and R@10 can also be seen from the Table II, both of which achieve the optimal and suboptimal results.

Table III shows the performance of each model on the Flickr30K dataset. CAMERA [21] has the best performance on image-to-text retrieval, which aggregates context-enhanced visual information from multiple views of the image. CSCC [57] has the best performance on the text-to-image retrieval, which simultaneously considers the semantic information of the concept and syntactic. It is obvious that our HGAN model outperforms existing models by a large margin, achieving 80.3%, 96.5% and 98.3% for R@1, R@5 and R@10 on image-to-text retrieval. And the performance on text-to-image retrieval is 62.3%, 87.8% and 93.1% for R@1, R@5 and R@10 respectively. Compared to the CAMERA method, our model has a significant improvement in image-to-text retrieval and text-to-image retrieval tasks (by 2.3% and 2% on R@1). In summary, our HGAN model outperforms other state-of-the-art models on the Flickr30K dataset.

In this section, the ablation experiments are performed on the MS-COCO 1K and Flickr30K datasets. We analyze the effectiveness of the MFAR module and the unified features, the influence of the different similarity functions and the parameters of the model, respectively.

1) Effect of the MFAR module. In Table IV, the MFAR module is the proposed multi-granularity feature aggregation and rearrangement. The MFA module denotes the multi-granularity feature aggregation module and the MFR module is the multi-granularity feature rearrangement module. To demonstrate the effectiveness of the MFAR module, we disable the MFR module for performance testing to explore the impact of the designed modules. We can observe that our MFAR module has always outperformed the MFA-based model, which brings about a 2% performance promotion on two benchmark datasets.

2) Effect of the unified features. In Table IV, GIE denotes the global image embedding and LIE denotes the local image embedding. GIE+LIE is the unified image representation. We can observe that global matching using GIE module is not as effective as fine-grained matching using LIE module, and further, using the unified image feature including GIE and LIE module in our HGAN model has the optimal performance. Specifically, comparing the results of ‘MFAR+GIE’ and ‘MFAR+LIE’, it can be found that using only local features to construct image features is much better than using only global features, indicating that local features describe more image details and can generate more discriminative features. Furthermore, the comparison of the results of ‘MFAR+LIE’ and ‘MFAR+GIE+LIE’ shows that better results can be achieved by using both global and local features, which means that the global features supplement the contextual information lacked by the local features, allowing the reconstructed features to better represent the integral image and achieve improved retrieval results.

3) Effect of hierarchy. The hierarchy in our proposed method is directly reflected by multiple levels of similarity. Specifically, the three similarity functions ${{S}_{1}}$, ${{S}_{2}}$, and ${{S}_{3}}$ corresponding to three levels of fine-grained feature, unified feature and multi-granularity feature, respectively. In Table V, ${{S}_{1}}$ represents the similarity function between the image local feature and text feature, and ${{S}_{2}}$ works on the image unified feature and text feature. ${{S}_{3}}$ cannot be removed in our HGAN model, which represents the similarity function between image and text after optimization by MFAR module. The bold font indicates the optimal results. We can find that the application of multi-level similarity function achieves about 1.5% improvement on R@1 for image-to-text retrieval and text-to-image retrieval. In terms of performance improvement, ${{S}_{1}}$ is more effective compared to ${{S}_{2}}$, since ${{S}_{1}}$ takes into account the alignment of local features.

4) Effect of the parameters. In this section, we analyze the impact of model parameters on performance, including the number of layers of the MFAR module, the learning rate, and the batchsize. In Fig. 3, $M$ represents the number of layers of the MFAR module, and R denotes the initial learning rate. First, we can see that for the MS-COCO dataset, the performance is improved as the number of layers of the MFAR module increases, and it has the best performance when $M=4$. Due to equipment limitations, we have no way to conduct experiments with $M=8$, but we think that $M=4$ has achieved the ideal performance. For the Flickr30K dataset, $M=2$ works best because the two-layer MFAR module is sufficient to extract the information contained in the dataset. Also, setting an initial learning rate of 0.0002 is the most appropriate for several datasets of our model. Furthermore, Fig. 4 shows the optimal batchsize on several benchmark datasets. The best batchsize value is 320 for the MS-COCO dataset, and for the Flickr30K dataset, the optimal batchsize is 256.

In this section, we analyze the training process of the proposed HGAN model. Firstly, in order to measure the computational efficiency of our model, we consider the computational complexity of the model (measured by FLOPs) and the time complexity (measured by the number of parameters), which are 36.81G and 211.29M respectively. Compared to the trillions of parameters of large-scale pre-trained models in the image-text cross-modal domain, our approach achieves reasonable calculation consumption.

Subsequently, we analyze the change of Rsum values (the sum of recall value) with epoch and the change of loss values with iteration. Fig. 5 records the values of Rsum when varying the number of epochs on the MS-COCO 1K and Flickr30K test sets. The maximum value of the epoch is set to 20 and 12 for the MS-COCO 1K and Flickr30k datasets, respectively. For MS-COCO 1K, when the epoch reaches about 10, Rsum reaches the maximum and then stabilizes. For Flickr30k, the model has basically converged when epoch is 6 because it contains less image and text. Fig. 6 records the variation of loss values with different iteration numbers on the MS-COCO 1K and Flickr30K test sets. It can be seen that our model can converge to a satisfactory value on both the MS-COCO 1K and Flickr30K, and there is no underfitting and overfitting. Comparing the datasets MS-COCO 1K and Flickr30K, MS-COCO 1K needs more epochs of training because of its large amount of data and complex information. Nonetheless, the proposed model has better performance in MS-COCO 1k dataset because of the richer relations contained in.

In this section, we discuss the qualitative results of the proposed HGAN model. The retrieval results of two models are visualized, including the proposed HGAN model and the HGAN w/o MFR model. HGAN w/o MFR means that we disabled the multi-granularity feature rearrangement (MFR) module of the HGAN model.

In Fig. 7, we show the top 5 retrieval results for three query images. The correct matches are marked with a green “T”, the wrong matches are marked with a red “F”, and the whole incorrectly matched sentences are marked in red. The sentences on the left are the retrieval results of the proposed HGAN model, and the sentences on the right are the retrieval results of the HGAN model without MFR. We can observe that the HGAN model can retrieve the correct matching sentence in most cases, and the retrieval accuracy is significantly superior to the latter. In Fig. 8, the top 3 retrieval results for three query sentences are displayed. The correct retrieved images are highlighted in green boxes, while the wrong ones in red boxes. The upper column of images are the retrieval results of the proposed HGAN model, and the lower column of images are the retrieval results of the HGAN model without MFR. We can observe that our HGAN model can match the correct image in the top 3 results in Fig. 8. Besides, compared with the HGAN w/o MFR model, the image that matches the query sentence has a higher rank in our HGAN model.

We argue that there are two reasons for this phenomenon. First, the MFR module achieves multi-granularity feature denoising through feature rearrangement, which filters out the noisy parts and retain the dominate parts for the multi-granularity alignment of object-context-fused information. By mining the multi-granularity semantic relationship of multimodal features, the retrieval performance is improved. Then, the multi-level similarity functions can measure the image and text similarity at different levels in the multi-granularity shared space, which further improves the performance of ITR in terms of hierarchical alignment.

In this section, we perform visual analyses of the features modeled by our HGAN model to better demonstrate the effect of our model.

First, we display the detailed matching relationships between images and words. Specifically, we calculate the scores of image-word matching for four sample pairs, as shown in Fig. 9, where (a) and (b) are the samples for which our model retrieves the correct result at top-1, (c) and (d) are samples for which the top-10 items retrieved by our model do not contain correct results. The incorrect top-1 result given by HGAN is ‘A man buying some food at a food stand’ for (c) and ‘An adult skier carries a child skier under their arm on the slopes’ for (d), respectively. In each subplot, the selected image is shown on the left, the corresponding GT text is shown on the right, and in the middle is the similarity between the image and each word in its GT text.

All four subplots reveal that our model successfully recognizes the objects with key semantics in the samples, such as “dog”, “towl” in (a) and “women”, “donuts” in (c). Besides, since our model relies heavily on the regions obtained by object detection in constructing the features of images, it has poor understanding of articles, prepositions and verbs. For example, “the” in (b) and “with” in (d) mistakenly have the maximum similarity with the image.

Observing (c), (d) with their corresponding incorrect top-1 retrieval results, respectively, we can find that they describe a very similar scene, but there are detail errors and focus deviations. For example, “woman is smelling donuts” in the picture of (c) is wrongly judged as “the man is buying food”, and people in the background of the picture in (d) are mistakenly focused.

In conclusion, our model has an excellent ability to find objects and nouns containing important semantics in the sample to achieve cross-modal image-text matching. However, the poor understanding of articles, prepositions and verbs can lead to misunderstanding in semantic details, which needs further improvement.

Then, we conduct a T-SNE visualization experiment by using a subset of the MS-COCO test dataset as shown in Fig. 10. Concretely, we randomly chose samples from the MS-COCO test set with four class labels (aeroplane, boat, cat, and dog) and fed their data into our HGAN model to generate features. The high-dimensional features of the samples are transformed into two-dimensional by T-SNE and displayed as points in Fig. 10, where different colors used to differentiate their classes. For each of these four categories, a set of corresponding image and text samples labeled with (a)$\sim$(d) is displayed. In each subplot, we can find that the points of each color are aggregated in a single region, indicating that the model has learned the discriminative information belonging to different classes of samples. Comparing the points of same color between the left and right subplots, it is found that they appear in similar areas, indicating that the model achieves cross-modal semantic matching. In addition, the distribution of the yellow and green points is closer because the classes they belong to (cat and dog) are more similar. Moreover, some points appear in the wrong colour area, such as the red points (aeroplanes) that appear in the blue area (boats). We analyze that there are two reasons for this. One is that planes and ships often appear in similar scenes, such as the blue sky and the sea, which leads to the error of model judgment. Another reason is that the selected samples may contain multiple types of objects, so there will be some areas of color mixing in this experiment.