Survey of Visual-Semantic Embedding Methods
for Zero-Shot Image Retrieval

In the method of mapping global representations, features from the whole image and features from the text are transformed into a common space, and their similarity is measured.

An early method for mapping images and language into a common space is deep visual-semantic embedding (DeViSE) [1], proposed by Frome et al. DeViSE embeds images and their corresponding words in the same space and trains them so that their similarity is high. Image features were extracted from convolutional neural networks (CNNs) trained with ImageNet, and word features were extracted from models trained with the skip-gram language model. For $z_{i}=Mx$, which is the feature $x$ extracted by inputting image $i$ into the CNN and multiplying by the transformation matrix $M$, for $z_{t}$, which is the feature obtained from word $t$, the cosine similarity is calculated as follows:

For every negative example caption $\hat{t}$ for image $i$, the matrix $M$ that linearly transforms the image features into the word feature space is trained using the following hinge rank loss:

where $\alpha$ is the margin.

Unifying visual-semantic embeddings (UVS) [2], proposed by Kiros et al., uses a CNN trained on ImageNet (AlexNet [3]) for image feature extraction and LSTM for text feature extraction. Let $x,y$ be the features of the image and text, respectively, and $M_{x}$ and $M_{y}$ be the transformation matrices, respectively. Then, the vectors in the embedded common space can be denoted as $z_{i}=M_{x}x$ and $z_{t}=M_{y}y$. The similarity between image data $i$ and text data $t$, ${\rm sim}(i,t)$, is calculated using the cosine similarity of vectors $z_{i},z_{t}$ in the common space. The loss function is a bi-directional hinge rank loss, as shown below:

where $\hat{i}$ and $\hat{t}$ are data unrelated to $t$ and $i$, respectively.

A multimodal recurrent neural network (m-RNN) [4] has also been proposed; it introduces a multimodal layer to a recurrent neural network (RNN) and has a mechanism to input image features obtained from the CNN.

Deep structure-preserving embedding (DSPE) [5] has also been proposed; it includes a constraint that preserves the correspondence between different modalities (image and text) even when training the correspondence between the same modality.

Improved visual semantic embedding (VSE++) [6] is an improved version of the UVS method described above, with the following changes to the loss function:

UVS sums over all the data, while VSE++ includes only the data with the largest loss at each data point. This allows us to utilize hard negatives and has been shown to significantly improve accuracy, regardless of the image feature extraction method used, such as VGG or ResNet.

A generative cross-modal feature learning framework (GXN) [7] has also been proposed; it incorporates generative processes into the training of cross-modal feature embedding.

Dual attention networks (DANs) [8], which aim to capture the detailed interactions between images and text, incorporate attention mechanisms in both images and text, and experiments have shown their effectiveness. DANs involve multiple steps to collect essential information from both modalities, focusing on specific regions of the image and words in the text.

Text-image modality adversarial matching (TIMAM) [9], which uses adversarial correspondence, has also been proposed to train modality-invariant feature representations. The paper also describes the successful inclusion of BERT as a language model.

Methods for extracting global representations of images and text are unable to find the relationship between objects in an image and words in a sentence, which limits the accuracy of image-text matching. For this reason, many methods have been proposed to measure relevance by mapping parts of an image or a sentence and aggregating their similarities.

Karpathy et al. were the first to locally map images to text by optimizing the correspondence between the most similar image regions and word pairs with deep fragment embeddings [10].

Deep visual-semantic alignments (DVSAs) [11] estimate the latent alignment between regions in an image obtained by region CNN (R-CNN) and words in a sentence obtained by a bidirectional recursive neural network (BRNN). This allows us to deal with sentences whose meaning is interpretable and whose length is not fixed.

A method called semantic concepts and order (SCO) [12] has also been proposed. This method improves the image representation by training semantic concepts and then organizing them in the correct semantic order.

In the above methods, the similarity was calculated for all possible pairs without focusing on regions in the image or words in the sentence. In contrast, the stacked cross attention network (SCAN) [13], a method to improve interpretability by focusing on regions of an image or words in a sentence using the bottom-up attention model [14], has been proposed. SCAN has been used as a baseline for many methods and has led to technological developments since its proposal. Examples include the bidirectional focal attention network (BFAN) [15] and the position focused attention network (PFAN) [16]. In BFAN, irrelevant image regions and words cause deterioration in the correspondence between images and text; thus, they are removed. PFAN trains the correspondence between images and text using the positions of objects in the images as cues.

Training coarse correspondences based on object co-occurrence statistics does not allow for fine phrase correspondences. To train more detailed correspondences, a graph structured matching network (GSMN) [17], which models objects, relationships, and attributes as structured phrases through node- and structure-level correspondences, has been proposed and demonstrated greatly improved accuracy.

Scene concept graph (SCG) [18], which extends visual concepts and enhances image representation using image scene graphs as external knowledge, is another effective method. Another method, consensus-aware visual-semantic embedding (CVSE) [19], is inspired by the SCG method and is the first attempt to map images to text using common sense information. CVSE utilizes consensus information by computing the statistical co-occurrence correlations between the semantic concepts from an image caption corpus and expanding the constructed concept correlation graph to generate consensus-aware concept (CAC) representations.

Currently, because of the availability of large datasets of image and language pairs and the success of large pre-trained models such as transformers [20] and BERT [21] in the field of natural language processing, image retrieval using pre-trained models is becoming mainstream, as shown in Fig. 3. Pre-training models are mostly based on multi-layer transformers, and typical datasets used include Flickr30k [22], MSCOCO [23][24], Conceptual Captions [25], SBU Captions [26], and graph question answering [27]. In the following, we introduce several methods that have been proposed in recent years.

Vision and language BERT (ViLBERT) [28], which extends the BERT model to a multimodal model of images and text, can train common task-independent representations of images and natural language. In ViLBERT, the image and text inputs are processed in separate sequences by the transformer, after which the cross-modal relationship is trained by the co-attentional transformer, as shown in the lower part of Fig. 3. Experiments show that it is possible to achieve higher accuracy than when images and text as a single sequence and training them with a single BERT. The authors used the Conceptual Captions dataset to pre-train ViLBERT and then fine-tuned it for tasks such as VQA, visual sense reasoning (VCR), and image retrieval to achieve the highest accuracy.

Unicoder-VL [29] is a universal encoder that aims to pre-train common visual and semantic representations using a large image caption dataset. Language and images are input to a single transformer model with multiple sequential encoders, which train a common embedding. The following three tasks were used for pre-training:

• •

  Masked Language Modeling (MLM): Predicting masked input token

• •

  Masked Object Classification (MOC): Predicting features of masked objects

• •

  Visual-Linguistic Matching (VLM): Predicting whether image and text pairs are relevant

ImageBERT [30], a transformer-based model trained on multiple tasks simultaneously, achieves high retrieval accuracy by crawling a large number of image-text pairs from the Web for pre-training and fine-tuning them in the image retrieval and text retrieval tasks. The four tasks used in the training are as follows:

• •

  Masked Language Modeling (MLM): Predicting masked input token

• •

  Masked Object Classification (MOC): Predicting labels of masked objects

• •

  Masked Region Feature Regression (MRFR): Predicting features for object locations

• •

  Image Text Matching (ITM): Predicting whether image and text pairs are relevant

UNiversal Image-TExt Representation (UNITER), which pre-trains BERT models for four tasks using a large database of image/text pairs, has also been proposed. The following four tasks are used for pre-training:

• •

  Masked Language Modeling (MLM): Predicting masked input token

• •

  Image Text Matching (ITM): Predicting whether image and text pairs are relevant

• •

  Word-Region Alignment (WRA): Training to optimize the alignment between words and image regions

• •

  Masked Region Modeling (MRM): Training to reconstruct the masked image

A pre-training task was designed to map visual and language input at both the global (image-text) and local (word-region) levels.

In the above methods, the features of the image domain and the text are concatenated and fed into the model, and the model trains the semantic association between the image and the text in a brute force fashion using self-attention. In contrast, object-semantics aligned pre-training (Oscar) [32] uses object tags detected in the image as anchor points, which greatly improves the generalizability of the pre-trained models. The pre-training task performed well with only two loss functions: masked token loss and contrastive loss. Masked token loss considers object tags as language data and trains the system to mask and recover the input text and object tags. Contrastive loss takes an object tag as image data, replaces it with a different tag sequence with 50% probability, and trains it to predict whether the object tag is appropriate.

The contrastive-language-image pre-training (CLIP) [33] proposed by OpenAI has also become a hot topic in the field of image and text retrieval since the beginning of 2021. CLIP achieves zero-shot, highly accurate image retrieval without fine tuning by pre-training on a large number of text/image pairs (over 400 million).

We have introduced some recent transformer-based methods for image retrieval, but many more methods have been proposed. In addition to image retrieval, research on transformers in computer vision was presented in [34]. Among the methods using transformers, there are methods that treat images and language as one sequence, as typified by UNITER and Oscar, and methods that treat images and language as separate sequences, as typified by VilBERT, but no conclusion has yet been reached as to which is more appropriate. There is also room to consider whether the region obtained by object detection should be used as the image sequence input to the transformer, or whether the entire image region should be used by a grid-based method.

Databases that are frequently used in training models to generate descriptions (captions) from images include Flickr8k [35] and Flickr30k [22].

Flickr8k is a crowdsourced dataset of 8,092 images with five corresponding human-generated captions per image, focusing on people or animals (e.g., dogs) performing some action. Of the total dataset, 6,000 images are often used as training data, 1,000 images as validation data, and 1,000 images as test data.

Flickr30k is a dataset that extends Flickr8k by annotating each of the 31,783 images with five sentences, in the same way as Flickr8k. Of the total dataset, 28,000 to 29,783 images are often used as training data, 1,000 as validation data, and 1,000 as test data.

Microsoft common objects in context (MSCOCO) is a dataset created for object recognition, object detection, and caption generation in images, and it is also frequently used for image-to-language matching and retrieval. In total, the dataset consists of more than 300,000 images, but the most commonly used are the 123,287 images in the MSCOCO 2014 dataset and the caption data [24], which contains five descriptions for each image. The original database consisted of 82,783 training data and 40,504 validation data, but it has become standard practice to use 5,000 of the validation data for testing and the remaining $82,783+40,504-5,000=118,287$ for training data. Test data for the task of image retrieval are often evaluated by averaging the accuracy of five captions over 1,000 or 5,000 images.

A well-known dataset that assigns descriptions to regions in an image is Visual Genome [36], which was created in Li Fei-Fei’s lab at Stanford University. The number of images is 108,077, but the number of labels is overwhelmingly large, with approximately 5.4 million captions assigned to image regions. In addition, there are approximately 2.8 million labels for attributes of objects in the image, and approximately 2.3 million labels for relationships among objects.

Conceptual Captions [25] is a dataset of over three million image and caption pairs published by Google. The images and captions were collected from the Internet and thus contain a wide range of expressions. The captions were obtained from the HTML alt attribute associated with each image. Because of the very large dataset, it is used to train the transformer-based pre-training model shown in Subsection I-C.

SBU Captions is a database of one million images with one caption for each image. The captions are written by humans and filtered to leave only those with at least two nouns, noun–verb pairs, or verb–adjective pairs. This dataset has been used for research purposes with several tasks, including Google’s Show-and-Tell [37] and Microsoft’s UNITER.