Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph

Entity-aware image captioning is attracting increasing attention in recent years. Some existing methods generate entity-aware captions by utilizing hashtags or named entities retrieved from the web [13, 32]. Several other methods extract the background knowledge from the associated news articles for generating the named entities [17, 13, 2, 6, 22, 30], which are closer to our method. Among these methods, the early works [17, 13, 2] generate entity-aware captions by two separate steps: a template caption is first generated with placeholders indicating the entity types, and then the placeholders are filled by selecting the named entities from the news article. For instance, Biten et al. [2] first generate a template caption by attending to both the image and the sentences in the news article, and then draw the named entities from the sentences with the highest attention weights.

More recently, a few end-to-end methods [6, 22, 30] are proposed to generate entity-aware captions in one pass. Tran et al. [22] encode the news article at word level using a pre-trained language model, and handle the rare words in the named entities using byte-pair encoding [19]. Hu et al. [6] first retrieve the sentences that are most relevant to the image, and then attend to the retrieved sentences.

Although remarkable progress has been achieved through leveraging textual information from the news article at the sentence or word level in the aforementioned methods, the explicit association between the named entities and visual objects is still under-explored, as well as the fine-grained relationship between named entities. This paper focuses on constructing a multi-modal knowledge graph to associate the visual objects with named entities and capture the entity relationships, simultaneously, which is important for generating informative captions.

Multi-modal knowledge graphs introduce information in multiple modalities, such as images, videos or text, to represent the entities and relations. Several recent studies have validated the effectiveness of multi-modal knowledge graphs in different fields. For instance, some methods [28, 15] incorporate images as additional features of the entities in the knowledge graph to learn better entity representations for knowledge graph completion and triple classification. Kannan et al. [8] construct multi-modal knowledge base to excavate the facts in deep learning literatures. A multi-modal knowledge graph based recommendation system is proposed in [21], where images and text entities are introduced to model user behaviour.

To the best of our knowledge, our method is the first to employ multi-modal knowledge graphs in entity-aware image captioning. Compared with the methods mentioned above, constructing multi-modal knowledge graphs for the images and the associated news articles is more challenging since the relationship between the entities in the news article and the visual objects in the image is unknown. Therefore, we leverage the external knowledge from the web to train a cross-modal entity matching module that establishes the connections between entities in different modalities.

In this paper, we propose a multi-modal knowledge graph (MMKG) for entity-aware image captioning. It explicitly models the association between named entities and visual objects in the image and simultaneously captures the fine-grained relationships between named entities in the article. MMKG consists of a text sub-graph and an image sub-graph. The text sub-graph models the interaction between the named entities in the article text, where the nodes represent the named entities and the directed edges represent the entity relationships. The image sub-graph represents the visual objects detected in the image. To connect these two sub-graphs for generating the complete knowledge graph, we introduce a cross-modal entity matching module that measures the similarity between the named entities in the text sub-graph and the visual objects in the image sub-graph. To facilitate the training of the cross-modal entity matching module, we collect an external multi-modal knowledge base from Wikipedia. Finally, the captioning model encodes the multi-modal knowledge graph as well as the image and article to generate the entity-aware captions. Figure 2 illustrates the framework of our method.

The external multi-modal knowledge base contains pairs of named entities and images, formulated as $D^{M}=\{({e}_{i},{v}_{i})|_{i}\}$, where ${e}_{i}$ and ${v}_{i}$ denote the named entity and the corresponding image, respectively. The named entities in $D^{M}$ are collected from the news articles in the training splits of the GoodNews and NYTimes800k datasets. We perform named entity recognition on all the articles and keep the concrete entities, including persons, organizations, artifacts and facilities. The abstract named entities, e.g. numbers and dates, are not included. To reduce the ambiguity in the original text, we perform entity linking [20] to connect the recognized named entities in the news article to Wikipedia entries.

Ideally, the image corresponding to a named entity should be representative, i.e. the most salient object in the images reflects the entity. For instance, the image that best reflects a person is the person’s portrait, and the image corresponding to a building is the close-up of the building. Note that the purpose of the images in each Wikipedia page is to demonstrate the named entity itself, and we believe that the images in $D^{M}$ collected from Wikipedia are with superior representativeness. For each entity ${e}_{i}$, we use the first image in the corresponding Wikipedia page as ${v}_{i}$ for the sake of simplicity. The multi-modal knowledge base contains 126,295 pairs of named entities and images in total.

The cross-modal entity matching module measures the similarity between a named entity ${e}_{i}$ and an image ${v}_{j}$. A pre-trained language model and a pre-trained CNN are used to encode the vector representations of ${e}_{i}$ and ${v}_{j}$, respectively, denoted as $\bm{u}_{{e}_{i}}$ and $\bm{u}_{{v}_{j}}$. The cross-modal entity matching module is trained to map the vectors $\bm{u}_{{e}_{i}}$ and $\bm{u}_{{v}_{j}}$ into a common embedding space, where the similarity between positive entity-image pairs is larger than any negative pairs by a margin $\delta$.

We use the pairs of named entities and images in the multi-modal knowledge base to train the cross-modal entity matching module. The loss function used to train the cross-modal entity matching module is formulated as

	$\displaystyle L_{r}=$	$\displaystyle\max_{{e}^{\prime}}(\delta+{\rm sim}({e}^{\prime},{v})-{\rm sim}({e},{v}))_{+}$		(1)
	$\displaystyle+$	$\displaystyle\max_{{v}^{\prime}}(\delta+{\rm sim}({e},{v}^{\prime})-{\rm sim}({e},{v}))_{+},$

where the pairs $({e}^{\prime},{v})$ and $({e},{v}^{\prime})$ denote negative samples, ${\rm sim}({e},{v})$ denotes the similarity between ${e}$ and ${v}$, and $(x)_{+}={\rm max}(x,0)$.

Compared with conventional knowledge graphs that only contains entities extracted from text, multi-modal knowledge graphs introduce additional modality of entities and relationships, such as images. Formally, a multi-modal knowledge graph is denoted as $G^{M}=\langle V,E\rangle$, where $V$ denotes the entity set and $E$ denotes the edge set. The entity set $V$ contains both the entities from the article and the visual objects from the image. We construct the multi-modal knowledge graph by first building a text sub-graph and an image sub-graph, and then connecting these two sub-graphs using the cross-modal entity matching module. An example multi-modal knowledge graph is shown in Figure 3.

To build the text sub-graph, denoted as $G^{T}=\langle V^{T},E^{T}\rangle$, where $V^{T}$ and $E^{T}$ denote the entities in the news article and the edges that connects them, we perform information extraction and coreference resolution by utilizing the Stanford CoreNLP toolkit [14]. Specifically, for an article $T$, the output of information extraction is a set of triples $R^{T}=\{\langle e^{h}_{i},e^{r}_{i},e^{t}_{i}\rangle|_{i}\}$, where $e^{h}_{i}$, $e^{r}_{i}$ and $e^{t}_{i}$ denote the head entity, the relation and the tail entity, respectively. For each triple $\langle e^{h}_{i},e^{r}_{i},e^{t}_{i}\rangle$, we create two directed edges $e^{h}_{i}\rightarrow e^{r}_{i}$ and $e^{r}_{i}\rightarrow e^{t}_{i}$. To simplify the graph structure, for the same entity that appears more than once in the article, we only keep the entity that appears first. For instance, for two triples $\langle e^{h}_{i},e^{r}_{i},e^{t}_{i}\rangle$ and $\langle e^{h}_{j},e^{r}_{j},e^{t}_{j}\rangle$ where $e^{h}_{i}$ and $e^{h}_{j}$ refer to the same entity and $e^{h}_{i}$ appears first in the article text, we remove $e^{h}_{j}$ from the knowledge graph and create an edge $e^{h}_{i}\rightarrow e^{r}_{j}$.

To construct the image sub-graph, denoted as $G^{I}=\langle V^{I},E^{I}\rangle$, we use the YOLOv3 object detector [18] and the MTCNN [31] network to detect the objects and faces in the image, and the detected objects and faces are denoted by $V^{o}=\{v_{i}^{o}|_{i}\}$ and $V^{f}=\{v_{i}^{f}|_{i}\}$, respectively. The entity set $V^{I}$ of the image sub-graph contains both the set of objects and the set of faces, i.e. $V^{I}=V^{o}\cup V^{f}$. For each object or face $v_{i}^{\sharp}$, we obtain its vector representation $\bm{u}_{v_{i}^{\sharp}}\in\mathbb{R}^{2048}$ using the pre-trained ResNet-152 model [4], where $\sharp\in\{o,f\}$.

The similarities between the entities in the image sub-graph and the text sub-graph are measured by the cross-modal entity matching module. For all the entity pairs $e_{i}^{*},v_{j}^{\sharp}$ that satisfies ${\rm sim}(e_{i}^{*},v_{j}^{\sharp})>0.4$, we create a directed edge $e_{i}^{*}\rightarrow v_{j}^{\sharp}$ that connects them, where $*\in\{h,r,t\}$.

The generated multi-modal knowledge graph $G^{M}$, the input image $I$ and the associated news article $T$ are encoded by the encoder of the entity-aware captioning model. The image $I$ is encoded using the pre-trained ResNet-152, and the output before the last pooling layer is taken and flattened into a matrix $\bm{X}^{I}=\{\bm{x}^{I}_{j}|_{j}\}$, $\bm{x}^{I}_{j}\in\mathbb{R}^{2048}$.

The news article $T$ is encoded by the pre-trained RoBERTa [12] into a sequence of subword units $\{w_{1},w_{2},...,w_{L_{T}}\}$, where $L_{T}$ denotes the length of the subword unit sequence. The output of the last layer is used as the representation, denoted as $\bm{X}^{T}=\{\bm{x}^{T}_{j}|_{j}\}$, $\bm{x}^{T}_{j}\in\mathbb{R}^{1024}$.

To obtain rich representations of the entities in the multi-modal knowledge graph, we encode the nodes in the multi-modal knowledge graph using a two-layer graph attention network (GAT) [25]. The node representations in the multi-modal knowledge graph, denoted as $\bm{U}^{G}=\bm{U}^{T}\cup\bm{U}^{I}$, is used as the input of the GAT. $\bm{U}^{T}=\{\bm{u}_{e_{i}^{*}}|e_{i}^{*}\in V^{T}\}$ and $\bm{U}^{G}=\{\bm{u}_{v_{j}^{\sharp}}|v_{j}^{\sharp}\in V^{I}\}$ denote the vector representations of the nodes in the text sub-graph $G^{T}$ and the image sub-graph $G^{I}$, respectively, where $\bm{u}_{e_{i}^{*}}$ and $\bm{u}_{v_{j}^{*}}$ denote the initial vector representations of named entity $e_{i}^{*}$ and visual object $v_{j}^{\sharp}$, respectively. The output of the last GAT layer is used as the input to the decoder, denoted as $\bm{X}^{G}=\{\bm{x}_{e_{i}^{*}}|e_{i}^{*}\in V^{T}\}\cup\{\bm{x}_{v_{j}^{\sharp}}|v_{j}^{\sharp}\in V^{I}\}$ where $\bm{x}_{e_{i}^{*}}$ and $\bm{x}_{v_{j}^{\sharp}}$ denote the vector representations of named entity $e_{i}^{*}$ and visual object $v_{j}^{\sharp}$ that are encoded by the graph attention network, respectively.

The decoder of the captioning model generates the tokens in the entity-aware captions sequentially and consists of $N$ identical Transformer layers. The initial input to the decoder is denoted as $\bm{X}_{0}=[\bm{X}^{T};\bm{X}^{I};\bm{X}^{G}]$, where the operator $[;]$ denotes matrix concatenation. At the $t$-th time step, the decoder predicts the probability of the current token $\bm{p}_{t}\in\mathbb{R}^{D}$ using the initial input and the embeddings of the previously generated subword units $\bm{M}_{t-1}=\{\bm{m}_{0},\bm{m}_{1},...,\bm{m}_{t-1}\}$, where $\bm{m}_{i}$ denotes the embedding of the $i$-th subword unit and $D$ denotes the vocabulary size.

During training of the captioning model, we fix the parameters of the pre-trained RoBERTa and the pre-trained CNN in the encoder. The parameters of GAT and the decoder are optimized using the following cross-entropy loss function:

$\displaystyle L_{p}=-\sum_{t=1}^{|Y|}\log p(w_{t}|w_{1},w_{2},...,w_{t-1}),$

(2)

where $|Y|$ denotes the length of the ground-truth caption, and $w_{i}$ denotes the $i$-th token in the caption $Y$.

Our experiments are conducted on two news image captioning datasets, GoodNews [2] and NYTimes800k [22]. The images, captions and news articles in these datasets are collected from New York Times, and each image is annotated with one ground-truth caption. In the GoodNews dataset, since the URLs to some images are not accessible, we employ the dataset split in [22], where the train, validation and test splits contain 445,259 images, 19,448 images and 24,461 images, respectively. The average lengths of the news articles and the ground-truth captions are 451 words and 18 words, respectively. The NYTimes800k dataset contains 763,217 images, 7,777 images and 21,977 images for training, validation and testing, respectively. Compared to the GoodNews dataset, NYTimes800k is more complex since the average article length and the average caption length of NYTimes800k are 974 words and 18 words, respectively.

We use standard image captioning metrics, including Bleu-4 [16], ROUGE-L [11] and CIDEr [24], to evaluate the similarity of the generated captions to the ground-truth captions. Since the goal of the model is to generate entity-aware captions, we also evaluate the named entities in the genrated sentences. Specifically, the Spacy toolkit [5] is used to recognize the named entities in both ground-truth sentences and generated sentences. We use the exact string matching to compare the generated and ground-truth named entities, and report the F1 score of the generated named entities.

To detect the objects in the image $I$, we use the YOLOv3 detector [18] and filter out the object bounding boxes with a confidence score of less than 0.3, while a maximum of 64 objects are kept for each image. To detect the faces, we use the MTCNN [31] network. Following the practice in [22], we keep no more than 4 face bounding boxes with the highest confidence scores for each image. To extract the features of the input images as well as the detected objects and faces, we use the pre-trained ResNet-152 [4] model.

The hidden dimension of the decoder in the captioning model is set to 1024. The sub-word vocabulary of the decoder is identical to the sub-word vocabulary of the pre-trained RoBERTa [12] model, which contains about 50,000 sub-word units. We set the maximum length of the article text $T$ and the caption $Y$ to 512 tokens and 50 tokens, respectively. We optimize the parameters of the model with the Adam optimizer [10]. We apply $L_{2}$ regularization to the parameters, and we use a weight decay of $10^{-5}$. The initial learning rate is set to $10^{-7}$. We increase the learning rate to $10^{-4}$ in the first 4,000 warmup steps, and decrease the learning rate linearly in the following training steps. We clip the gradients by the norm of $0.1$, and the batch size is set to 16.

We compare our method with several types of entity-aware captioning methods, i.e. conventional image captioning methods [29, 1], and recent methods with associated articles including template-based methods [17, 2] and end-to-end methods [2, 22, 6]. To evaluate the conventional models (SAT [29] and TopDown [1]), the original image captioning model only takes the image as input and generate a template caption, and the placeholders are filled by comparing the contextual information of the placeholders and the named entities in the news article, denoted as “CtxIns”.

Different form the conventional methods, the template-based methods (Ramisa et al. [17] and Biten et al. [2]) generate the template caption using both the image and the news article. Ramisa et al. [17] encode the article into a single vector. Biten et al. [2] attend to both the image regions and the sentences in the article, where two named entity selection strategies are employed, i.e. using sentence-level attention scores (denoted as “AttIns”) and using contextual information (denoted as “CtxIns”).

In contrast to the template-based methods that generate the captions in a two-stage manner, end-to-end methods [22, 6] directly generate entity-aware captions in one pass. Transform and Tell [22] is a Transformer-based [23] captioning model that uses byte-pair encoding to handle rare words and named entities. ICECAP [6] progressively attends to the sentences and the words in the article, and uses the contextual information to select the named entities.

Table 1 shows the comparison results between the aforementioned state-of-the-art methods and our method. We observe that our method achieves the best results in terms of most evaluation metrics on both datasets, especially Bleu-4 and CIDEr, which validates the superority of the proposed multi-modal knowledge graph on entity-aware image captioning. The entity F1 score of our method outperforms all existing methods on both datasets, which indicate that by modeling the association between the named entities and the visual cues in the image, our method selects named entities from the news articles more accurately.

To verify the effect of each component in our method, we conduct ablation studies on the GoodNews dataset. We evaluate the following variants of our model:

• •

  w/o graph: To evaluate the effect of the multi-modal knowledge graph, the entire multi-modal knowledge graph is removed, and the input to the captioning model only includes the region features of the image $\bm{X}^{I}$ and the token-level features of the news article $\bm{X}^{T}$.

• •

  image sub-graph: To evaluate the contribution of the text sub-graph $G^{T}$, the text sub-graph is removed and the input to the decoder includes $\bm{X}^{T}$, $\bm{X}^{I}$ and the features of the objects and faces in the image.

• •

  text sub-graph: To evaluate the contribution of the image sub-graph, we remove the objects and faces in $G^{I}$. The input to the decoder includes $\bm{X}^{T}$, $\bm{X}^{I}$ and the representations of the nodes in the text sub-graph $G^{T}$.

The results are shown in Table 2. From these results, we can make the following observations: First, the performance of “w/o graph” significantly degrades on all the metrics, which indicate that the multi-modal knowledge graph is beneficial to describing the events in the image as well as selecting named entities. Second, when using only the image sub-graph or the text sub-graph, the values of most metrics slightly increase, demonstrating that either sub-graphs contribute to improving the performance. Third, compared to the image sub-graph, using the text sub-graph achieves better results, indicating that the background knowledge in the news article is of greater importance than the objects and faces in the image. Finally, our full model performs best when the text sub-graph and the image sub-graph are both encoded, validating the effectiveness of learning the association between the named entities and the visual objects.

In addition to selecting named entities correctly, an entity aware captioning model is also required to describe the events depicted in the images properly. To evaluate the performance of event description, the influence of named entities should be eliminated. We perform named entity recognition using the SpaCy toolkit on both the ground-truth captions and generated captions, and replace the named entities with corresponding class labels. The standard image captioning evaluation metrics, including Bleu-4, ROUGE-L and CIDEr, are reported.

The event description evaluation results on GoodNews and NYTimes800k are shown in Table 3. From these results, we observe that both the image sub-graph and the text sub-graph contribute to event description. The model using the text sub-graph performs better than the model using image sub-graph, which indicates that the named entity relationships in the text sub-graph play a more important role in describing the events in the image.

We show some exemplars of generated entity-aware captions in Figure 4. As illustrated in the figure, our model selects multiple types of named entities more accurately, including concrete entities including persons, as well as abstract entities, such as places and times. For example, in Figure 4(b), our method correctly describes the person name in the image, while the model without multi-modal knowledge graph uses another person name in the article. The cases where multiple named entities are related to the image is also well handled by our model, thanks to the modeling of entity relationships using the multi-modal knowledge graphs. For instance, in Figure 4(d) where the image shows multiple persons, our model successfully identifies all the persons. It is also interesting to observe that even if the visual objects are not clear, our method still works well, which further validates the advantage of the entity relationship captured by the multi-modal graph using the external knowledge. Taking Figure 4(c) for example, though the Renault’s chairman Jean-Dominique Senard (the person on the left) is shadowed and is difficult to recognize, our model correctly describes both person names using the relationships involving him and the Nissan’s executive, Hitoro Saikawa (the person on the right).