Matching Visual Features to Hierarchical Semantic Topics for
Image Paragraph Captioning

Image captioning aims to describe images with natural language, in which a popular research line is generating a single sentence to depict an image (Vinyals et al. 2015; Xu et al. 2015a), denoted as image sentence captioning. However, as a single-sentence description is often too short to capture all detailed information, image paragraph captioning has been proposed to describe an image by generating a paragraph consisting of multiple sentences. Besides, other research directions in image captioning have also gradually attracted attention. For example, visual storytelling (Tang et al. 2019) aims to generate narrative creations from ordered photo sequences. In addition, optical character recognition (OCR) based image captioning (Wang et al. 2021) aims to automatically describe an image with a sentence according to all the visual entities (both visual objects and scene text) in the image. While these problems are also challenging and attractive, they are beyond the scope of this work that is focused on image paragraph captioning. Regions-Hierarchical (Krause et al. 2017) designs HRNN to produce a generic paragraph for an image. To generate diverse and semantically coherent paragraphs, Liang et al. (2017) extend the HRNN by proposing an adversarial framework between structured paragraph generator and multi-level paragraph discriminators. Considering the difficulties associated with training GANs and deficiency of explicit coherence model, Chatterjee and Schwing (2018) augment HRNN with coherence vectors and a formulation of VAE (Kingma and Welling, 2014). To encapsulate region-level features of an image into the topics, Wang et al. (2019) design a convolutional auto-encoding (CAE) module for topic modeling, where the extracted topics are further integrated into a two-level LSTM-based paragraph generator. Motivated by some models that utilize semantic topics to generate single-sentence captions (Fu et al. 2017; Zhu et al. 2018; Yu et al. 2018), Mao et al. (2018) pre-train the LDA (Blei et al. 2003) from the caption corpus of the training images at the first step, then train a topic classifier for semantic regularization and topic prediction based on the learned topics. In short, most of these existing paragraph captioning models either adopt deterministic networks ($e.g.$, high-level RNN or CAE) to construct a topic for each sentence within the whole paragraph or utilize a pre-trained LDA in a two-stage manner to learn the shallow semantic topics of images. Different from them, this work distills the multi-stochastic-layer semantic topics by capturing the correlations between the image and text at multiple levels of abstraction.

Probabilistic topic models (PTMs), such as latent Dirichlet allocation (LDA) (Blei et al., 2003; Griffiths and Steyvers, 2004) and Poisson factor analysis (PFA) (Zhou et al., 2012), often represent each document as a bag of words (BoW), capturing global semantic coherency into semantically meaningful topics. To explore the hierarchical semantic structures, PGBN (Zhou et al. 2016), a deep generalization of PFA that can also be viewed as a deep LDA (Cong et al. 2017), is proposed to extract interpretable hierarchical topics and capture the relationship between latent topics across multiple stochastic layers. Despite its effectiveness, PGBN, requiring the texts at training and testing stages, is not suitable for the image paragraph captioning task, where only images are given during the testing stage. To this end, we develop a variational encoder to match the visual features to the hierarchical topic information and generalize PGBN to build a novel visual-textual coupling model (VTCM), which can be jointly optimized with the paragraph generator. Although the idea of introducing a variational encoder is similar to neural topic models (NTMs) (Miao et al., 2016; Srivastava and Sutton, 2017; Burkhardt and Kramer, 2019; Zhang et al., 2018; Zhao et al., 2021), our VTCM captures the correlations between the image and text at multiple levels and is proposed to adapt to the image paragraph captioning task. To our knowledge, the works that connect deep topic modeling with visual features are still very limited. Different from Zhang et al. (2020) that propose to match visual features with a topic model, where the topics are fed into a GAN-based image generator, we focus on integrating the learned hierarchical topics from VTCM into the LMs to guide the paragraph generation, a task distinct from image generation.

Existing LMs are often built on either recurrent units, as used in RNNs (Cho et al., 2014; Hochreiter and Schmidhuber, 1997), or purely the attention mechanism based modules, as used in Transformer and its various generalizations (Vaswani et al., 2017; Radford et al., 2018). RNN-based LMs have been successfully used in image paragraph captioning systems (discussed above), while single-sentence Transformer-based image captioning systems have started to attract attention (Huang et al., 2019; Li et al., 2019; Cornia et al., 2020), not to mention the research on image paragraph captioning. Note that we can flexibly select the LM for our plug-and-play system since we pay more attention to assimilating the multi-layer semantic topic information into the paragraph generator. We consider both the LSTM-based and Transformer-based LMs to investigate the effectiveness of integrating the deep semantic topics.

There are two mainstream ways to learn topics from an image. One is to encode the visual image features into a global vector with a high-level RNN, which is explained as the topic vectors and used to guide a low-level RNN. Note that these topics are very different from the semantic topics, which are usually represented in the form of semantically related words in a topic model. Another way is first applying LDA on the training captions and then training a downstream topic predictor over image features, where the semantic topics are assimilated into the LM for sentence or paragraph generation. Moving beyond them, we design an end-to-end variational deep topic model to capture the correlations between image features and descriptive text by distilling the semantic topics, jointly trained with the LM. The basic idea follows the philosophy that the generation from topics to descriptive captions via topic decoder and the topics extraction over image visual features via variational encoder can enforce the mined multi-layer topics to be related to the visual features.

Inspired by Wang et al. (2019), who integrate the topics learned from CAE into a two-level LSTM-based paragraph generation framework with the attention mechanism in Anderson et al. (2018), we design a paragraph generator with a hierarchy constructed by a paragraph-level LSTM, a sentence-level LSTM, and an attention module, shown in Fig. 1(c). The paragraph-level LSTM first encodes the semantic regions based on all previous words into the paragraph state. Then the attention module selects semantic regions with the guidance of the current paragraph state and semantic topics of the image. Finally, the sentence-level LSTM incorporates the topics, attended image features, and current paragraph state to facilitate word generation.
Paragraph-level LSTM: To generate $w_{j,t}$ as the $t$-th word of the $j$-th sentence in a paragraph caption, we set $\boldsymbol{x}_{j,t}^{p}$ as the input vector of the paragraph-level LSTM. By concatenating the previous output $\boldsymbol{h}_{j,t-1}^{s,1}\in\mathbb{R}^{H}$ of the sentence-level LSTM at layer $1$, the image feature $\overline{\boldsymbol{v}}$, and previously generated word $w_{j,t-1}$, the $\boldsymbol{x}_{j,t}^{p}$ is formulated as $\boldsymbol{x}_{j,t}^{p}=\left[\boldsymbol{h}_{j,t-1}^{s,1},\overline{\boldsymbol{v}},{\bf W}_{e}w_{j,t-1}\right]$, where $[\cdot,\cdot]$ indicates concatenation, ${\bf W}_{e}\in\mathbb{R}^{E\times V}$ is a word embedding matrix, $V$ the vocabulary size of LM, $E$ the embedding size, and $H$ the size of hidden state unit. This input provides paragraph-level LSTM the maximum contextual information, capturing both visual semantics of the image and long-range inter-sentence dependency within a paragraph caption (Anderson et al., 2018). Then the hidden state of the paragraph-level LSTM is computed as

where $\boldsymbol{h}_{j,t}^{p}\in\mathbb{R}^{H}$ and $\boldsymbol{h}_{j,0}^{p}=\boldsymbol{h}_{j-1,T_{j-1}}^{p}$ are set to explore inter-sentence dependency.
Attention Module: Given the paragraph state $\boldsymbol{h}_{j,t}^{p}$ and the concatenation of multi-layer topic weight vectors $\boldsymbol{\theta}^{1:L}$, denoted as $[\boldsymbol{\theta}^{1:L}]\in\mathbb{R}_{+}^{\sum_{l=1}^{L}K_{l}}$, we build an attention module to select the most information-carrying regions of the visual features for predicting $w_{j,t}$, defined as

where $a_{j,t}^{m}$ is the $m$-th element of $\boldsymbol{a}_{j,t}\in\mathbb{R}^{M}$, and $\boldsymbol{w}_{att}\in\mathbb{R}^{1\times A}$, ${\bf W}_{va}\in\mathbb{R}^{A\times D}$, ${\bf W}_{ha}\in\mathbb{R}^{A\times H}$, ${\bf W}_{ta}\in\mathbb{R}^{A\times(\sum_{l=1}^{L}K_{l})}$ are the learned parameters and $\operatorname{softmax}$ is a function that turns the $M$-dimensional vector into a non-negative vector with $M$ elements summed to $1$. Defined in this way $\boldsymbol{p}_{j,t}$ is a probability vector over all regions in the image. The attended image feature is calculated with $\hat{\boldsymbol{v}}_{j,t}=\sum_{m=1}^{M}p_{j,t}^{m}\boldsymbol{v}_{m}$, providing a natural way to integrate the multi-layer semantic topics as auxiliary guidance when generating attention.
Sentence-level LSTM: The input vector $\boldsymbol{x}_{j,t}^{s}$ to the sentence-level LSTM at each time step consists of the output $\boldsymbol{h}_{j,t}^{p}$ of the paragraph-level LSTM, concatenated with the attended image feature $\hat{\boldsymbol{v}}_{j,t}$, stated as $\boldsymbol{x}_{j,t}^{s}=\left[\hat{\boldsymbol{v}}_{j,t},\boldsymbol{h}_{j,t}^{p}\right]$. Specifically, sentence-level LSTM in turn produces a sequence of hidden states $\{\boldsymbol{h}_{j,1}^{s,l},..,\boldsymbol{h}_{j,T_{j}}^{s,l}\}\in\mathbb{R}^{H}$ at layer $l$, one for each sentence in the paragraph, denoted as

where $\boldsymbol{u}_{j,t}^{l}$ is the coupling vector combining the topic weight vectors $\boldsymbol{\theta}^{l}$ and hidden output of the sentence-level LSTM $\boldsymbol{h}_{j,t}^{s,l}$ at each time step $t$. Following Guo et al. (2020), we realize $\boldsymbol{u}_{j,t}^{l}=g^{l}\left({\boldsymbol{h}}_{j,t}^{s,l},{\boldsymbol{\theta}}^{l}\right)$ with a gating unit similar to the gated recurrent unit (Cho et al. 2014), described in Appendix A.2. The probability over words in the dictionary can be predicted by taking a linear projection and a softmax operation over the concatenation of $\boldsymbol{u}_{j,t}^{l}$ across all layers. This method enhances the representation power and, with skip connections from all hidden layers to the output (Graves et al., 2013), mitigates the vanishing gradient problem. We denote the parameters of the LSTM-based LM as $\boldsymbol{\Omega}_{\text{LSTM}}$.

To demonstrate the proposed plug-and-play system, we also explore how to integrate the multi-layer topic information into existing Transformer-based LMs, due to their representation power and computational efficiency coming from pure attention mechanisms. Typically, attention operates on a set of queries $\boldsymbol{Q}$, keys $\boldsymbol{K}$, and values $\boldsymbol{V}$, defined as $\text{ Attention }(\boldsymbol{Q},\boldsymbol{K},\boldsymbol{V})\!=\!\operatorname{softmax}\left({Q\boldsymbol{K}^{T}}/{\sqrt{d}}\right)\boldsymbol{V}$, where $\boldsymbol{Q}$ is a matrix of $n_{q}$ query vectors, both $\boldsymbol{K}$ and $\boldsymbol{V}$ contain $n_{k}$ keys and values, all with the same dimensionality, and $d$ is a scaling factor. Cornia et al. (2020) design a novel Transformer-based architecture to improve the image encoder and language decoder, and prove its effectiveness on sentence-level image captions. However, they neither consider the paragraph-level image captioning task nor the semantic topics underlying the image. On the basis of this Transformer-based architecture, we here devise a semantic topic-guided Transformer model, which is conceptually divided into an encoder and a decoder module, shown in Fig. 8. Following Cornia et al. (2020), the encoder processes region-level image features and devises the relationships between them. The decoder not only reads from the output of each encoding layer like that of Cornia et al. (2020) but also assimilates the hierarchical topic information to generate the paragraph caption word by word.
Memory-augmented Encoder: Denoting the aforementioned set of features $\{\boldsymbol{v}_{1},\cdots,\boldsymbol{v}_{M}\}$ as ${\bf X}$ for clarity, we adopt the memory-augmented attention operator to encode image regions and their relationships, defined as

where ${W_{q},W_{k},W_{v}}$ are matrices of learnable weights, a usual practice in the original Transformer, and $\boldsymbol{M}_{k}$ and $\boldsymbol{M}_{v}$ are additional keys and values implemented as plain learnable memory matrices. Following the implementation of Vaswani et al. (2017), the memory-augmented attention can be applied in a multi-head fashion, whose output can be fed into a feed-forward layer, denoted as $\mathcal{F}(\cdot)$. Both the attention and feed-forward layers are encapsulated within a residual connection and a layer norm operation, denoted as $\text{AddNorm}(\cdot)$. For the encoder with $L$ layers, its $l$-th encoding layer is therefore defined as

where $\tilde{\boldsymbol{X}}^{l}\in R^{d}$ and $\tilde{\boldsymbol{X}}^{0}=\boldsymbol{X}$. A stack of $L$ encoding layers will produce a multilevel output $\tilde{\mathcal{X}}=\left(\tilde{\boldsymbol{X}}^{1},\ldots,\tilde{\boldsymbol{X}}^{L}\right)$.
Topic-guided Meshed Decoder: Given the region encodings $\tilde{\mathcal{X}}$ and topic information $\boldsymbol{\theta}^{1:L}$, our decoder aims to generate the paragraph caption, denoted as $\boldsymbol{Y}=\{y_{1},...,y_{I}\}$ consisting of $I$ words for clarity. Inspired by Cornia et al. (2020), we construct the topic-guided Meshed Attention operator to connect $\boldsymbol{Y}$ to all elements in $\tilde{\mathcal{X}}$ and $\boldsymbol{\theta}^{1:L}$ hierarchically through gated cross-attentions, formulated as

where we combine the topic information $\tilde{\boldsymbol{\theta}}^{l}$ and the hidden output of the memory-augmented encoder $\tilde{\boldsymbol{X}}^{l}$ at each layer $l$ although other choices are also available, $\tilde{\boldsymbol{\theta}^{l}}\in R^{d}$ is projected from $\boldsymbol{\theta}^{l}\in R^{K_{l}}$ into the encoder embedding space, and ${\mathcal{C}(\cdot,\cdot)}$ stands for the cross-attention, computed using queries from the decoder and keys and values from the encoder and topic information:

By computing $\boldsymbol{\alpha}_{l}=\operatorname{sigmoid}\left(W_{l}\left[\boldsymbol{Y},\mathcal{C}\left(\tilde{\boldsymbol{X}}^{l}\!+\!\tilde{\boldsymbol{\theta}^{l}},\boldsymbol{Y}\right)\right]\right)$, we can measure the relevance between cross-attention results, where $W_{l}$ is the learned weight matrix. Similar to the encoding layer, the final structure of each decoding layer is written as

where $\mathcal{S}_{\operatorname{m}}$ is a masked self-attention used in the original Transformer (Vaswani et al. 2017), due to the prediction of a word should only depend on previously predicted words, and ${\tilde{\boldsymbol{Y}}^{0}=\boldsymbol{Y}}$. After taking a linear projection and a softmax operation over $\tilde{\boldsymbol{Y}}^{L}$, our decoder finally predicts the probability over words in the vocabulary. Similar to the LSTM-based LM, Transformer-based LM is also guided by the multi-layer semantic topics and attended image features when generating the caption, whose parameters are represented as $\boldsymbol{\Omega}_{\text{Trans}}$.

Under the deep topic model described in Section 3.1 and LSTM-based LM in Section 3.2, the joint likelihood of the target ground truth paragraph $P$ of Img and its corresponding BoW count vector $\boldsymbol{d}$ is defined as

which is similar as the likelihood of the topic-guided Transformer-based captioning system, described in Appendix A.3. As discussed in Section 3.1, we introduce a variational topic encoder to learn the multi-layer topic weight vectors $\boldsymbol{\theta}^{1:L}$ in (2) with the image features as the input. Thus, a lower bound of the log of (3.4) can be constructed as

which unites the first two terms primarily responsible for training the topic model component, and the last term for training the LM component. The parameters $\boldsymbol{\Omega}_{\text{TM}}$ of the variational topic encoder and the parameters $\boldsymbol{\Omega}_{\text{LSTM}}$ of LSTM-based LM can be jointly updated by maximizing $L_{\text{all}}$. Besides, the global parameters $\boldsymbol{\Phi}^{1:L}$ of the topic decoder can be sampled with TLASGR-MCMC in Cong et al. (2017), described in Appendix A.4. The training strategy is outlined in Algorithm 1.

To sum up, as shown in Fig. 1, the proposed framework couples the topic model (VTCM) with a visual extractor, which takes the visual features of the given image as input and maps the hierarchical topic weight vectors. The learned topic vectors in different layers are then used to reconstruct the BoW vector of the given image paragraph caption and as the additional features for the LSTM (or Transformer)-based LM to generate the paragraph. Moreover, our proposed model introduces randomness into the topic weight vector, which captures the uncertainty about what is depicted in an image and hence encourages the diversity of generation.

We conduct experiments on the public Stanford image-paragraph dataset (Krause et al. 2017), where 14,575 image-paragraph pairs are used for training, 2,487 for validation, and 2,489 for testing. Following the standard evaluation protocol, we use the full set of captioning metrics: METEOR (Denkowski and Lavie 2014), CIDEr (Vedantam et al. 2015), and BLEU (Papineni et al. 2002). Different from the BLEU scores primarily measuring the $n$-gram precision, METEOR and CIDEr are known to provide more robust evaluations of language generation algorithms (Vedantam et al. 2015). In our experiments, the hyper-parameters and model checkpoints are chosen by optimizing the performance based on the average of METEOR and CIDEr scores on the validation set.
Following the publicly available implementation of Anderson et al. (2018) and Wang et al. (2019), we use Faster R-CNN (Ren et al. 2015) with VGG16 network (Simonyan and Zisserman 2015) as the visual extractor, which is pre-trained over Visual Genome (Krishna et al. 2017). The top $M=50$ detected regions are selected to represent image features. The size of each image feature vector is $4096$, which is embedded into the $1024$-dimensional vector before being fed into our topic model. For our LMs, we tokenize words and sentences using Stanford CoreNLP (Manning et al. 2014), lowercase all words, and filter out words that occur less than $1$ time. We set the maximum number of sentences in each paragraph as $6$ and the maximum length of each sentence as $30$ (padded where necessary) for VTCM-LSTM. For our topic model, all the words from the training dataset, excluding the stopwords and the top $0.1\%$ most frequent words, are used to obtain a BoW caption for the corresponding image. The hidden sizes of paragraph-LSTM, sentence-LSTM, and attention module are all set to $512$. For our Transformer-based LM, we set the dimensionality $d$ of each layer as $512$, the number of heads as $8$, and the number of memory vectors as $40$. Both our Transformer-based and LSTM-based LMs are a three-layer model, same with the topic model with the topic number of $[K_{1};K_{2};K_{3}]=[80;50;30]$. Besides, we directly set hyper-parameters in VTCM as $\{\eta^{l}=0.1,\tau^{l}=1,\boldsymbol{r}=1\}$. We use the Adam optimizer (Kingma and Ba 2015) with a learning rate of $5e-4$ for VTCM-LSTM and $1$ for VTCM-Transformer. The gradients of both VTCM-LSTM and VTCM-Transformer are clipped if the norm of the parameter vector exceeds $0.1$. The dropout rate is set to $0.5$ for VTCM-LSTM and 0.9 for VTCM-Transformer, and adopted in both the input and output layers to avoid overfitting. During inference, we adopt the penalty on trigram repetition proposed by Melas-Kyriazi et al. (2018) and set the penalty hyperparameter as $2$. We also provide additional experimental results on the radiology report generation task in the Appendix A.5.

For a fair comparison, we consider the following baselines: 1) Image-Flat (Karpathy and Fei-Fei 2015), directly decoding a paragraph word-by-word via a single LSTM; 2) Flat-repetition-penalty (Melas-Kyriazi et al. 2018), training the non-hierarchical LSTM-based LM with an integrated penalty on trigram repetition to improve the diversity in image paragraph captioning; 3) Regions-Hierarchical (Krause et al. 2017), using a hierarchical LSTM to generate a paragraph, sentence by sentence; 4) RTT-GAN (Liang et al. 2017), training the Regions-Hierarchical in a GAN setting, coupled with an attention mechanism; 5) TOMS (Mao et al. 2018), generating multi-sentences under the topic guidence, which trains a downstream topic classifier to predict the topics mined by the LDA; 6) Diverse-VAE (Chatterjee and Schwing 2018), leveraging coherence vectors and global topic vectors to generate paragraph, under a VAE framework; 7) IMAP (Xu et al. 2020), proposing an interactive key-value memory-augmented attention into the hierarchical LSTM; 8) LSTM-ATT, which refers the outputs of paragraph-level LSTM as the topic vectors and adopts the attention mechanism in Anderson et al. (2018), as a degraded version of the proposed VTCM-LSTM; 9) $\mathcal{M}^{2}$-Transformer (Cornia et al. 2020), a novel Transformer-based architecture for single-sentence image captioning and a degraded version of VTCM-Transformer; 10) CAE-LSTM (Wang et al. 2019), which adopts the CAE to extract topics and integrates them into the two-level LSTM-based paragraph generator; 11) Splitting to Tree Decoder (S2TD) (Shi et al. 2021), which models the paragraph decoding process as a top-down binary tree expansion and consists of a split module, a score module, and a word-level LSTM; 12) Retrieval-enhanced adversarial training with dynamic memory-augmented attention for image paragraph captioning (RAMP) (Xu et al. 2021), which makes full use of the R-best retrieved candidate captions and adopts the hierarchical LSTM as the paragraph generator.

Main Results: The results of different models on the Stanford dataset are shown in Table 1, where we only report the results of different models trained with cross-entropy rather than self-critical sequence training to eliminate the influence of different training strategies. As it can be observed, our proposed VTCM-LSTM surpasses all the other LSTM-based captioning systems in terms of BLEU-4, BLEU-3, and CIDEr, while being competitive on BLEU-1, BLEU-2 and METEOR with the best performer. Moreover, on all metrics, our proposed VTCM-LSTM and VTCM-Transformer improve their corresponding baselines, $i.e.$, LSTM-ATT and $\mathcal{M}^{2}$-Transformer, respectively. These results demonstrate the effectiveness of integrating the semantic topics mined from VTCM into language generation in terms of topical semantics and descriptive completeness. Moreover, VTCM-Transformer leads to a performance boost over VTCM-LSTM on almost all the metrics, indicating the advantage of the memory-augmented operator and meshed cross-attention operator with a Transformer-like layer. We also replace the $\mathcal{M}^{2}$-Transformer with the original Transformer pretrained on a diverse set of unlabeled text, which however produces poor performance, suggesting the importance of designing specialized architectures for multi-model image captioning. Of particular note is the large improvement under both VTCM-LSTM and VTCM-Transformer on CIDEr, which is proposed specifically for image descriptions evaluation and measures the $n$-gram accuracy by term-frequency inverse-document-frequency (TF-IDF). Interestingly, by bridging the visual features to the textual descriptions, our proposed VTCM is suited for extracting paragraph-level word concurrence patterns into latent topics, which capture the main aspects of the scene and image descriptions. The assimilation of topic information into language models thus leads to a large improvement in CIDEr, correlated well with human judgment. However, it is often not the case in other image captioning systems unless the CIDEr score is treated as the reward and directly optimized with policy-gradient based reinforcement learning techniques to finetune the model (Wang et al. 2019; Cornia et al. 2020; Melas-Kyriazi et al. 2018; Xu et al. 2020).

Notably, the CAE-LSTM of Wang et al. (2019), the S2TD of Shi et al. (2021), and the RAMP of Xu et al. (2021) additionally adopt self-critical (SC) training after the pre-training with cross-entropy (CE). For a fair comparison, we also treat the CIDEr as the reward and introduce the self-critical into the pre-trained VTCM-LSTM, where the results are reported in Table 2. Our proposed model outperforms all the baselines on METEOR and CIDEr and achieves a comparable performance with S2TD and RAMP on BLEU-4. It indicates the effectiveness of assimilating the hierarchical semantic topic from VTCM into the paragraph generator.

Generated Captions: To qualitatively show the effectiveness of our proposed methods, we show descriptions of different images generated by different methods in Fig. 3. As we can see, all of these models can produce paragraphs related to the given images, while our proposed VTCM-LSTM and VTCM-Transformer can generate more coherent and accurate paragraphs by learning to distill the semantic topics from an image via the VTCM module to guide paragraph generation. Therefore, instead of only attending to some visually salient image regions, the generated descriptions of our models are also highly related to the given images in terms of their semantic meanings but not necessarily the words same as the original caption. Taking the first row as the example, the proposed VTCM-Transformer can generate coherent and meaningful paragraphs to describe the image, while capturing meta-concepts like “steam train” and “driving on” based on the scenes including “train” and “smoke”. Notably, these concepts are even not described in the ground truth but are very relative to the whole image. However, without the high-level semantic information, these baselines tend to only describe all the salient visual objects in the image, ignoring the “main plot” underlying the images, such as the “the train is in motion” in the first row and “food” in the last row in Fig. 3. These observations suggest that the proposed VTCM has successfully captured hierarchical semantic topics by matching visual features to descriptive texts with a similar VAE structure, and our proposed ways of assimilating the topic information into LSTM or Transformer can successfully guide the paragraph generation.

The Attention Mechanism in VTCM-LSTM: To evaluate the effectiveness of the attention mechanism in VTCM-LSTM on image captioning, in Fig. 7, we visualize the attended image regions with the biggest attention weight for different words. As we can see, our proposed VTCM-LSTM can reason where the model is focusing on at different time steps. Here we take Image 1 as an example. When predicting “person,” the attention module precisely chooses the bounding box covering the main part of the body. While predicting the word “snowboard,” our model decides to attend to the surrounding area about the snowboard. It proves that our proposed VTCM-LSTM can capture the alignment between the attended area and the predicted word, which reflects the human intuition during object description.