Efficient Modeling of Future Context for Image Captioning

The visual encoder aims to learn the high-level visual representations of the given image, which includes $L$ same network layers. Each layer consists of two sub-layers: a self-attention sub-layer and a position-wise feed-forward network sub-layer. The input of the network layer is the hidden states of the previous layer, on which the multi-head scaled dot-product attention computation is performed. Assuming that $h^{l}_{e}$ presents the hidden states of the $l$-th encoder layer, the visual encoder layer can be computed as:

Layer normalization with residual connection is added after both two sub-layers. Note that $h^{0}_{e}$ is initialized as the patch embedding of the extracted image region features, and the hidden states of the $L$-th layer $h^{L}_{e}$ are served as input to the language decoder.

The decoder of the AIC and NAIC models are introduced separately. For the autoregressive decoder, which usually consists of three sub-layers: a masked self-attention sub-layer, a cross-attention sub-layer, and an FFN sub-layer. In particular, to maintain the autoregressive generated property at each time step, the masked self-attention sub-layer performs self-attention with a causal attention mask to prevent the decoder from seeing subsequent words. To generate the hidden states $h^{l}_{d}$ of the $l$-th decoder layer, the autoregressive decoder can be formulated as:

Layer normalization with residual connection is also added after each sub-layers. Finally, with the given image $x$, the generated sentence $w_{<t}$ and the learned top-layer hidden states $h_{d,t}$, the decoder models the probability distribution as:

where $W$ denotes the learnable parameter.

For the non-autoregressive decoder, which aims to predict a set of masked target words $S_{m}$ given an image $x$ and a set of observable target words $S_{o}$. The NAIC decoder also contains the same $L$ identical layers, each of which also includes a self-attention sub-layer, a cross-attention sub-layer, and a feedforward sublayer. Unlike the masked self-attention sub-layer of the AIC decoder, the attention mask is removed in the NAIC decoder. Finally, with the learned top-layer hidden states $\tilde{h}_{d}$ of the NAIC decoder and partially observed sentence $S_{o}$, the predicted probability distribution for every masked word $w_{t}\in S_{m}$ can be calculated as:

where $\tilde{W}$ is the learnable parameter. Note that since the decoder of the NAIC model takes $S_{o}$ rather than $w_{<t}$ as input, which includes both history and future words with respect to every masked target word, it should embody the global contextual information.

Experimentally, we adopt the basic configuration of the Transformer-based image captioning model without the mesh-memory module, which is publicly available at GitHub ¹¹1https://github.com/aimagelab/meshed-memory-transformer. To be specific, the model is comprised of 6 standard transformer layers of visual encoder and language decoder. Moreover, the regional features of images extracted from faster r-cnn (Ren et al., 2015) on the backbone of ResNet (He et al., 2016) are utilized to retrain the image captioning model with the current configuration (Vaswani et al., 2017). For the training process, the image captioning model is first trained with cross-entropy loss and then fine-tuned with sentence-level self-critical reward (Rennie et al., 2017) follow the default training settings with Adam (Kingma and Ba, 2014) optimizer on the MS COCO (Chen et al., 2015) dataset of Karpathy training split.

After obtaining a fully-trained image captioning model, we record the predicted probability of the ground-truth words given the correct context including image and the previous subsentence in the MS COCO training set. In order to characterize the results, we plot the proportion of total words in different predicted probability chunks in Figure 2. Meantime, we also plot the average predicted probability of the corresponding ground-truth words for different caption lengths in Figure 3. Here we normalized the caption to eliminate the influence of absolute sentence length.

According to the estimation results of Figure 2, we can find that even provided with the totally correct context, there is an obvious portion of ground-truth words that the conventional image captioning model predicts with relatively low probabilities. For instance, the model predicts 25.67% ground-truth words with probabilities chunk between 0.0-0.1. The reasonable cause of this phenomenon is that image captioning model cannot confidently predict these ground-truth words according to only the local context of image and history words.

To further prove where the low confidence words locate, we calculate the average predicted probability for various locations in the total length. We can see that with the relative length increase, i.e., the generation position moves from left to right, and the average probability of ground-truth words increases gradually. We attribute that with the generated sentence length increase, the determined context increase, and future context becomes less, which results in the strengthening of model confidence. According to the experimental results, it is natural to consider improving the image captioning model on these correct while unconfident words with effective future information to assist the current decision.

We evaluate the proposed method on the MS COCO (Chen et al., 2015), which is a standard estimation benchmark for image captioning tasks. To be consistent with previous work (Huang et al., 2019; Cornia et al., 2020), we adopted the Karpathy split (Karpathy and Fei-Fei, 2015) that contains 113,287 training images and 5,000 images for validation and test splits, respectively. Each image corresponds with 5 different captions. We omit words that occur less than 5 times and the vocabulary size is 10,369 words. Image features are extracted with CLIP (Anderson et al., 2018) for 512-dim vectors.

Following the common paradigm, we utilize five metrics to comprehensively estimate the captioning performance: BLEU-$N$ (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), ROUGE (Lin, 2004), CIDEr (Vedantam et al., 2015), and SPICE (Anderson et al., 2016). We denote as B-$N$, M, R, C, and S for simplicity.

Our implementation is based on Pytorch and repository (Cornia et al., 2020) to build the model under Transformer-base configuration with the memory module, where AIC and NAIC hold the identical architecture. Concretely, both the network is comprised of 6 visual encoder and 6 language decoder layers, each with 512 as hidden size, the FFN sublayers of 2,048 dimensions, and 8 heads in multi-head attentions. We set the dropout rate to 0.1. The neurons are sampled from the top-layer decoder with a unifying distribution. For parameter updating, we employ the Adam optimizer (Kingma and Ba, 2014) with the default setting. As for learning rate schedule, we adopt the same strategy as (Vaswani et al., 2017; Cornia et al., 2020) and set warm-up steps to 4,000. In the first stage, we train all models by sharing their encoders for 300k steps. In the second stage, we separate their encoders and fix the parameter of the NAIC model. Then, the AIC model is solely optimized with the fixed NAIC by additional 200k steps.

We compare the results of our FutureCap model with those of several recent image captioning models trained without large-scale vision-and-language pre-training on the offline MS COCO dataset. The evaluation results are listed in Table 1. First, we can see that FutureCap surpasses the original memory-incorporated Transformer by +0.6 BLEU-4 and 7.1 CIDEr scores, respectively, verifying that modeling the future information brings a significant performance improvement. Next, it is encouraging that our proposed framework outperforms the most recent competitive models. As it can be observed, our proposal reaches 136.3 CIDEr points, beating almost all the compared approaches. It is encouraging that our strategy can be combined with other advance improved strategies as well as untouched the internal structures. What’s more, as most of the previous literature has boosted caption quality by increasing the model capacity, which leads to an extra burden for application devices, our approach represents an outliner in this trend and demonstrates that state-of-the-art CIDEr levels can be obtained even with a very lightweight efficient model.

We also report the performance of our method on the online MS COCO test server. In this case, we employ an ensemble of four models trained with the same configuration of NAIC decoder assistance, for which ground-truth annotations are not publicly available. Comparison results with the top-performing approaches of the leaderboard are reported in Table 2. As it can be seen, our method surpasses the current state-of-the-art model on all metrics, achieving an advancement of 1.3 CIDEr points with respect to the best performer.

To better understand the influence of each design in our FutureCap model, we conduct ablation studies on the offline MS COCO dataset. Table 3 reports the evaluation results on the testing set. We first validate the necessity of shared visual encoder supervision by training the AIC and NAIC model using the separate visual encoder, respectively, denoted as “w/o. VES”. The automatic metrics of final performance decrease by a 1.1 CIDEr score. This shows that the visual encoder of the AIC model benefits a lot from the global supervision information through joint training with the NAIC decoder. As for “w/o. CDC”, which means not performing causal dynamics calibration on any target words at the fine-tuning stage, its performance also decreases, e.g. 0.4 for the BLEU-4 score and 1.5 for the CIDEr score. Moreover, to illustrate the superiority of CDC, we replaced it with the conventional knowledge distillation, i.e., remove the $L_{IA}$ in Equation 14, the results show poor performance. These demonstrate the effect of each part in our design as well as incorporating specific future context into the image captioning model on its unconfident words.

There exist important hyper-parameters in the FutureCap framework that we need to tune on the validation set to achieve a good performance, i.e., the balancing factor $\lambda$ in Equation 8 and the confidence threshold $\epsilon$ for determining mask words set $S_{m}$. To balance the training of the AIC model and NAIC model at the pre-training stage, we try to select the optimal $\lambda$ that can bring steady improvements to the AIC model. Specifically, we gradually vary $\lambda$ from 0.5 to 1.0 with an increment of 0.1 and evaluate the performance on the validation set, the evaluation results are as shown in Figure 5. We can see that the final image captioning model achieves its peak when $\lambda=0.7$. Hence, $\lambda$ is set to 0.7 by default. Given the selected $\lambda$, at the fine-tuning calibration stage, we also analyze the impact of $\epsilon$ on the validation set. Practically, we change the value of $\epsilon$ from 0.0 to 0.3 with an interval of 0.05. As shown in Figure 6, the AIC model performs the best when the $\epsilon$ comes to 0.2. Therefore, we set $\epsilon$ = 0.2 as the confidence threshold for the causal dynamics calibration stage.

In our future context modeling framework, for each generated caption pair, we adopt causal dynamics calibration to transfer the knowledge of the NAIC model into the AIC model only on the masked word set $S_{m}$. The set is determined by masking words whose AIC-predicted probabilities of the corresponding ground truths are lower than a pre-set threshold $\epsilon$. It is natural to question if exists other masked word selection patterns and how they perform. Therefore, we further investigate the following four variant masking methods:

• •

  Random: For the given sentence $S$, randomly select $k$ words to be masked and input to the teacher NAIC model to conduct causal dynamics calibration accordingly.

• •

  Highest: As a contrast, we mask the generated words from the AIC model whose predicted probabilities of the ground truth words are higher than the preset threshold $\epsilon$.

• •

  Wrong: Since the ground-truth labels are given, we try to mask the words where the highest probability predictions of the AIC model are different from the corresponding labels.

• •

  OnlyOne: In this variant, to illustrate the necessity of selectively distilling knowledge on a portion rather than all of the target words, we generate NAIC-predicted probability distributions for all target words. As an extreme case, we iteratively and only mask one word at once with the given image and residual sentence as input to the NAIC model.

The evaluation results for different masked word selection strategies are presented in Table 4. We can observe that: 1) For “Random” and “Highest” masking strategies, both variants are inferior to our threshold-based causal dynamics calibration method. In particular, the results of “Highest” indicate that conducting dynamics calibration on the confident words is less effective, resulting in a decrease of 0.6 CIDEr score. Meantime, the heuristic selection of masked words is necessary rather than randomness; 2) The result of “Wrong” is lower than our approach. It may be due to the distribution difference between low confidence and incorrect predicted words. 3) “OnlyOne” represents one approach to generating NAIC-predicted probability distributions for all target words iteratively. It is also reasonable for “OnlyOne” to obtain a worse performance since some words can be easy to generate conditioned on local context and over-calibration hold some side effects. All these results demonstrate that it is crucial for the AIC model to exploit the global context on its unconfident words. At the same time, a more advanced and learnable selection strategy may contribute to better captioning performance.

As the prior experiments show one drawback of conventional image captioning lies in the low confidence to correct words, here we also investigate the change of image captioning model confidence with respect to ground-truth words on the MS COCO training set with future context modeling. We list the percentage of words within each interval, in terms of AIC-predicted probability in Table 5. As the probability higher than 0.5 must be the maximum across the vocabulary, we chunk 0.5-1.0 as a high-confidence interval while the others are subdivided into low-confidence intervals. According to the evaluation results, it is obvious that the number of words in low-confidence intervals drops with FutureCap. For instance, the number of words located in [0.1, 0.2) becomes 0.35% fewer. It indicates that our FutureCap model becomes more confident about the ground-truth words given the accurate context.