ECOL-R: Encouraging Copying in Novel Object Captioning with Reinforcement Learning

Following the setup in Agrawal et al. (2019), we use two object detectors: the Visual Genome object detector from Anderson et al. (2018b), producing image objects and regions $G$ (represented by embedding vectors $[{\bm{x}}^{g}_{1},\dots,{\bm{x}}^{g}_{k^{g}}]$) with detailed visual features; and a task specific object detector, producing image objects $F$ (represented by $[{\bm{x}}^{f}_{1},\dots,{\bm{x}}^{f}_{k^{f}}]$) and their corresponding labels $L_{f}=[l_{1},\dots,l_{k^{f}}]$ used as copy candidates during caption generation. We will introduce object representations ${\bm{x}}_{i}$ below and define them in Eq. 1.

Following Anderson et al. (2018b); Lu et al. (2018a), we represent both sets of objects with Region-Of-Interest (ROI, ${\bm{r}}_{i}\in{\mathbb{R}}^{2048}$) vectors from the Visual Genome object detector and object positional features (${\bm{p}}_{i}\in{\mathbb{R}}^{8}$), including bounding box coordinates and size, and an object label confidence score. In addition, to transfer knowledge from the seen objects to the novel ones, we propose Abstract Labels for the task specific objects, described below.

The task specific object detectors we use provide taxonomies of object classes, and every detected object is assigned a label from that taxonomy. More general object classes conceptually include all the labels lower in the taxonomy. ¹¹1If the task specific object detector does not provide such a taxonomy, a suitable taxonomy could be obtained from sources such as Wordnet. This provides us with a mechanism for associating class labels not present in the training data with those that do occur in the training data by mapping them to a common ancestor in the hierarchy. Inspired by Ciaramita and Johnson (2003), we define Abstract Labels to be a fixed set of ancestor class labels that spans the entire taxonomy (see Figure 3). Using the abstract labels to drive copy decisions allows the usage of known object types to inform the word generation of novel objects. Each object from the task specific detector is associated with its nearest abstract label ancestor. We choose the set of abstract labels such that the objects in the training data are evenly distributed across the set of abstract labels. We represent abstract labels with trainable embeddings ${\bm{e}}_{i}\in\mathbb{R}^{d}$, where $d$ is the hidden size of our base model. We use the Open Images V4 class hierarchy for the nocaps benchmark and a merged 8 coco super-categories hierarchy Lin et al. (2014) for the held-out COCO benchmark.

The final representation for each object ${\bm{x}}_{i}$ is:

$\displaystyle{\bm{x}}_{i}=\mathit{LN}({\bm{W}}_{r}{\bm{r}}_{i}+{\bm{e}}_{i})+\mathit{LN}({\bm{W}}_{p}{\bm{p}}_{i})$

(1)

where $\mathit{LN}$ is layer normalization and ${\bm{W}}_{r}\in\mathbb{R}^{d\times 2048}$, ${\bm{W}}_{p}\in\mathbb{R}^{d\times 8}$ are trainable projections. The two sets of object representations are concatenated as $X=F\odot G$ where $\odot$ represents concatenation.

We use a transformer model Vaswani et al. (2017) with an $N_{enc}$-layer encoder and an $N_{dec}$-layer decoder ($N_{enc}$ = $N_{dec}$ = 3 in our experiments). We denote the encoder output $\mathbf{H}=\mathit{Encoder}(X)$. The decoder uses frozen word and positional embeddings $\mathit{WE}$ and $\mathit{PE}$ from GPT2 Radford et al. (2019) which are helpful in producing better captions describing novel objects. In step $t$:

	$\displaystyle{\bm{w}}_{1:t-1}$	$\displaystyle=\mathit{WE}(y_{1:t-1})+\mathit{PE}(y_{1:t-1})$		(2)
	$\displaystyle{\bm{h}}_{t}$	$\displaystyle=\mathit{Decoder}(\mathbf{H},{\bm{w}}_{1:t-1})$		(3)

where $y_{1:t-1}$ is the generation history and ${\bm{h}}_{t}\in\mathbb{R}^{d}$.

The ECOL model either generates words from the vocabulary or copies from task specific objects. We deploy a copy mechanism similar to the dynamic pointer network in Hu et al. (2020). Given the decoder output ${\bm{h}}_{t}$, we first calculate a raw score for each vocabulary word:

$\displaystyle\mathit{VScore}({\bm{h}}_{t})$

$\displaystyle={\bm{W}}_{e}{\bm{h}}_{t}$

(4)

where ${\bm{W}}_{e}\in\mathbb{R}^{|V|\times d}$, $|V|$ is the GPT2 vocabulary size. We then calculate raw additive attention scores over the encoder output of task specific image objects (i.e., $\mathbf{H}_{1:k^{f}}$):

(5)

where $i\in[1,k^{f}]$ and ${\bm{W}}_{f}\in R^{d\times d}$, ${\bm{W}}_{h}\in R^{d\times d}$ and ${\bm{w}}_{c}\in R^{d}$. Finally, we concatenate the raw scores from $\mathit{VScore}$ and $\mathit{OScore}$ and jointly softmax:

(6)

where $\odot$ represents concatenation. ${\bm{v}}_{t}$ provides probabilities for GPT2 vocabulary words and ${\bm{c}}_{t}$ provides probabilities for copying task specific object labels.

Object labels can appear in inflected forms in captions. For example, in Figure 1, after selecting the object hamburger, the ECOL model should generate “hamburgers” after “Two”. We propose a morphological selector (M Selector) to refine the copy probability of each task specific image object label $l_{i}$ (i.e., ${\bm{c}}_{t,i}$) into the probabilities of generating all possible morphological forms $y_{t}^{l_{i}}$ (i.e., $P(y_{t}^{l_{i}}|l_{i})$). Specifically, we use ${\bm{h}}_{t}$ to choose an inflected form from its possible inflected forms (e.g., Singular or Plural in English):

$\displaystyle P(y_{t}^{l_{i}}|l_{i})$

$\displaystyle=\mathrm{softmax}({\bm{W}}_{l_{i}}{\bm{h}}_{t})$

(7)

Here ${\bm{W}}_{l_{i}}\in\mathbb{R}^{s_{i}\times d}$ where $s_{i}$ is the number of inflected forms of label $l_{i}$ (in most cases $2$ for English, singular and plural). Finally, the ECOL model concatenates the above refined probabilities as follows:

	$\displaystyle P(y^{v}_{t})$	$\displaystyle={\bm{v}}_{t}$		(8)
	$\displaystyle P(y^{l_{i}}_{t})$	$\displaystyle={\bm{c}}_{t,i}\cdot P(y_{t}^{l_{i}}|l_{i})$		(9)
	$\displaystyle P(y_{t})$	$\displaystyle=P(y^{v}_{t})\odot P(y^{l_{1}}_{t})\odot\dots\odot P(y^{l_{k^{f}}}_{t})$		(10)

where $\odot$ represents concatenation. Some novel object labels are included in the GPT2 vocabulary. However, these words are not present in the training captions and thus the model always assigns them very low probabilities in $P(y^{v}_{t})$. The only way novel object labels can appear in captions is through copy operations.

In this paper, we focus on the Novel Object Captioning task. However, in general, our copy mechanism is capable of copying any type of information. The Abstract Label approach is general to zero shot learning problems where novel items share characteristics with training instances. The Morphological Selector is also applicable to linguistic copy mechanisms in other contexts such as Commonsense Reasoning Lin et al. (2020) where copied terms may require linguistic alignment with the generated text.

We add an Inference Bias (IB) $b\in\mathbb{R}^{+}$ to increase $P(y^{l_{i}}_{t})$ at inference time. Eq. 9 is changed to:

$\displaystyle P(y^{l_{i}}_{t})$

$\displaystyle=b\cdot{\bm{c}}_{t,i}\cdot P(y_{t}^{l_{i}}|l_{i})$

(12)

and remaining probabilities normalised accordingly. IB is functionally equivalent to adjusting the threshold for the copy decision during inference. Surprisingly, this simple inference trick provides a strong baseline (see Table 3). This shows that after the CE training, many correct copy operations are assigned with low probabilities, compared to the fixed vocabulary items. However, we believe that it is better to train the model to increase the probabilities of these copy operations than adding ad hoc adjustments at inference time.

Rennie et al. (2017) shows that SCST with the CIDEr reward fine-tuning leads to noticeable caption quality improvement for standard image captioning tasks (i.e, improvement in various automatic metrics). Previous work Cornia et al. (2020); Anderson et al. (2018b) use CIDEr as the standard reward function in their SCST optimization. This shows suggests that the problem of overfitting of SCST training with CIDEr reward is minimal. Intuitively, the CIDEr reward is positively correlated with the number of salient object label mentions and should encourage the model to copy salient novel object labels. However, CIDEr equally rewards both generation of object labels present in training data via the vocabulary $P(y^{v}_{t})$ and via copy operations $P(y^{l_{i}}_{t})$. Novel objects labels however can only be generated by copy operations (see Sec. 3.1), thus the CIDEr reward function does not sufficiently encourage these operations. We propose two orthogonal modifications to the standard SCST algorithm to address this issue:

We propose combining the standard CIDEr-D reward with a reward function that gives captions with words copied from object labels an extra bonus, which we intend to encourage copy operations. One straightforward way to implement this idea is to provide a constant bonus to each triggered copy operation:

$\displaystyle\mathit{R_{a}}(X)=\mathit{CIDEr}(X)+a*C$

(13)

where $X$ is a generated caption, $C$ is the number of copy actions in the caption $X$ and $a\in R^{+}$ is a fixed hyper-parameter. We refer this as additive bias. Optimizing with the additive bias, the captioning model only needs to trigger the copy operation for arbitrary objects at arbitrary generation steps. That is, the model may encourage copying object labels at the expense of caption quality (i.e., high CIDEr-D scores). Therefore, we propose a proportional bias that assigns different rewards to the copy operations in different images by making a connection between the copy bonus and the generated captions CIDEr-D score:

$\displaystyle\mathit{R_{p}}(X)=\mathit{CIDEr}(X)*(1.0+p*C)$

(14)

where $p\in R^{+}$ is a fixed hyper-parameter. Although $\mathit{R_{a}}$ can effectively encourage the model to copy objects, it may introduce noisy object mentions. $\mathit{R_{p}}$ penalizes those noisy object mentions via the low caption CIDEr score.

VOA Images refers to the set of training images where the reference captions contain at least one word from retained object labels. During SCST training, images that contain no object label words (i.e., non-VOA images) will not utilise copy operations, thus these images encourage the model NOT to copy. VOA images account for approximately 70% of the full COCO-2017 training images set. Although restricting training to VOA images can be done with arbitrary models, this may hurt the diversity of generated captions. Experiments in Table 3 confirm that restricting to VOA images only improves performance when used with SCST training.

The above approaches introduce two additional parameters: $a$ and $p$. In our experiments, $a$ and $p$ range over 0.2, 0.3 and 0.4; we found that 0.3 works the best for both reward functions. Combined with restricting SCST training to VOA images, $\mathit{R_{p}}$ works better than $\mathit{R_{a}}$ and sets a new SOTA for novel object image captioning.

The ECOL model produces better captions using the frozen GPT2 parameters (row 1 vs. 2). Our copy mechanism (C) helps the model to explicitly integrate novel objects, substantially improving the out-of-domain split by 15.3 CIDEr and 0.3 SPICE (row 2 vs. 3). The Inference Bias (IB) introduces noticeable performance improvement: 8.4 CIDEr and 0.3 SPICE (row 3 vs. 4) in models that do not use our reinforcement learning approach. The ECOL model trained with the standard SCST reward function obtains an overall 8.1 CIDEr improvement, but most of the improvement is from the in-domain and near-domain splits (row 8 vs. 6). Compared with the ECOL + IB model, the ECOL model trained with standard SCST algorithm is 8.1 CIDEr lower in the out-of-domain split (row 5 vs. 4). As discussed in Sec. 3.2, standard SCST cannot provide sufficient copy encouragement as object words can be generated from either pathways (fixed vocabulary or copy). Optimizing either the $\mathit{R_{a}}$ or $\mathit{R_{p}}$ reward functions improves the ECOL + CIDEr model by 7.0 CIDEr and 7.8 CIDEr respectively (row 7 and 5). $\mathit{R_{a}}$ achieves 3.7 CIDEr higher than $\mathit{R_{p}}$ in the out-of-domain split. Interestingly, after restricting the model training to VOA images, $\mathit{R_{p}}$ achieves 7.8 CIDEr improvement in the out-of-domain split (row 8 vs. 7), outperforming the ECOL + $\mathit{R_{a}}$ w/ VOA model by 1.4 CIDEr (row 10 vs. 8).

We directly measure the copy quantity by counting the number of copied object labels and Object CIDEr. Row 5 and 3 confirm that the standard SCST algorithm has little impact on the copy quantity (only + 0.1 object per image and + 1.4 Object CIDEr). Inference Bias (IB), $\mathit{R_{a}}$ and $\mathit{R_{p}}$ rewards substantially improve the quantity of copied objects (row 4, 6, 7 vs. 3). Among these three components, the models trained with $\mathit{R_{a}}$ and $\mathit{R_{p}}$ work better than the IB baseline (row 6, 7 vs. 4). The model trained with the $\mathit{R_{a}}$ reward copies more objects than the $\mathit{R_{p}}$ reward, especially training with all training images. This is because the $\mathit{R_{a}}$ reward assigns constant positive reward for all copied objects. However, such a naive reward appears to encourage noisy copying operations (i.e., copying non-salient objects). As a result, the ECOL + $\mathit{R_{a}}$ model performs worse than the ECOL + $\mathit{R_{p}}$ model in terms of caption quality (row 7 vs. 6). After restricting training with VOA images, the models trained with $\mathit{R_{a}}$ and $\mathit{R_{p}}$ copy similar amount of objects, but the model with $\mathit{R_{a}}$ produce better captions than the one with $\mathit{R_{a}}$, especially in the out-of-domain split (row 10 vs. 8). The $\mathit{R_{p}}$ reward maintains a good balance between copying more objects and high caption quality.

Restricting training to the VOA images can be done with any captioning models. However, this does not necessarily encourage copy operations and improve the output caption quality. When we restrict training to VOA images, the ECOL-R model performs consistently worse in all three splits compared to our proposed training scheme (row 9 vs. 10). The only difference is that the ECOL model is not trained with diverse images during the cross-entropy stage. That is, restricting to VOA images is only suitable for fine-tuning in the SCST stage.

Here we investigate whether our ECOL-R model mentions a sufficient number of salient objects. We apply an increasing amount of inference bias to the ECOL, ECOL + CIDEr and ECOL-R models in Table 4. We note that only ECOL-R model is negatively impacted (measured by CIDEr score) by different Inference Bias values. This shows that the ECOL-R model does not benefit from further copy encouragement.

Table 5 shows the effect of Abstract Labels (AL) and the M Selector (M) in the ECOL + IB and ECOL-R models. Removing AL and M from the ECOL + IB model drops 2.3 CIDEr, 0.6 SPICE and 4.6 CIDEr, 0.8 SPICE respectively. AL and M have a large positive impact on the SPICE score. As SPICE is sensitive to long-range object word relationships, such as attributes and predicate words, Anderson et al. (2016a) Abstract Labels and the M Selector improve the semantic coherence and fluency of the captions. The performance gap in the ECOL-R model becomes smaller. Our copy encouragement approach contributes to the generation coherency and fluency.

To further show the generalization of our model for In-Domain images (i.e., without novel objects), we run the ECOL + IB and ECOL-R models on the COCO 2017 Validation Set and compare with another two novel object captioning models (NBT + CBS and Up-Down + ELMo + CBS) reported in Agrawal et al. (2019) in Table 6. Both of our models outperforms the Up-Down and NBT model by a large margin. Our models produce high-quality captions for images with novel objects as well as known objects.