R³Net:Relation-embedded Representation Reconstruction Network
for Change Captioning

When the bi-directional difference representation $\hat{X}_{diff}$ is computed, we exploit it as the query to localize the changed feature in $\hat{X}_{bef}$ and $\hat{X}_{aft}$, respectively. Specifically, the dual change localizer first predicts two separate attention maps $a_{bef}$ and $a_{aft}$:

where [;], conv, and $\sigma$ denote concatenation, convolutional layer, and sigmoid activation function, respectively. Then, the changed features $l_{bef}$ and $l_{aft}$ are localized via applying $a_{bef}$ and $a_{aft}$ to $\hat{X}_{bef}$ and $\hat{X}_{aft}$:

Finally, we compute the local difference feature w.r.t. both $l_{bef}$ and $l_{aft}$ from two directions:

A syntactic skeleton predictor (SSP) is introduced to learn a set of syntactic skeletons based on the outputs derived from the R³Net. The predicted skeletons can provide the caption decoder with high-level semantic cues about changed objects and supervise the modeling of difference representation. This aims to enhance the semantic interaction between change localization and caption generation. Inspired by Gao et al. (2020); Gan et al. (2017), we treat this problem as a multi-label classification task. Suppose there are $N$ training image pairs, and ${y}_{j}=\left[y_{j1},\ldots,y_{jK}\right]\in\{0,1\}^{K}$ is the label vector of the $j$-th image pair, where $y_{jk}=1$ if the image pair is annotated with the skeleton $k$, and $y_{jk}=0$ otherwise.

Specifically, first, we apply a mean-pooling layer over the concatenated semantic representations of $\hat{X}_{bef}$, $\hat{X}_{aft}$, and $\hat{X}_{diff}$ to obtain a global semantic representation ${S}_{j}$:

Then, the probability scores $p_{j}$ of all syntactic skeletons for $j$-th image pair is computed by:

where ${p}_{j}=\left[p_{j1},\ldots,p_{jK}\right]$ denotes the probability scores of $K$ skeletons in $j$-th image pair. To maximize the probability scores of syntactic skeletons, we use the multi-label loss to optimize the SSP. It can be formulated as:

where $N$ and $K$ indicate the number of all training samples and annotated skeletons of an image pair. The loss can be considered as the supervision signal to regularize the learning of difference representation in the R³Net.

Since the predicted skeletons are the explicit semantic concepts of the changed object and its surrounding references, the captions are generated under the guidance of them. Specifically, first, the predicted probability scores $p_{j}$ are embedded as a skeleton feature $E[p_{j}]$:

where $E_{q}\in\mathbb{R}^{k\times M}$ is a skeleton embedding matrix and $M$ is the dimension of the skeleton feature. $W_{q}\in\mathbb{R}^{M\times M}$ and $b_{q}\in\mathbb{R}^{M}$ are the parameters to be learned. Then, we exploit a semantic attention module to focus on the key semantic feature from $l_{bef}$, $l_{aft}$, and $l_{diff}^{b\leftrightarrow a}$, which is relevant to the target word:

where $i\in(bef,aft,diff)$. $\beta_{i}^{(t)}$ is computed by an attention LSTM_a under the guidance of the predicted skeleton feature $E[p_{j}]$:

where $W_{a_{1}},b_{a_{1}},W_{a_{2}},$ and $b_{a_{2}}$ are learnable parameters. $h_{a}^{(*)}$ and $h_{c}^{(*)}$ are hidden states of the attention module LSTM_a and the caption decoder LSTM_c, respectively.

Finally, the caption generation process is also guided by the predicted skeleton feature. We feed it, attended visual feature, and the previous word embedding to the caption decoder LSTM_c to predict a series of distributions over the next word:

where $E$ is a word embedding matrix; $W_{c}$ and $b_{c}$ are learnable parameters.

We jointly train the caption decoder and SSP in an end-to-end manner. For the SSP, the multi-label loss is minimized by the Eq. (14). For the decoder, given the target ground-truth words $\left(w_{1},\ldots,w_{m}\right)$, we minimize its negative log-likelihood loss:

where $\theta_{c}$ are the parameters of the decoder and $m$ is the length of the caption. The final loss function is optimized as follows:

where the hyper-parameter $\lambda$ is to seek a trade-off between the decoder and SSP.

CLEVR-Change dataset Park et al. (2019) is a large-scale dataset with a set of basic geometry objects, which consists of 79,606 image pairs and 493,735 captions. The change types can be categorized into six cases, i.e., “Color”, “Texture”, “Add”, “Drop”, ‘’Move” and “Distractors (e.g., viewpoint change)”. We use the official split with 67,660 for training, 3,976 for validation and 7,970 for testing.

Spot-the-Diff dataset Jhamtani and Berg-Kirkpatrick (2018) contains 13,192 well-aligned image pairs from surveillance cameras. Based on the official split, the dataset is split into training, validation, and testing with a ratio of 8:1:1.

Following the state-of-the-art methods Park et al. (2019); Shi et al. (2020); Tu et al. (2021), we use five standard metrics to evaluate the quality of generated sentences, i.e., BLEU-4 Papineni et al. (2002), METEOR Banerjee and Lavie (2005), ROUGE-L Lin (2004), CIDEr Vedantam et al. (2015) and SPICE Anderson et al. (2016). We get all the results based on the Microsoft COCO evaluation server Chen et al. (2015).

We use ResNet-101 He et al. (2016) pre-trained on the Imagenet dataset Russakovsky et al. (2015) to extract object features, with the dimension of 1024 $\times$ 14 $\times$ 14. We project these features into a lower dimension of 256. The hidden size of overall model is set to 512 and the number of attention heads in relation-embedded module is set to 4. The number of skeletons in an image pair is set to 50. The dimension of words is set to 300. For the hyper-parameter $\lambda$, we empirically set it as 0.1. In the training phase, we use Adam optimizer Kingma and Ba (2014) with the learning rate of 1 $\times$ $10^{-3}$, and set the mini-batch size as 128 and 64 on CLEVR-Change and Spot-the-Diff. At inference, for fair comparison, we follow the pioneer works Park et al. (2019); Jhamtani and Berg-Kirkpatrick (2018) in the two datasets to use greedy decoding strategy for caption generation. Both training and inference are implemented with PyTorch Paszke et al. (2019) on a Tesla P100 GPU.

To figure out the contribution of each module of the proposed network, we conduct the following ablation studies on CLEVR-Change: (1) Baseline which is based on DUDA Park et al. (2019); (2) RRM which is the representation reconstruction module; (3) R³Net which augments the RRM with a relation-embedded module; (4) R³Net+SSP which augments the R³Net with a syntactic skeleton predictor.

The evaluation on Total Performance. Total performance is to simultaneously evaluate the model under both scene change and none-scene change. Experimental results are shown in Table 1. We can observe that each module and the full method improve the total performance of Baseline. This indicates that our method not only can correctly judge whether there is an semantic change between a pair of images, but also can describe the change in an accurate natural language sentence.

The evaluation on the settings of Scene Change and None-scene Change. In the setting of scene change, both object and viewpoint changes happen. In the setting of none-scene change, there are only distractors, such as viewpoint change and illumination change. The experimental results are shown in Table 2. Under the setting of scene change, we can observe that 1) the RRM, R³Net, and R³Net+SSP all significantly improve the Baseline; 2) the R³Net is much better than the RRM; 3) the best performance is achieved when augmenting the R³Net with the SSP. The above observations indicate that 1) compared to direct subtraction between a pair of unaligned images, it is effective to capture difference representation via the R³Net, because it can overcome the distraction of viewpoint change; 2) learning semantic relations among object features is important, because these relations can enrich the raw object features, helpful for exploring fine-grained changes; 3) the SSP can enhance the semantic interaction between change localization and caption generation, and thus further improve the quality of generated sentences.

Besides, under the setting of non-scene change, we can observe that the RRM is worse than the Baseline on some metrics. Our conjecture is that, on one hand, due to the large amount of clutter and only representing the image pair at feature-level, the RRM cannot learn the exact semantic similarities of corresponding locations in the two images, performing worse on some metrics. On the other hand, the Baseline learns a coarse difference representation between two unaligned images by a direct subtraction, so it is prone to learn a wrong change type or simply judge nothing has changed. This leads to the results that it performs worse than the RRM with the total performance and scene change, but achieves higher scores than the RRM on some metrics with none-scene change. In fact, when embedding semantic relations among object features, the R³Net outperforms the Baseline in the both settings. This further indicates that it is beneficial to thoroughly understand image content via modeling semantic relations among object features.

Figure 3 shows an example about the case of “Move” from the test set of CLEVR-Change. We can observe that DUDA localizes a wrong region on the “before” and thus misidentifies “Move” as “Add”. By contrast, the R³Net+SSP can accurately locate the moved object on the “before” and “after” images, which benefits from two merits. First, the R³Net is able to localize the fine-grained change in the presence of viewpoint changes. Second, the SSP can predict the key skeletons based on the representations of image pair and their difference learned from the R³Net. For instance, the skeletons of “changed” and “location” has the higher probability scores than “newly” and “placed”. This can provide the decoder with high-level semantic cues to generate the correct sentence.

Figure 4 illustrates two cases about “Move”. In the left example, the R³Net+SSP successfully distinguishes the changed object (i.e., small grey cylinder) and predicts accurate skeletons with high probability scores. The right example is a failure case. In general, we can observe that the grey cylinder is localized and the main skeletons are predicted, which indicates that the R³Net learns a reliable representation of difference. However, the decoder still generates the wrong sentence. The reason behind the failure may be that the movement of this cylinder is very slight and the decoder receives the weak information of change (including skeletons). In our opinion, a possible solution for this challenge is to model position information for object features, which would enhance their position representation ability and help localize the slight movements.