Align Vision-Language Semantics by Multi-task Learning for Multi-Modal Summarization

First of all, we formalized the input and output of our ViL-Sum as $(D,I)$ and $(S,I_{S})$, where $D=\{t_{1},t_{2},\dots,t_{T}\}$ refers to the sequence of tokens from the input document, $I=\{img_{0},img_{1},\dots,img_{M}\}$ refers to the sequence of input images from the input document, $S=\{t_{1},t_{2},\dots\}$ refers to the sequence of tokens from gold text summary, and $I_{S}=\{img_{1},img_{2},\dots,img_{K}\}$ refers to $K$ selected images for the multi-modal summary.

The vision-language representations $H_{L}$ from the previous multi-modal encoder contain multi-modal features of input text and images. After encoding, we feed the representations $H_{L}$ into the decoder to generate a text summary. The target of the summary generation task is to minimize the negative log-likelihood of the reference $y$ tokens as given input document $D$ and images $I$ via updating model parameters $\theta$. The loss function of the summary generation task is as follows:

Different from single-modal summarization tasks, this optimization target also depends on the features from input images $I$, which enhance the final summary generation.

To align the paragraphs and images from the input, in this section, we introduce a simple yet effective task, image reordering, to guide the model to learn semantic alignment. Specifically, we shuffle the order of input images and then force the ViL-Sum model to predict the original order of input images with a classification head:

where all input images share one classification head. To train the classification layer, the model computes loss and minimizes the objective function:

where $C$ is the number of categories, depending on the number of input images. We set $C=10$. if the number of input images is greater than 10, we only keep the first 10 images as input images.

We also train our ViL-Sum with multi-modal output reference following Zhu et al. (2020). To build pseudo image selection labels of training data, we employ similarity between image caption and gold summary to select top-$K$ images as labels $\hat{y}$ ($K$ is empirically set as 3). The similarity is the average of ROUGE-1, ROUGE-2, and ROUGE-L scores. The probability to select each image is as follows:

The loss function of the image selection task is as follows:

We train our ViL-Sum with a text summary generation task and two well-designed auxiliary tasks in a multi-task manner, which are used to enhance vision-language representation and paragraph-level semantic alignment. In previous sections, we have introduced the details of them. Finally, ViL-Sum is trained with three tasks: summary generation, image selection, and image reordering, jointly by simultaneously minimizing three loss functions as follows:

It is deserved to mention that the caption of the image is not always available. So the image selection task is not generalization for all datasets. If we remove the image selection task, we can select images by measuring the similarity between generated summary and vector representations of images. Our proposed multi-modal encoder and the image reordering task still help the model achieve excellent performance.

We employ the MSMO dataset Zhu et al. (2018) to evaluate the effectiveness of our proposed ViL-Sum. MSMO dataset is a large-scale dataset for the Multi-modal Summarization with Multi-modal Output tasks. Each example in the dataset is a triplet (document, images, summary), which contains more than one image in each example. This dataset contains online news articles (723 tokens on average) paired with multiple image-caption pairs (6.58 images on average) and multi-sentence summaries (70 tokens on average). For test data, based on text reference, at most three images are annotated to produce a multi-modal reference by humans. The detailed statistical information of the MSMO dataset is shown in Table 1.

We report the existing multi-modal summarization methods (ATG, ATL, HAN, GR) Zhu et al. (2018) and MOF${}^{RR}_{dec}$ Zhu et al. (2020) using multiple metrics. We also report the result of PGC See et al. (2017), which is a single-modal summarization model.

To prove the effectiveness of our proposed joint representation and multi-task learning, we mainly compare with BART-base Lewis et al. (2020) model and a reproduced two-stream model BART-cross which has the same structure with MOF${}^{RR}_{dec}$ and replace GRU and VGG19 Liu and Deng (2015) with BART and ViT Dosovitskiy et al. (2021) respectively. To be fair, we mainly compare our model with BART-base and BART-cross due to previous methods did not employ pre-trained models. The details of these models are as follows:

• •

  PGC is the BiGRU-based pointer-generator network that allows both copying words from the input text and generating words from a fixed vocabulary.

• •

  ATG is based on the PGC model. It fuses static visual features from VGG19 with text features after the BiGRU encoder. Besides, ATG selects final images by the visual-text attention weight.

• •

  ATL replaces the image global features of ATG with local features (multiple pooling features), which select images by measuring the sum of visual attention distribution over the local patch features of each image.

• •

  HAN is based on the ATL model and adds a hierarchical attention mechanism. This attention mechanism first attends to the image patches to get the intermediate vectors to represent images and then attends to these vectors to get the visual context vector.

• •

  GR is an extractive method that employs LexRank Erkan and Radev (2004) to rank captions of images and select images based on the rank score. The text summary of it is generated by the PGC model.

• •

  MOF${}^{RR}_{dec}$ is based on ATG model. This model first constructs pseudo-labels of image selection for the final summary. Specifically, it employs the ROUGE score to measure the relevance of image caption and summary text.

• •

  BART-base is a pre-trained seq2seq generation model, which achieved promising results in many generations of NLP tasks, especially on text summarization. We employ this model to confirm visual features’ contribution to a summary generation.

• •

  BART-cross is a BART-based model with the same model structure as previous ATG, ATL, HAN, GR, and MOF${}^{RR}_{dec}$. It first encodes images with ViT and then fuses text representation from the BART encoder output. The fusion of image and text representations employs cross-attention like the ATG model. This is the main comparison model.

For a fair comparison, we construct this BART-cross model to prove the effectiveness of joint multi-modal encoder and multi-task training in our ViL-Sum. Because our ViL-Sum without multi-task training only changes the encoding mechanism from separate encoders to the joint multi-modal encoder.

We train our model for 10 epochs on 8xV100 GPUs using Adam Kingma and Ba (2015) with $\beta_{1}=0.9$, $\beta_{2}=0.99$, a batch size of 64. We also use a linear learning rate warm-up with 1,000 steps. The weight-decay is set as $10^{-4}$. The model is initialized with ViT-B/16 and BART-base parameters. The max length of input images and tokens are 10 and 512 respectively. For the image tokenizer, we employ the same setting with ViT-b/16 in Dosovitskiy et al. (2021). During testing, we generate the summary with a beam size of 3, and the minimum and maximum decoding lengths are set as 15 and 150 separately.

We evaluate the pictorial summary with the MMAE metric Zhu et al. (2018). ²²2Comment: Zhu et al. (2020) also proposed a MMAE+ to better evaluate MSMO task. However, the author did not release their MR model, which is the core component of their MMAE+. We find that the performance of MMAE and MMAE+ is very closer and consistent.

MMAE consists of three sub-metrics: ROUGE score (ROUGE-L), Image Precision (IP), and Image Text Relevance (MAX_sim). ROUGE Lin (2004) score can measure the salience of text in generated summary, which is widely used for measuring summarization systems. The image precision can measure the salience of selected images and is computed as Equ. (11).

where $ref_{img}$ and $rec_{img}$ denote reference images and recommended images by MSMO systems respectively. MAX_sim can measure the relevance between selected images and generated text summary, which trains an image-text retrieval Faghri et al. (2018) model with max-margin loss to evaluate Image-Text relevance. Finally, Zhu et. al. Zhu et al. (2018) choose the linear regression results of 3 metrics as MMAE with human judgments and the weight for ROUGE-L, MAX_sim, and IP is 1.641, 0.854, 0.806 respectively, the intercept is 1.978.

We report the results of ROUGE-1/2/L, MAX_sim, IP, and MMAE of each model to comprehensively measure their performance. The results of our model are all the averages of three different checkpoints.

The main results of all models are shown in Tab. 2. Models in block 1 are based on the pointer network with Bi-GRU. Models in block 2 are based on the BART-base model. SEL means selection task and REO means reordering task. All reported results of ours are the average of 3 different checkpoints.

We can see that compared with the baselines, our ViL-Sum gains significant improvement on all metrics, and ViL-Sum+selection, reordering achieves the best comprehensive performance. Compared with BART-cross, we can see that the joint representation and multi-task training both bring satisfactory improvement, which proved the effectiveness of our proposed methods. Interestingly, the introduction of image features hurts the performance of all single-modal summarization models, especially Bi-GRU-based models.

Firstly, we can see that the performance of ATG, ATL, HAN, and GR all hurt ROUGE scores by simply introducing images as independent visual features. Through the multi-modal objective optimization, MOF${}^{RR}_{dec}$ has a significant improvement on IP and does not decrease the quality of generated text summary. This situation proves that modelling vision and language information independently did not bring in the revenue for text summary generation. The results of BART-cross, which also introduces images as independent features, also have lower ROUGE scores than BART-base. This situation proved again the previous conclusion.

Different from previous performance on ROUGE score, our ViL-Sum with joint vision-language representation obtains better ROUGE scores, and the Image Precision (IP) and MAX${}_{s}im$ both have a significant improvement. This demonstrated that using the joint multi-modal encoder to obtain vision-language representation is better than using separate encoders with cross-attention to fuse multi-modal features.

The result of ViL-Sum without multi-task learning has achieved new state-of-the-art performance and is better than BART-cross. In this section, we will analyze the influence of our proposed multi-task learning. From the results, we can see that the introduction of image selection and reordering bring a slight decrease in ROUGE scores. Meanwhile, the IP and MAX_sim scores increase significantly, which makes the overall score MMAE better than ViL-Sum without multi-task training.

We report the ablation study results of two auxiliary tasks in the second block of Table 2. From the results, we can see that image selection and reordering both can bring improvement in IP and MAX_sim scores. The combination of two tasks can push the overall score MMAE higher. The comparison of these models demonstrated that the introduction of multi-task learning exactly improved the vision-language representation and semantic alignment, which is reflected in the improvement of the multi-modal metrics: IP, MAX_sim and MMAE.

We randomly sample 100 examples from the test set to conduct the human evaluation. The multi-modal summary of golden reference, BART-base, BART-cross, and our ViL-Sum (best) are evaluated by three human annotators. Each annotator will score each example with a rating scale from 1 (worst) to 5 (best). Table 3 shows the average scores from three annotators (t-test, $p<0.05$). We can see that annotators tend to give the multi-modal summary from BART-cross and our ViL-Sum higher scores. In addition, our ViL-Sum outperforms two strong baselines by a wide margin and is close to the references.

Tab. 4 depicts the experimental results of our model performance varying with different $K$ (the image number at the final summary). Since the golden reference in the test set contains three images, the consistency between training and test makes the model perform best when $K$ is 3. Overall, our model is not very sensitive with $K$. With different $K$, our ViL-Sum all achieve excellent performance, which proves our method can identify the real importance images from multi-modal inputs. Besides, we guess the image selection of the MSMO dataset is simple due to the data from the news.

To further evaluate the effectiveness of joint modelling and multi-task learning, we replace the backbone of the image tokenizer to observe the performance of ViL-Sum. We replace the ViT backbone with Linear Layer and an image tokenizer from Vision Transformer Wu et al. (2020). Both of them have much smaller parameters than the ViT backbone. Specifically, linear is the simple version of ViT which replaces the transformer image encoder with a simple linear layer to map the images into visual token embeddings. Vision is an image tokenizer from Vision Transformer Wu et al. (2020), which can convert one image into several visual token embeddings. Table 5 reports the results of them. We can see that the ViT exactly provides better visual features than the other two backbones. However, the performance does not drop sharply with the replacement of the image tokenizer. This proves that Our proposed two strategies are robust and the ViL-Sum is flexible with different image tokenizers.

We select one typical example from the test set and visualize the relevance of 1) summary sentences and selected images; 2) selected paragraphs and images; 3) all tokens and images in Figure 4 and 5. Each colour block means a cosine similarity between the image and text object. The darker colour refers to a higher similarity in the heatmap. With our proposed methods, the generated summary contains high-quality summary with three related images as shown in Figure 4a. From three different relevant visualizations, we can see that our ViL-Sum can effectively align the semantic representation of summary sentences and selected images as shown in Figure 4b. The input images can be aligned with paragraphs by training with image reordering as shown in Figure 4c. We also report the heatmap of all input tokens and images in Figure  5, which is consistent with Figure 4b and 4c. This case proves that the multi-task training really helps ViL-Sum learn reasonable relations between images and input paragraphs.

Recently, text summarization models have achieved remarkable performance with the development of pre-trained language models. Liu and Lapata Liu and Lapata (2019) first apply the pre-trained language model BERT to summarization tasks. They added several transformers as the decoder to the BERT encoder and then train them with different learning rates. Their work outperformed all traditionally trained neural models. Pegasus Zhang et al. (2019) and BART Lewis et al. (2020) are two fully pre-trained models for summarization generation with well-designed self-supervised tasks. Their appearance provided powerful base models for summarization and totally changed the research paradigm in the summarization task. After that, more and more summarization works begin to focus on pre-trained language models, including supervised and unsupervised methods Zhong et al. (2020); Liang et al. (2021b, a, 2022); Dong et al. (2023).

Large-scale Transformers-based Vaswani et al. (2017) vision and language representation models Radford et al. (2019); Devlin et al. (2019); Lewis et al. (2020) have achieved state-of-the-art results on many Natural Language Processing (NLP) tasks. They first pre-trained on a large-scale corpus with self-supervised tasks and then fine-tuned on specific downstream tasks. Most existing vision and language pretraining (VLP) models Tan and Bansal (2019); Li et al. (2021) adopt two different encoders to model vision and language separately, which extracts visual features by an object detection model and then combines the derived object-centric representation of the image and text. Recently, large-scale vision and language representation learning has tried to jointly encode different modalities with the same encoder and achieved promising improvements Li et al. (2020a, b, c); Zhang et al. (2021b); Xu et al. (2021); Zhou et al. (2020). Their success proves the joint modelling of different modalities is practicable.

Different from single-modal text summarization, multi-modal summarization is a task to generate a condensed summary to cover the primary information from multimedia data. One of the most significant characteristics of this task is it is not only based on text information, but can also employ rich visual information from images, audio, and videos. Multi-modal summarization tasks can be divided into two types with different outputs: single-modal output Evangelopoulos et al. (2013); Chen and Zhuge (2018); Li et al. (2018) and multi-modal output Bian et al. (2015); Zhu et al. (2018, 2020). Compared with single-modal output, a multi-modal output summary can increase users’ satisfaction Zhu et al. (2018) and first proposed a large-scale Multi-modal Summarization with Multi-modal Output (MSMO) dataset. To tackle the gap between training and testing in the MSMO task, Zhu et. al. Zhu et al. (2020) proposed two methods to obtain pseudo image labels and train the model with multi-modal optimization objectives.

However, previous works all obtain vision-language representation via separate encoders for different modalities, which has been proved weaker than joint representation in vision-language representation learning research Zhou et al. (2020); Xu et al. (2021). Besides, they ignored the special paragraph-level semantic alignment between different modalities. In this paper, we proposed a novel vision-language summarization ViL-Sum model with a multi-task learning framework to tackle these issues.