Proposal Report for the 2nd SciCAP Competition 2024

Figure 1 illustrates the overall architecture of our solution, which contains three components: Prepare training data, Generate summaries and Model-ensemble. Next, we will specifically introduce the three components mentioned above.

OCR information extraction from images aims to convert textual information contained within images into a text format, enabling the generation of more accurate and detailed image descriptions, thereby enhancing understanding of the image content. However, we found that OCR information in the images provided by the official dataset is not accurate, as illustrated in the figure 2, making it difficult to generate precise image descriptions. To address this issue, we have chosen to use the PaddleOCR to re-extract OCR information from the images. This step aims to ensure that we obtain more accurate and reliable textual data to support subsequent summary generation tasks.

Filter gold sentence in paragraphs aims to filter out redundant information in paragraphs to create more refined and effective training data, which is beneficial for model training. Specifically, we first concatenate the provided multiple paragraphs into a complete paragraph, then use the SpacyTextSplitter tool to divide this complete paragraph into several different blocks, represented by $c$. Afterwards, we construct two types of input sequences: one is the mention $m$ combined with the chunk $c$, and the other consists only of mention $m$, which can be represented as $m\oplus c$ and $m$ respectively. We will measure the differences in the probabilities of model $M_{\text{filter}}$ to generate the given output $o$, this process can be represented as

$$I(c)=\frac{M_{\text{filter}}(o\mid c\oplus m)}{M_{\text{filter}}(o\mid m)}$$

. Furthermore, to achieve the goal of filtering out irrelevant sentences, we set a threshold $\lambda$. Sentences with probabilities below this threshold are filtered out, thereby the set of sentences that ultimately make up the paragraph is $\{c\mid I(c)\geq\lambda\}$ , considering that the sentences in the set are beneficial for generating the final summary. Finally, we concatenate all the sentences in the set to form the final paragraph, denoted as $\hat{p}$.

However, during the training phase, since the correct answers are already known, we can filter out irrelevant information from the paragraph using the aforementioned method. But during the testing phase, as the correct answers are unknown, in order to filter out irrelevant information from the test data, we trained the flan-t5-xl model. In the construction of training data for the model, the input consists of the prompt ”Filter out the irrelevant information in the text:” concatenated with the original paragraph $p$, and the model’s output is $\hat{p}$ obtained through the above method. The training process can be represented as $M_{filter}(\hat{p}\mid\text{prompt}\oplus p)$. In the testing phase, given the paragraph $p$ and the prompt, we use the model $M_{filter}$ to predict the filtered text, $\hat{p}=M_{filter}(\text{prompt}\oplus p)$. The filtered text is then used in the summary generation phase. Through this method, we filter out paragraph descriptions that are beneficial for generating image captions and reduce the model’s overhead during the summary generation stage.

Construction of prompts aims to ensures clarity in instruction and consistency in data input across different model architectures, facilitating effective training and evaluation. We further develop the training dataset for use in the summary generation phase. We have designed two types of prompts for the PegasusZhang et al. (2020), Pegasus-x-Large, and LLaMA2-13B models, respectively. These are structured as follows:

Prompt 1: ”OCR Text: Mentions: Paragraphs:”

Prompt 2: ”Summarize the following OCR text, mentions, and paragraphs, extracting key information and generating a concise summary. OCR Text: Mentions: Paragraphs: ”

Through the steps outlined in the previous section, we can obtain the text description $t$ corresponding to each image. During the training phase, the input to the generative model $M_{gen}$ is the constructed description $t$, and the ground truth is the actual caption $ans$ corresponding to the image, used as the output, which can be formalized as $M_{gen}(\text{ans}|t)$. In the inference phase, we input the constructed text description $t$ into the model, enabling the model to generate a summary for the given text.

Model ensemble is the final stage. Specifically, we train the Pegasus, Pegasus-x-Large, and LLaMA2-13B models and obtain their weights at different epochs. For Pegasus and Pegasus-X-Large, we use the weights from the 4th, 5th, and 6th epochs for inference, and each model generates 16 candidate summaries. For the LLaMA2-13B model, we use the weights from the 3rd, 4th, 5th, and 6th epochs for inference, with each weight generating one candidate summary. Thus, for each image, we obtain 100 candidate summaries, from which we select the summary that best describes the image. We adopt a method of sequentially assuming each of the 100 candidate summaries as the correct summary. The score of this summary is the average of the cumulative scores of the ROUGE-2-normalized scores compared to the other summaries, which can be formulated as:

$$s_{n}=\frac{1}{|R_{N}|-1}\sum_{r_{m}\in R_{N}\setminus\{r_{n}\}}\text{sim}(r_{n},r_{m})$$

, where $s_{n}$ is defined as an averaged score to all the other captions $(r_{m}\in R_{N}\setminus\{r_{n}\})$, $\text{sim}(r_{n},r_{m})$ is the ROUGE-2-normalized score between two captions $r_{n}$ and $r_{m}$ , and $R_{n}$ represents all the candidate summaries. Ultimately, we select the summary with the highest score as the caption for the image.

The dataset used in this study is the official dataset provided, which contains approximately 500,000 data samples. Each sample includes a description of an object, a paragraph of original text about the object (without preliminary extraction), and several sentences related to the object (extracted from the paragraph). About 400,000 samples were used for training, and 47,639 samples were used for testing.

We used two A100 GPUs to train the base models, including pegasus-x-large, and Pegasus, among others. The training was conducted for varying epochs, ranging from 3 to 10, with a learning rate set at 1e-5.

Our experimental results for the short track is presented in Tables 1. BLEU (Bilingual Evaluation Understudy) is a metric used to evaluate the quality of machine-translated text against one or more reference translations by measuring the n-gram overlap. ROUGE-1 (Recall-Oriented Understudy for Gisting Evaluation) calculates the overlap of unigrams (single words) between the generated text and reference text, focusing on recall. ROUGE-2 extends this by calculating the overlap of bigrams (two-word sequences), providing a more detailed measure of similarity. ROUGE-1 Normalized adjusts the ROUGE-1 score by the length of the reference text, giving a normalized recall value to account for text length differences. ROUGE-2 Normalized does the same for ROUGE-2, normalizing the bigram overlap score to ensure comparability across texts of varying lengths.