Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction

Large language models (LLMs), with their capacity to interpret and process human language in a manner akin to human cognition, facilitate the relation understanding of complex tasks. However, the enormous size of LLMs renders them impractical for industrial use. In the present study, we endeavor to explore an efficient methodology to distill the reasoning prowess of LLMs into a compact model that retains superior performance while ensuring minimal inference latency.

However, traditional knowledge distillation methods Gou et al. (2021) are incompatible to achieve this goal. The reasons are mainly twofold. (1) Prevailing LLMs are predominantly generative models that either produce unstructured output logits Touvron et al. (2023) or are solely accessible through official API ¹¹1https://openai.com/blog/chatgpt. (2) Existing LLMs exhibit suboptimal performance in NER and RE tasks Li et al. (2023a); Kai Zhang (2023), attributable to their lack of training under task-specific supervision. To address these issues, we introduce Chain-of-Thought Prompt Distillation (CoTPD), a novel approach designed to distill the explicit CoT knowledge into the student model. This method seeks to facilitate knowledge transfer from the parameterized LLM $\stackrel{{\scriptstyle prompt}}{{{\longrightarrow}}}$ contextual demonstration $\stackrel{{\scriptstyle distillation}}{{{\longrightarrow}}}$ parameterized student model. Moreover, by training on NER and RE tasks, the student model can obtain promising performance with the help of CoT knowledge.

In addition, existing methods Wang et al. (2022c); Xu et al. (2022); Li et al. (2020) typically demonstrate their intuition and superiority through case studies or Class Activation Mapping (CAM) visualization. However, these strategies still fall short in providing comprehensive interpretability about how models interpret the multimodal context to reach the final prediction Sun et al. (2019). In this investigation, we demonstrate that CoT knowledge, which is akin to how humans understand each sample, benefits the model performance in various domains and tasks.

Given an text and image pair $(\bm{x},\bm{I})$ where $\bm{x}=\{x_{1},x_{2},..,x_{n}\}$ has $n$ words, our method aspires to predict the outputs $\bm{y}$, which could represent the label sequence or the relations for NER and RE tasks respectively. As shown in Figure 2, we initially employ image caption technology, i.e., BLIP2 Li et al. (2023b), to convert image $\bm{I}$ into an image caption. Subsequently, we utilize langchain²²2https://github.com/hwchase17/langchain with preordained CoT prompts to query the LLM, thereby acquiring CoT knowledge $\bm{c}$ pertaining to each sample. Ultimately, the student language model amalgamates the text-image pair with CoT knowledge as the knowledge-enhanced view, and concatenates the text with a learnable conditional prompt to create a prompt-enhanced view. The objective is to minimize the Kullback-Leibler divergence over the probability distributions of the two views, thereby distilling the contextual CoT knowledge into the parameterized prompt.

Multi-grain CoT Prompts. To fully understand the text-image pair, we design CoT prompts in noun, sentence, and multimodality perspectives to query LLMs³³3Weaker LLM may need accurate descriptions in prompts. We take GPT4 in the Twitter dataset as an example., as shown in Figure 2.

Noun: We query LLMs with ‘Help me explain the meaning of special words for understanding. + $\bm{x}$’, which can explain potential entities, slang, and terminology.

Sentence: For each sample, we query LLMs with ‘Explain the sentence to me with necessary background. + $\bm{x}$’ . It can explain the sentiment, cause, and subject of users.

Multimodality: Similarly, we obtain multimodal relation by prompt LLMs with ‘What is the relation between the text and the attached image? + $\bm{x+I}$’. In this way, LLM can illustrate the vague relation between entity and object and explicitly determine whether the visual information is noise.

Data Augmentation CoT Prompts. We also leverage LLMs to do fact-based multimodal data augmentation, involving linguistic style, entity, and image perspectives.

Style: We query LLMs with ‘Transform the sentence in Twitter style without changing the meaning. + $\bm{x}$’ It diversifies the textual expression with consistent entity and meaning.

Entity: Initially, we adhere to Wu et al. (2022) to substitute the entity in $\bm{x}$ with an entity of the same type sourced from the MNER dataset. For instance, we replace the entity ’Tobey Maguire [PER]’ with ’Barack Obama [PER]’. Then, we assess the factual accuracy of the augmented sample by prompting the LLM with the query ’Whether the sentence is possible in fact, answer yes or no. + $\bm{x}$’. We retain only the samples that receive a positive response indicating factual plausibility.

Image: We generate the visual information of text by querying LLMs ‘What is a possible image with the text in a tweet? + $\bm{x}$’

We unify multi-grain CoT knowledge with the original image and text pair. For data augmentation CoT knowledge, we reorganize it as a new sample. Notably, the total knowledge may exceed the max token limits for the student model, therefore, we first summarize the over-length knowledge.

To distill CoT knowledge $\bm{c}$ into a compact model, we first try multi-view alignment Wang et al. (2021a) to match the txt-only output distribution and text-image-CoT output distribution. However, we find it results in suboptimal performance (Sec 4.3). We believe the reasons are twofold: (1) CoT knowledge is higher-level context than previous retrieve knowledge. (2) The knowledge is personalized which needs extra parameters to distill. Therefore, inspired by prompt learning Li and Liang (2021); Zhou et al. (2022a), we propose a conditional prompt distillation with respect to the text to hint the student model adjustably.

We introduce our conditional prompt generator, which is composed of a transformer decoder with learnable queries $\bm{q}\in\mathbb{R}^{N\times d}$. The conditional prompt $\bm{p}$ is generated by:

where $W_{q},W_{k}$, and $W_{v}$ are learnable projectors for query, key, and value. We concatenate text, image caption, and CoT knowledge as the knowledge-enhanced view $[\bm{x};\bm{I};\bm{c}]$. Similarly, the prompt-enhanced view is composed by concatenating text and conditional prompt as $[\bm{x};\bm{p}]$. We feed these two views to the same text encoder $E$ separately:

where $H=\{h\}_{1}^{n+l}$ denotes the output of encoder with max token limit $n+l$. The subscripts $k$ and $t$ represent knowledge-enahcned and prompt-enhanced view respectively. The former $n$ tokens correspond to the embedding of the text part in two views, which are used for following NER or RE prediction.

Then, we reduce the gap between these two views over the output distributions, so that the CoT knowledge and image are distilled into the student model under specific task. We denote it as conditional prompt distillation loss:

Besides, for MNER and MRE, we use the negative log-likelihood (NLL) as the training loss for the knowledge-enhanced view with gold labels $\bm{y}^{*}$:

Thus, the total loss $\mathcal{L}_{total}$ is the combination of $\mathcal{L}_{CPD}$ and $\mathcal{L}_{NLL}$ with coefficient factor $\alpha$:

Datasets. For MNER evaluation, we verify the effectiveness of our method on Twitter2015 Zhang et al. (2018), Twitter2017 Yu et al. (2020), SNAP Lu et al. (2018) and WikiDiverse Wang et al. (2022a). These datasets containing 4,000/1,000/3,357, 3,373/723/723 and 4,290/1,432/1,459, 6,312/755/757 sentences in train/development/test split respectively. The former three datasets are collected from social media domain, while WikiDiverse is reorganized by Wang et al. (2022a) from Wikinews as a multi-modal NER dataset. For MRE, we use MNRE dataset Zheng et al. (2021b) constructed by user tweets, which contains 12,247/1,624/1,614 sentences in train/development/test split.

Model Configuration. To fairly compare with most of the recent works, we adopt BERT-base-uncased as our text encoder. Besides, we follow Wang et al. (2021a, 2022a) to adopt XLM-RoBERTa-large (XLMR) to achieve state-of-the-art performance. For LLM prompting, we use langchain to query each sample in a dialogue way and record the answer of LLMs as CoT knowledge. The version of ChatGPT used in our experiments is gpt-3.5-turbo and the version of GPT4 is gpt-4. All LLMs used in our method set the sampling temperature to 0 for stable output.

Training Configuration. Our approach is implemented in Pytorch framework on an NVIDIA A100 GPU. We adopt AdamW optimizer to minimize the loss function. Besides, we follow Wang et al. (2022a) to grid search the learning rate of our models within [$1\times 10^{-6}$, $5\times 10^{-5}$]. The max length of sentence input is set to 512 to cover more CoT knowledge. The model undergoes a fixed 15 epochs of training with 32 mini-batch.

Baseilne Model and Variants. We refer to a model training and testing with [TXT]+[IMG] input as our baseline model. For the knowledge variants, we take retrieve knowledge from MoRe Wang et al. (2022a) as Retri, and knowledge queried from Wikipedia Wang et al. (2021b) as Wiki, and our CoT knowledge as CoT. For Retri, MoRe provides image-retrieved knowledge Retri_txt and text-retrieved knowledge Retri_img. For distillation variants, we refer to multi-view alignment from ITA Wang et al. (2021a) as MV. Furthermore, in our method, considering different kinds of learnable prompts, we compare prefix Li and Liang (2021) and unconditional prompt Zhou et al. (2022b), which are denoted as PrefixD and UPD respectively, to verify the effectiveness of the proposed conditional prompt distillation (CPD).

We compare our approach with other BERT-based Wang et al. (2022c); Lu et al. (2022); Zhao et al. (2022) and XLMR-based Wang et al. (2021a, 2022b, 2022a, 2022c) state-of-the-art methods on MNER and MRE. To fully show the effectiveness of our approach, we evaluate ChatGPT and GPT4 on Twitter2015, Twitter2017 and MNRE. For our approach, we also show the model performance using different LLM teacher and student models. As shown in Table 1, we observe LLMs are inferior to existing state-of-the-art methods, but the CoT knowledge significantly boosts our method. We think the training data of LLMs includes the dataset but the training of LLM is not in NER or RE supervision. Therefore, LLMs can generally understand each sample but still underperform in specific tasks. Besides, compared with other previous SOTA methods, our approach suppresses them by a large margin. Compared to MoRe, our method achieves 0.82% and 5.96% improvement on Twitter2015 and MNRE respectively. We attribute such significant gain to the direct and comprehensive CoT knowledge from LLM, so that the student model can easily understand the multimodal sample, instead of analogically learning from the related samples. We also observe that the ability of LLMs and student models significantly influences the final performance. For example, with respect to the same student XMLR model, GPT4 can generally provide a further 0.4%-1.2% improvement over ChatGPT. Similarly, a powerful student model also brings large performance boosts.

Effectiveness of Different CoT Knowledge. Multi-grain CoT knowledge focuses on explaining each sample from different perspectives, leading to significant improvement by enhancing the understanding of the student model. Specifically, as shown in Table 2, noun knowledge contributes more to MNER than MRE while sentence knowledge brings more improvement to MRE. We believe these two kinds of knowledge meet the demand of two tasks where MNER focuses on entity mention and MRE concentrates on the entity relation. Besides, it is worth noting that data augmentation CoT knowledge, i.e., style, entity, and image augmentation, contributes less than multi-grain CoT knowledge. It is because data augmentation mainly generates more samples for training, rather than directly explaining each sample. Therefore, it is more effective in cross-domain and low-resource cases, which will be elaborated on next section.

Comparison with Previous Retrieved Knowledge. In Table 3, we also report the results using retrieved knowledge to verify the effectiveness of our CoT knowledge. For retrieved knowledge from Wikipedia and the sample database, our CoT knowledge is better than them by a large margin. For example, CoT suppresses Wiki with 2.09% gains and outperforms Retri_txt with 2.19% improvements on Twitter2015. It demonstrates that such direct CoT knowledge is more effective to understand each sample. Furthermore, we investigate how knowledge difference influences the final prediction in Appendix. It shows accurate and direct CoT knowledge contains better potential in interpretability and performance over existing methods.

Distilling Knowledge into Student Model. As shown in Table 3, we also compare the results of different knowledge+distillation variants on MNER and MRE. It is obvious that multi-view alignment is unsuitable for distilling CoT knowledge to the student model, where ‘baseline+CoT+MV’ is 4.33% lower than ‘baseline+CoT’ on MNRE. We think such explicit and direct knowledge is higher-level than the retrieved knowledge. Thus, distilling it into the student model is harder. Alternatively, our conditional prompt distillation can maintain the performance with minimal decrease. Moreover, we test other two kinds of prevailing learnable prompts for distillation, i.e., prefix and unconditional prompt. It shows the proposed conditional prompt is better than them. We think the personalized/conditional prompt with respect to the text can hint the student model easier.

Cross-domain Generalization. The difference in type distribution and data characteristics often brings significant performance gaps in practice. Thus, we examine the cross-domain generalization of our method in Table 4. First, without data augmentation, our method achieves comparable results as the previous best model where our method suppresses CAT-MNER with 0.68% F1 improvement but is inferior to it with 0.28% decrease in two cross-domain settings respectively.

Second, when the entity augmentation uses entities from the source dataset (in-domain aug), our method achieves 1.59% and 0.73% F1 gains over CAT-MNER. It demonstrates the data augmentation strategy can substantially boost generalization ability. Besides, in this work, we aim to let the student model inherit the zero-shot ability of LLM. Therefore, we first query LLMs to provide a list of 400 Twitter entities, covering four types but without overlapping the entities in the source set, and then apply the novel entities for entity replacement. We refer to this implementation as zero-shot augmentation. As shown in Table 4, augmenting novel entities can further provide a 0.3%-0.6% F1 improvement. This demonstrates the effectiveness of utilizing data-augmentation CoT knowledge for achieving cross-domain generalization

Augmentation Times for Data Efficiency. The size of augmented samples is essential for final model performance. Typically, we randomly sample $\{50,100,200,400\}$ image-text pairs from Twitter2015 and Twitter2017 with different augmentation times to examine the effectiveness of our multimodal augmentation strategy. As depicted in Figure 3, increased augmentation yields improved performance, particularly when larger examples are utilized. Notably, our augmentation strategy demonstrates a linear enhancement with the progression of augmentation times. We attribute this improvement to the diversity and authenticity from linguistic style, factual entity replacement and image imagination, rather than simply substituting entities Wu et al. (2022).

Influence of BLIP2. We observe that BLIP2 Li et al. (2023b) has basic visual-entity alignment ability. It means it may recognize the entity mentions in the image, which may be unfair for performance comparison. We discuss the influence of BLIP2 in Table 5 and observe that, for both ITA and our method, BLIP2 provides 0.3%-0.5% improvement over VinVL Zhang et al. (2021) on two datasets. However, even using VinVL for image caption, our method can still achieve significant improvement over the previous best method MoRe. We think LLM can infer visual-entity relation from the general noun description, as shown in the multimodal CoT knowledge in Figure 2.

Case Study. We showcase how CoT knowledge can help improve the predictive performance of the model. As illustrated in Figure 4, LLM first provides a detailed explanation for ‘special’ words in the text, largely enriching the linguistic semantics. Especially for abbreviation, slang, and hashtag, LLMs directly return crucial definitions for easy understanding of the whole context while KB-NER and MoRe only provide related or vague context needing further consideration. For multimodal CoT knowledge, our method can distinguish the relatedness of text-image pair. For example, in (b) and (d), the images do not have a corresponding entity in the text. However, LLMs can also interpret the long-range image-text relation.

Furthermore, compared with the baseline model and MoRe, our method can easily recognize hard entities like ‘North and South Korea’ in (b) and ‘fikile_nhleko’ in (c). Besides, the correct prediction with and without CoT knowledge indicates that the student model inherits the reasoning ability of LLMs through conditional prompt distillation.

Limitations and Boarder Impacts. We notice that the quality of generated CoT knowledge heavily depends on the well-posed LLM that understands the prompt and example comprehensively. We believe next-generation multimodal large models, such as GPT5, will make the CoT knowledge more reliable.

Our method is simple yet effective to distill the general task-agonist knowledge from LLMs to a small model. It can be easily expanded to other tasks which need sophisticated reasoning.