Image Matters: A New Dataset and Empirical Study for Multimodal Hyperbole Detection

Not surprisingly, earlier studies on hyperbole focus more on its linguistic features. Mora (2009) made a detailed anylasis of the semantic features of hyperbole and first introduced a taxonomy which generally categorizes all exaggerations along two dimensions: quantity(overstating the quantitaive or objective property of something, such as time, mass, etc.) and quality(overstating the qualitative or subjective property of something, such as feeling, emotion, etc.). These two types can be subcategorized into smaller ones, but that is beyond our discussion.

Some multimodal analysis work has also been conducted. Ferré (2014) manually analyzed how hyperbole is expressed with text, video and audio modalities. He focused on gestures, facial expressions, tones, etc. His work directly pointed out that other modalities can help reveal hyperbole contained in texts or words.

For the automatic detection of hyperbole, Troiano et al. (2018) created a hyperbole detection dataset HYPO, which contains 709 hyperbolic sentences and their corresponding non-hyperbolic ones. They applied traditional machine learning methods and proposed some hand-crafted features (named QQ) to address this task.

Biddle et al. (2021) followed Troiano’s work. They introduced triplet sampling and used triplet loss for forcing their model to differentiate hyperbolic and non-hyperbolic sentences. They used BERT as the backbone text encoder instead of pre-trained word vectors(used by Troiano’s work).

As for Chinese hyperbole detection, Kong et al. (2020) followed Troiano’s idea and created the first Chinese hyperbole detection dataset HYPO-cn. They analyzed statistical features of the sentences in the dataset thoroughly. They also manually analyzed strategies humans used to express hyperbole. They compared several traditional and deep-learning methods on this dataset. They reached the conclusion that deep-learning methods outperform traditional methods on this task.

However, to the best of our knowledge, there hasn’t been any study on multimodal hyperbole detection.

Multimodal sarcasm detection has become a hot topic in recent years. Cai et al. (2019) first proposed a dataset built on Twitter, which is still widely used by researchers. They also proposed a hierarchical fusion model to address this task. Xu et al. (2020) followed this work and proposed D&R Net. Liang et al. (2021) introduced in-modal and cross-modal graphs(InCrossMGs) to leverage the ironic expressions from both in- and cross-modal perspectives by extracting the dependency relations of fine-grained phrases or patches within the distinct modality and across multiple modalities. Liu et al. (2022) leverages graph attention networks and image captions as knowledge enhancement.

On the other side, Castro et al. (2019) focused on video and text modalities and proposed a sarcasm detection dataset named MUStARD. MUStARD is compiled from popular TV shows and consists of audiovisual utterances annotated with sarcasm labels along with its corresponding context. Wu et al. (2021) continued on this work and proposed IWAN to model the word-level incongruity between modalities via a scoring mechanism.

Multimodal sentiment analysis is also a widely-discussed topic. Wang et al. (2020) proposed an end-to-end fusion method based on transformers. Yang et al. (2021) introduced multi-channel graph neural networks. Xue et al. (2022), Du et al. (2022) focused on attention-based methods such as gated attention. Inspirations can be gained from these methods.

Multimodal hyperbole detection bears both similarities and differences with the tasks mentioned above. It requires the understanding and interaction of both modalities. However, the relation between the two modalities can be rather complicated. This will be discussed in Section 4.

To avoid ambiguity, we need to give a clear definition to hyperbole. Based on the definition given by Merriam Webster ⁴⁴4https://www.merriam-webster.com/dictionary/exaggeration and Troiano et al. (2018), we here define hyperbole as “an expression (words, images, etc) that goes significantly beyond fact or common sense but not taken as lies”. A post is considered hyperbolic if and only if it contains hyperbolic expression.

For example, the post “I’m dying of heat” is hyperbolic because normally most people do not easily die of heat. But if there is a post “5 workers died of extreme heat” with real pictures, the post is certainly not considered hyperbolic. However, if a post says “100 billion workers died of extreme heat”, it is then considered neither hyperbole or non-hyperbole. Instead, it is probably a lie. As for images, Figure 1b provides a perfect example. Since people do not melt however hot it is, a melting face representing the unbearably hot weather is typical hyperbole.

Weibo is a popular Chinese social medium. We automatically crawl about 10000 posts from Weibo with 5 keywords: 堵车(traffic jam), 风景(scenery), 天气(weather), 心情(mood), 衣服(clothing). Each keyword contains roughly same number of posts (about 2000). We choose the five keywords because they are generally disjoint and close to daily lives. We carefully set criteria so that each post crawled is original and with at least one image.

After collecting these posts, we first clean up the data. The urls in the texts are replaced with a specific token <url>. Some emojis are actually hyperlinks to images, so they are replaced with another <emoji> token. Hashtags (words or phrases led by #) are not removed because there is no obvious distinction between hashtags in hyperbolic and non-hyperbolic posts according to our observation. Some posts have more than one image, so we randomly select one image and drop the others. Since this is done before any human work, there is no concern that it will introduce any inconsistency or bias. Then we conduct a rough selection. Posts with too long or unreadable texts or unclear pictures are manually dropped to save some annotation work. (Here we define “too long” as posts longer than 200 characters.) We set the limitation because longer posts are harder for annotation. Annotators may miss the hyperbolic part of a too long post thus leading to unreliable results. We believe this setting makes our dataset cleaner. Advertisement posts that appear repeatedly are also dropped to avoid polluting the dataset.

We employ several human annotators to annotate these posts as hyperbole or non-hyperbole. All of them are Chinese college students. They are required to follow an annotation specification given by us. We first ask them to label 200 randomly chosen posts, and select two of them whose annotation results fit the specification best. We communicate with them about their mistakes to help them understand our definition of hyperbole better. They then annotate the rest posts. We manually check each inconsistent case and discuss with them to determine whether each is hyperbole.

Since there are much more non-hyperbolic posts, we randomly choose from them and create the final dataset in which #hyperbole:#non-hyperbole is approximately 50%:50%. Some basic statistics of the final dataset are listed in Table 1.

As discussed in Section 4, images mostly serve as assistants rather than express hyperbole themselves. So it is natural to ask: is image modality more useful than misleading in hyperbole detection? If the answer is yes, which way is more suitable for utilizing image? For example, is deep modality fusion more helpful than shallow ones?

We describe our methods from shallower fusion to deeper fusion. First, we use two unimodal encoders: BERT (Devlin et al., 2019) and ResNet50 (He et al., 2016), to encode text and image modality individually. Specifically, we use BERT that is pre-trained on Chinese(Cui et al., 2020). We fine-tune them to achieve better performance.

After unimodal encoding, we apply different fusion methods on this task. Denote $f_{text}$ as the feature vector of text (outputted by BERT) and $f_{image}$ as the feature vector of image (outputted by ResNet), the most trivial idea is simply concatenating $f_{text}$ and $f_{image}$ (We name this method concat). A slight improvement on this idea is implementing a gating method to select more significant features. We name this method gate. To be more detailed, consider two feature vectors $f_{image}$ and $f_{text}$, the gating method we use can be summarized as:

where $W_{text},W_{image},b_{text},b_{image}$ are learnable parameters, $\sigma$ is the Sigmoid function ($\sigma(x)=\frac{1}{1+e^{-x}}$).

Attention is a widely used method which can utilize fine-grained features. We utilize both text-level and token-level features outputted by BERT as well as both image-level and patch-level features outputted by ResNet. We implement a multi-head attention mechanism to extract cross-modality fine-grained features. To be more detailed, we use the fine-grained features (i.e., token-level or patch-level features) of one modality as $Q$, the fine-grained features of the other modality as $K$ and $V$. Thus we obtain two cross-modality fine-grained feature representations. We use another two multi-head attention modules to further aggregate these fine-grained feature representations respectively (using image- or text-level feature representation as $Q$ and the corresponding cross-modality fine-grained feature representation as $K$ and $V$) and get the encoding vector of either modality. We then concatenate these two encodings and implement a gating mechanism similarly (We name this method attn-gate.) The main architecture of this method is shown in Figure 4.

After fusing with either of the above methods, we feed the feature vector into several fully connected layers to finish the final classification. We use cross-entropy as loss function and Adam (Kingma and Ba, 2014) as optimizer. All these models share the same hyper-parameters. We believe the above methods are enough to answer our question on this relatively small dataset.

We report 10-fold cross validation results of the models to ensure the reliability of our results. Our hyper-parameters are listed in Table 4.The average results are shown in Table 5.

Some conclusions can be drawn from the results. First, with single modality, text-only model (BERT) performs much better than image-only model (ResNet). This is compatible with our analysis that most images are not hyperbolic themselves. Therefore although we’ve already discussed the importance of image modality for humans to determine hyperbole, it’s still questionable that whether image modality is more useful rather than misleading under this circumstance.

Our following result proves that image modality is indeed more useful rather than misleading. Though with text modality only can achieve a fine result, introducing image modality, even simply giving it the same significance as text modality (concat), still apparently increases the performance. Comparing with BERT, this improvement is statistically significant. This result clearly shows that introducing image modality together with text modality is useful for determining hyperbole.

Further, gating mechanism may be useful for increasing model performance. This insight can be drawn from our analysis of the dataset. Since under most circumstances, there are probably some features serving as background information, we should give these features less focus. As can be seen from the results, even with trivial gating mechanism, the result is slightly improved compared with simply concatenating the two modalities’ outputs. However, this trivial gating method still has its own problems. First, using BERT only can achieve relatively fine results, which shows that text modality contains more information for detecting hyperbole. Thus we observe that the trivial gating method tends to give text modality a higher priority. This systematic bias leads to insufficient utility of image modality.

Another problem is that gate, as well as concat, is unable to model the fine-grained relationship between text and image. Figure 5a gives an example. All other methods except attn-gate failed on this case. Since they just simply use high-level features of two modalities, they are unable to notice the connection between certain tokens(Winter) and a certain feature(the temperature in the image). In contrast, attn-gate is able to extract fine-grained features between two modalities. In fact, comparing with the trivial concat method, attn-gate is statistically significantly better. Since we performed 10-fold cross validation, we believe that this improvement clearly shows that even though most hyperbole are not directly expresses through images, introducing images, especially using deep fusion, is still helpful in recognizing hyperbole.

There are also cases where all these methods failed. Figure 5b is a typical failure case. The reason is that these models do not know even under such hot weather, people still need to wear clothes. The expression “there’s no need wearing any clothes” is an overstatement of how hot it is. Since models do not have any common sense knowledge, they are unable to recognize this kind of hyperbole, which leads to the failure.

In summary, our answer to this question (RQ1) is that: images are more useful rather than misleading. With gating and attention mechanism, we can more accurately utilize fine-grained fused features between two modalities, which can help identify hyperboles of case-II and case-III. However, without common sense, the model can never fully understand hyperbole. To further improve the performance, the key is introducing common sense into the models.

Pre-trained multimodal models are proved effective in many downstream tasks including zero-shot learning, image-text retrieval, VQA, etc. However, when speaking of multimodal classification tasks, few discussions are made before. In this subsection, we apply two pre-trained models CLIP (Radford et al., 2021) and BriVL (Huo et al., 2021) on this downstream task to evaluate their performance. CLIP (Radford et al., 2021) is a popular multimodal encoder and BriVL (Huo et al., 2021) is a similar one that is pre-trained on Chinese. Besides, BriVL claims to use a more advanced algorithm(inspired by MoCo (He et al., 2020)) to obtain better encodings of both modalities.

We still report 10-fold cross-validation results. Specifically, we use CLIP that is pre-trained on Chinese texts (Yang et al., 2022). For BriVL, we directly use the open-source code and tools⁶⁶6https://github.com/BAAI-WuDao/BriVL.

We try two ways to use these pre-trained models. One is giving prompts (prompt) to the text (We add prompts “ 这是夸张 ”(This is hyperbole.) and “ 这不是夸张 ”(This is not hyperbole) at the beginning of each text. So we get two texts with different prompts and same content.). To be more detailed, for each case we encode one image and the two texts with different prompts, and try to maximize the cosine similarity between the image and the text with correct prompt. This is a common method used in image classification tasks.

Another way is using these models simply as encoders and treat their output as feature vectors. Then we concatenate them (concat) or use gating methods (gate) just like what we have done above. In this way no prompts are added to texts. Both the two ways are seldom discussed in similar tasks like multimodal sarcasm detection previously. Details of both ways are shown in figure 6.

We fine-tune theses models to achieve better results. However, the result turns out to be very disappointing. As can be seen in Table 6, they perform much worse than most methods mentioned in Section 4.

We analyze why the results are much worse. Radford et al. (2021) has already pointed out that CLIP performs badly on classifying abstract concepts like determining whether a scene is normal or not. Hyperbole detection is a typical task involving abstract concept, so it is not surprising that prompt does not work well on this task. What’s more, since this kind of task involves classifying image and text together rather than image itself, the prompt method is even more unsuitable. We believe the same reason works for BriVL.

Besides, the text in this kind of task is usually not a simple description of the image. It is a common case that there is no strong semantic correlation between corresponding text and image, which is quite different from the dataset CLIP is trained on. As is discussed in Radford et al. (2021) and Huo et al. (2021), CLIP performs badly under this circumstance. Huo et al. (2021) claims to have tried to avoid assuming this strong correlation in BriVL. As can be seen, it turns out to be effective to some extent when compared with CLIP. However, some texts in this task have very little or even no explicit semantic correlation with corresponding images. We believe that this bias probably leads to poor encoding of either modality and causes the bad performance.

In this part, we briefly evaluate the methods mentioned in Section 4 on cross-domain hyperbole detection. As is shown in Section 3.2, the posts are of five different keywords and their contents are generally disjoint. We use posts from three keywords for training, one for validating and one for testing. For each method, we repeat this procedure five times to test on all five keywords and report average performance. The results are shown in Table 7. Since pre-trained multimodal models perform badly, we do not discuss them in this part.

As can be seen, all methods perform worse than in Section 4, which meets our expectation because cross-domain hyperbole detection is certainly a tougher task. The results are not that terrible, which shows the generalization ability of the methods proposed above. Introducing image modality still apparently increases the performance as can be seen in concat. However, we are surprised to find that gating and attention mechanism do not bring higher performance. We take two examples (Figure 7) from the keyword 堵车(traffic jam)(traffic jam) to illustrate the reason.

As can be seen in Figure 7a, the sentence contains an expression won’t be stuck in a traffic jam which is seemingly too absolute here. However, in this case, it is probably a fact that bikes will not be stuck in traffic jams, so it is not hyperbole. When trained on other keywords, the models will not have such knowledge (“bikes don’t get stuck in traffic jams“), so they naturally fail on this case. This explains why all methods perform worse on this task. Similarly, the screenshot of a navigating software only appears in posts with the certain keyword 堵车(traffic jam), so the models have no idea of what it is. Therefore, models requiring deep interaction between text modality and image modality may be confused more seriously. We believe that this systematic inconsistency is the reason why deep fusion methods perform much worse on this task. And again, common sense knowledge is still helpful in easing this problem.