StyleM: Stylized Metrics for Image Captioning Built with Contrastive $N$-grams

We build our metric based on $n$-gram comparisons, which means we will first calculate a score for each $n$-gram in a caption that measures how aligned it is with a given style. Generally, we would like score an $n$-gram high if it only appears in the dataset associated with current style and if it appears frequently in this dataset. That is to say, if an $n$-gram has high contrastive $n$-gram score on style $p$, we would think this $n$-gram contains good characteristics for this style. Suppose we have a $n$-gram $t$, $n\in{\left\{1,2,3,4\right\}}$, and we would like to calculate how close this $t$ with a given style $p$. We calculate the score $s_{p,n}$ for this $n$-gram $t$ under style $p$ as follows:

	$\displaystyle f_{p,n}=|\left\{d\in D_{p}:t\in d\right\}|$		(1)
	$\displaystyle ECDF(f_{p,n})=\frac{1}{\|F_{p,n}\|}\sum_{x_{i}\in F_{p,n}}\mathds{1}_{x_{i}<=f_{p,n}}$		(2)
	$\displaystyle occur\_num=|\left\{s\in S:t\in d,d\in D_{s}\right\}|$		(3)
	$\displaystyle CNG_{p,n}=\frac{1}{\|S\|}\sum_{q\in S,q!=p}\frac{ECDF(f_{p,n})-ECDF(f_{q,i})}{occur\_num}$		(4)

Equation 1 calculates the document frequency for $t$ in $D_{p}$ where $t$ is a $n$-gram term and $D_{p}$ is the corpus that includes all sentences related to the style $p$. We intend to compare the $t$’s representations across different styles to see whether $t$ is salient in current style $p$. Since document frequency like $f_{p,n}$ is heavily influenced by the number of documents in $D_{p}$, we need to convert $f_{p,n}$ into probability. To do so, we calculate the empirical distribution for all $n$-grams under style $p$. With this, we can calculate $f_{p,n}$’s probability in Equation 2, where $ECDF(f_{p,n})$ is the empirical distribution function which converts a document frequency $f_{p,n}$ to a probability under empirical distribution. we call this the ecdf score for short in the discussion below. $F_{p,n}$ is a list that contains all $n$-gram document frequencies under style $p$.

To check how frequently $t$ occurs in documents of different styles, we calculate the number of stylized corpora $D_{s}$ where t has occurred (Equation 3). Here $S$ is the style set, which contains all styles. We use $occur\_num$ from Equation 3 to penalize the score of t in Equation 4. If t has occurred in multiple corpora, then it is not deemed significant enough to represent the current style. To see how much better $t$ aligns with style $p$ than another style $q$, we finally calculate the contrastive $n$-gram score for $t$ under style $p$ by calculating the average difference between the ecdf score under style $p$ and all other styles $q$, where $q$ is in the style set $S$ but $q$ is not equal to $p$ (Equation 4).
Bounds Analysis The best case scenario of $CNG_{p,n}$ score for a $n$-gram $t$ is when $t$ only occurs in $D_{p}$ but not in any other $D_{q}$ and occurs in every sentence $d\in D_{p}$. That is to say $t$ is only associated with style $p$ and is also associated with every sentence in $D_{p}$. With this, we can calculate the upper bound for $CNG_{p,n}$, where $CNG_{p,n}=\frac{\|S\|-1}{\|S\|}$. The worst case scenario is when $t$ never occurs in $D_{p}$ but occurs in all other datasets $D_{q},q!=p$ and appears in every sentence $d\in D_{q}$. This results in the lower bound for $CNG_{p,n}$, where $CNG_{p,n}=-\frac{1}{\|S\|}$. Notably, the stop words are naturally de-emphasized by $occur\_num$ and subtraction if they are nearly evenly distributed between different styles through Equation 4.

After we calculate the contrastive $n$-gram score for all $n$-grams in the dataset for a given style $p$, we can calculate how much a caption $c$ aligns with a style $p$. Since this score is designed to assess a single caption without any reference caption, we call it the OnlyStyle score. Specifically, for any caption $c$, we first extract all the $n$-grams from it, where $n\in\left\{1,2,3,4\right\}$. With these $n$-grams, we can compute the OnlyStyle score as follows:

	$\displaystyle OnlyStyle(c,p)=\frac{1}{4}\sum^{4}_{n=1}OnlyStyle_{n}(c,p)$		(5)
	$\displaystyle OnlyStyle_{n}(c,p)=\frac{1}{\|M\|}\sum_{i\in M}CNG_{p,n}^{i}$		(6)

In Equations 6, we calculate the average CNG score across all $n$-grams from caption $c$, where $CNG_{p,n}^{i}$ is the CNG score for the $i$th $n$-gram under style $p$ and M is the set of all the $n$-grams in caption $c$. We get the OnlyStyle score using Equation 5 for $c$ under style $p$ by averaging over $n$ of $OnlyStyle_{n}(c,p)$.
Bounds Analysis OnlyStyle displays the global view of a sentence’s association with a style $p$ by averaging across different $CNG$ scores, so the bounds stay the same as for $CNG$. Specifically, suppose we could always get the highest $CNG_{p,n}$ for every $n$-gram in $M$, this will give us the highest OnlyStyle score for $c$ under $p$, which is $\frac{\|S\|-1}{\|S\|}$. On the contrary, if we always get the lowest $CNG_{p,n}$ for every $n$-gram in $M$, this will result in the lowest OnlyStyle score, which is $-\frac{1}{\|S\|}$.

While the ability to evaluate a caption without a reference sentence has its benefits, the presence of a reference sentence allows us to better evaluate the impact that individual words have. This is especially useful for stylized image captioning because individual words can have a large impact on the style that a sentence exhibits. To that end, we propose StyleCIDEr, a metric meant to evaluate how well a generated caption aligns with a reference caption, which emphasis placed on stylized words. This metric uses CIDEr as a kernel function. To better understand this, we will first outline how to calculate CIDEr. Suppose that we have two captions, $c_{i},c_{j}$:

	$\displaystyle CIDEr(c_{i},c_{j})=\frac{1}{4}\sum^{4}_{n=1}CIDEr_{n}(c_{i},c_{j})$		(7)
	$\displaystyle CIDEr_{n}(c_{i},c_{j})=\frac{g^{n}(c_{i})\cdot g^{n}(c_{j})}{\|g^{n}(c_{i})\|\cdot\|g^{n}(c_{j})\|}$		(8)

where $g^{n}(c_{i})$ is a vector where each element is formed by a $n$-gram score calculated by TF-IDF. TF-IDF is good at scoring salient $n$-grams for current captions, but it does not necessarily focus on stylized words or $n$-grams. As a result, we replace this TF-IDF score with our CNG score. This means that each element in $g^{n}(c_{i})$ will be a score focusing on a given style, like $p$. Then we calculate using Equations 7 and 8, and we can compare $c_{i},c_{j}$ based on their stylized elements.
Bounds Analysis Since CIDEr score is bounded from (0,1), StyleCIDEr has the same range. In the best case scenario, when evaluating a caption that is the exact same as the reference caption, the StyleCIDEr would be 1. If there are no overlapping $n$-grams between the generated caption and the reference caption caption, the StyleCIDEr score would be 0. To achieve a score of 1, StyleCIDEr does not require every $n$-gram in the generated caption to be the same as in the reference caption. As long as $n$-grams related to current style overlap between the generated caption and the reference caption, the StyleCIDEr score can be 1. Also, if the generated caption and reference caption have several overlapping $n$-grams but none of them are related to current style, the StyleCIDEr can be 0.

Since PERSONALITY-CAPTIONS and FlickrStyle10K are frequently used in stylized image captioning work [17, 12, 6, 4, 9, 25], we decide to validate our proposed metrics on these two datasets. The PERSONALITY-CAPTIONS dataset was released in 2019 [17]. It contains ground truth captions that were collected via human crowdsourcing. During data collection, humans are required to create engaging captions based on the image context and the given personalities so that the caption accurately embodies the given personality. Each data entry within this dataset is represented as a triple containing an image, personality trait, and caption. The images are selected from YFCC100M dataset [18]. In total, 241,858 captions are included in this dataset. Each caption is associated with one of the 215 personality traits selected from a list of 638 traits [8]. Even though the number of potential styles in the dataset makes captions more diverse, some of the styles in this dataset could be difficult for a human to distinguish (e.g. “Happy” versus “Cheerful”).

In this work, we use the accessible data from this dataset which contains 186698 examples in the training set, 4993 examples in the validation set, and 9981 examples in the test set. The PERSONALITY-CAPTIONS dataset is large, which enables our contrastive $n$-gram metric to learn current style characteristics in a broader range. This naturally leads to the question of how our metric would perform with significantly fewer styles. To answer this question, we explore FlickerStyle10K.

FlickrStyle10K contains two styles: Humorous and Romantic. The ground truth stylized captions included in this dataset are based on factual image captions (descriptions without any style) that were modified so that they exhibit the given style. Thus, humorous captions and romantic captions are often very similar when describing the image, but different when using words that exhibit the desired style. Since only 7000 images are publicly available, we will use this FlickrStyle7K dataset in our experiment. When training a stylized captioning model, we use the protocol outlined in [9, 25, 12] did. This involves first randomly select 6,000 images as the training data and using the remaining 1000 images as testing data. We further split 10% data from the training set into a validation set.

UPDOWN[17], Multi-UPDOWN[12] and Visual Language Pretraining (VLP) on transformers[23] have shown good performance captioning tasks (stylized or otherwise) when evaluated using common automated metrics (BLEU, CIDEr, etc.). As such, we evaluate these three types of models using our proposed metrics.

For the UPDOWN captioning model, we use the parameters outlined in [17]. We use pretrained ResNext spatial (7*7*2048) features and mean-pooled features (2048), along with style as one-hot vector as inputs to this network. For the Multi-UPDOWN as captioning model, we use the process outlined in [12] and use ResNext spatial (7*7*2048) features and mean-pooled features (2048), 5 dense caption, along with style as a one-hot vector as inputs into its network. For SVinVL as a captioning model, we fine-tune VinVL on the PERSONALITY-CAPTION dataset by replacing object tags with 5 dense captions and styles (in text form) to learn the connection between image captions with ResNext spatial features (7*7*2048), dense captions, and styles. For each caption model, we also train a model with the same settings but without style inputs. So, in total, we train 6 models for each dataset and they are named: (1) UPDOWN; (2) UPDOWN_NoStyle; (3) Multi-UPDOWN; (4) Multi-UPDOWN_NoStyle; (5) SVinVL; (6) SVinVL_NoStyle.

For (1)-(4), we use entropy as loss function and Adam optimization with an initial learning rate of 5e-4. The learning rate decays every 5 epochs. For (5)-(6), we follow the fine-tuning steps of VinVL and use masked token loss and AdamW optimaztion with an initial learning rate of 3e-5. A linear scheduler is used for decaying the learning rate. In total, we train 30 epochs when using the PERSONALITY-CAPTIONS dataset [17] with a batch size of 128 and evaluate the model every 3000 iterations. We train for 100 epochs when using the FlickrStyle10K dataset [6] following the process outlined in [17, 12, 13].

We will evaluate our proposed metric under three cases. First, we directly evaluate the ground truth captions in the dataset since each of them is associated with a style. Second, we evaluate different caption models and compare them with models that do not contain any style inputs. Third, we perform a human study and ask them to evaluate some generated captions from a model. We then compare the human evaluation results with our proposed metric results. More details are included below.

To evaluate the consistency between human judgments and our proposed metrics, we perform a human evaluation by asking humans to rank some sample generations from one of our models based on how well it represents a given style. For this evaluation, we focus solely on OnlyStyle since StyleCIDEr is based on the CIDEr metric, which has already been shown to align with human judgment.
Human Study Setup For this experiment, we chose the following six styles from the PERSONALITY-CAPTIONS dataset: Abrasive (Annoying, Irritating), Angry, Curious, Fearful, Gloomy, Happy. We chose these styles because they closely resembled the six basic emotions cited in Psychology literature [5], and we felt that this would make it easier for users to identify them.

For each of these styles, we sample 3 captions generated by the Multi-UPDOWN model on test data. After approved by Institutional Review Boards, we then asked crowdsourced users from Prolific to rank three generated sentences according to how well they are associated with a given style. Users provided answers in the range of 1-3 with 1 indicating the worst association, and 3 indicating the best association. Users were asked to perform this task 6 times, once for each style in the evaluation. In total, we received 96 responses to this task.

Note that this study and all the materials associated with it were reviewed and approved by our Institutional Review Board to ensure that subjects were exposed to no more than a minimal risk during their participation.

Consistency with Human Judgment

We pick the most common ranking of the three sentences for each style from the 96 users as human’s judgment. We then compare these rank scores with the OnlyStyle scores on the sample generations. We use Pearson and Spearman correlation coefficients to quantify the consistency between our proposed OnlyStyle metric and human judgments. We report both scores for the six styles we examined in Table  4.

As we see in Table 1, all of the scores are above 90%. This means in most cases, Equation 9 and Equation 10 are satisfied. This matches our expectations, which means that OnlyStyle can differentiate correct and incorrect styles, and StyleCIDEr can measure the accuracy of stylized word generation. However, we should point out that all ratings did not achieve 100% performance. This is because some sentences may not be representative of the style that they are associated with in the dataset. As an example, it is possible that some sentences appear in “Happy” and also appear in “Cheerful”. Despite this, some of these sentences might have a higher score in “Cheerful” than “Happy”. So, when we evaluate $OnlyStyle(s,``Happy")>OnlyStyle(s,``Cheerful")$, Equation 9 is not satisfied. In addition, when we evaluate $StyleCIDEr(s,s_{Happy})>StyleCIDEr(s,s_{Cheerful)})$, it is not satisfied either.

With StyleCIDEr, it is also possible that the two captions used for evaluation do not have overlapping words, regardless of if they are drawn from the same style or different styles. In this situation, both the within-style StyleCIDEr score and between-style StyleCIDEr score would be zero, which violates Equation 10.

With Style Versus Without. From Tables 2 and 3, we can see that models with style modality as inputs have higher OnlyStyle and StyleCIDEr scores across all captioning models. This provides support to our claim that both StyleCider and OnlyStyle can measure whether a model can effectively generate output that aligns with a given style.
Style Metric vs Accuracy Metric. From Table 2, we can clearly see that when evaluated using only BLEU and CIDEr, UPDOWN_NoStyle and SVinVL_NoStyle perform better than UPDOWN and SVinVL. One reason for this may be because the stylized captions in FlickrStyle7K are created by modifying the factual captions. In this situation, models without a style vector as an input would generate the same output for both romantic and humorous styles, which strongly resembles the original factual caption. These outputs would score highly when evaluated using CIDEr and BLEU.

Models with style included as an input will produce different sentences to better exemplify the desired style. As a result, these generated captions may have fewer overlapping words than the reference sentence as they potentially transfer stylized words from other examples, which in turn results in lower BLEU and CIDEr scores. This shows that we can not simply rely on these metrics in this situation.
Caption Model performance If we only look at models with style inputs, from Table 2, we can see that the UPDOWN Model achieves the highest OnlyStyle score for PERSONALITY-CAPTIONS and ranks second on FlickrStyle7K, which means that the UPDOWN Model are good at generating more stylized sentences. However, the UPDOWN model has the worst score on StyleCIDEr. This means that while the UPDOWN model can produce stylized words that align with the desired style, they may not accurately describe the image in question.

Multi-UPDOWN has the best StyleCIDEr score as well as excellent performance on BLEU and CIDEr scores. This is likely because it is directly trained address limitations with UPDOWN where it would overly focus on building connections between styles and generated captions and miss information in the visual space. The Multi-UPDOWN model ranks second on PERSONALITY-CAPTIONS and is the worst performing model on FlickrStyle7K on OnlyStyle, which means that the gain in visual understanding likely came at the expense of generating stylized output.

The SVinVL model has the worst accuracy performance on both datasets. Its StyleCIDEr rating ranks as the best on FlickrStyle7K and the second on PERSONALITY-CAPTION. Its OnlyStle rating ranks the best on FlickrStyle7K and worst on PERSONALITY-CAPTION. This means that the SVinVL Model tends to connect style with caption generations on FlickrStyle7K which has 2 styles, and struggles to make correct connections on the PERSONALITY-CAPTIONS dataset where size scales up.

In Table 4, we can see Pearson and Spearman correlation coefficients are very high with style Abrasive (Annoying, Irritating), Gloomy and Angry, which indicates that rankings determined using the OnlyStyle metric are consistent with rankings derived based on human judgment. However, Pearson and Spearman scores are low for Happy, Curious and Fearful. Because of this finding, we decided to further investigate this discrepancy.

Recall that for each style, we asked users to rank three generated sentences. By investigating how users ranked these sentences, we hope to uncover the source of any ranking discrepancy. We found that for the “Happy” style, people ranked the sentence “what a beautiful day” higher than “i love rugby”, which means they think the first sentence is more associated with “Happy” than the latter one. OnlyStyle scores these sentences in the opposite way. After further investigation, we discovered that the phrase, “what a beautiful day,” appears in multiple styles in the dataset (e.g. “Warm”, “Cheerful”, and “Enthusiastic”) but only occurs in “Happy” as a part of a larger sentence. The phrase, “i love rugby,” only occurs in “Happy” as a complete sentence.

For “Curious”, people rank “i wonder what these people are thinking about” higher than “i wonder how many lamps there are” and OnlyStyle scores the opposite. The first sentence never appears in “Curious” and only appears in “Freethinking” in the dataset. The second sentence only appears in “Curious”.

For “Fearful”, people rank the caption, “i hope he doesn t fall,” higher than the caption, “i hope that statue doesn t fall”. The first sentences appears in “Fearful” and also appears in “Sympathetic” and “Gloomy” in the dataset. The second sentence only appears in “Fearful”.

Even though people’s judgments are very reasonable, OnlyStyle gives higher scores to phrases that appear frequently in one specific style, with the assumption that those are more indicative of a given style. To investigate a little further, we calculate the OnlyStyle scores for these sentences on all 215 styles in the PERSONALITY-CAPTIONS dataset. We then calculate the retrieval ranks for “Happy”, “Curious”, and “Fearful” for their respective sentences. We found that for each of the disagreeing sentences, the correct style was always in the top 10% of retrieved styles, indicating that these styles can be difficult to distinguish from other top styles. It also means that even though these sentences are ranked differently by humans and the OnlyStyle metric, they are all very close the given styles. Thus, it would be hard for human to differentiate them and rank them as well. Overall, we think all these disagreements could be resolved with better datasets.

We mentioned previously that basing an evaluation metric on TF-IDF can be problematic. Some words, such as “not” or “happy” can appear across all the styles, which would be scored as 0. We hypothesized that our use of a contrastive $n$-gram score could address some of these issues. We evaluated this by sampling several words including“not” and “happy” under six styles and calculating their contrastive 1-gram score (shown in Figure 2).

We can see that “not” generally scores very low due to its occurrence in every style in the dataset; however, it scores negatively in ”Happy” and ”Curious”, which we view as a positive result. The 1-gram, ”happy,” scores negatively in “Abrasive (Annoying, Irritating)” and “Fearful” and scores highest in the style, ”Happy”. Figure 2 also shows CNG deprecates word like “the” and doesn’t neglect noun’s contribution to style, like “car”. These all shows that using the contrastive $n$-gram score allows us to get a more nuanced understanding of how $n$-grams contribute to different styles.