Can Audio Captions Be Evaluated with Image Caption Metrics?

To generate candidate captions for evaluation, we involve multifaceted audio captioning systems: 1) Nearest neighbor (NN) retrieval system. For a test audio, we retrieve its most similar training audio and take its annotation as a prediction. The similarity between two audio clips are measured by the cosine similarity of their embeddings, extracted by a pre-trained CNN [18]. 2) Fully Connected (FC) input system. An architecture similar to [3], where the input audio is first transformed to a vector and then decoded into a caption. 3) Attention (ATT) input system. The same architecture as [5], where the input audio is first transformed into an embedding sequence and then decoded into a caption using temporal attention mechanism. 4) Reinforcement Learning (RL) system. The same architecture as ATT system while the model is further fine-tuned using reinforcement learning [5], achieving the second place in the recent DCASE challenge evaluated by CIDEr and SPICE.

FC, ATT and RL systems are all sequence-to-sequence encoder-decoder frameworks. They all use a 10-layer CNN as the encoder and a single-layer GRU as the decoder. The detail structure can be found in [19]. To increase the diversity of generated captions, for all systems except NN, we employ various decoding strategies, including greedy decoding, beam search, diverse beam search [20]. In beam search, different temperatures are utilized.

Based on captions generated by the aforementioned systems, we build the audio captioning evaluation dataset by collecting human judgments on captions. We build our datasets based on two benchmark audio caption datasets: AudioCaps [4] and Clotho [2]. There are two protocols to collect human judgments in image captioning: absolute rating and pairwise comparison. We first make a preliminary exploration on absolute rating protocol. Given an audio clip and a caption, raters are asked to score the caption from 1 to 4 considering both its relevance to the audio and its fluency. We randomly sample 100 audio-caption pairs. Each audio-caption pair is rated by four different human raters. However, Fleiss Kappa score [21] for different raters is only 0.18 on Clotho and 0.23 on AudioCaps, indicating poor inter-annotator agreement. Annotators also report difficulty in rating the caption.

Therefore, the second protocol, pairwise comparison, is adopted. Given an audio clip and a pair of candidate captions, four raters are asked to choose which candidate describes the audio better in terms of description accuracy and fluency. Raters are allowed to choose “I’m not sure” if they cannot distinguish which candidate is better. Following [11], we generate four pair groups, namely human-human correct (HC), human-human incorrect (HI), human-machine (HM) and machine-machine (MM). HC contains two human annotations describing the same audio. HI also contains two human annotations but one describes another randomly-picked audio. HM is formed by a human annotation and a machine generated caption for the same audio. MM is composed of two machine generated captions describing the same audio. On both Clotho and AudioCaps, we randomly pick 250 audio clips and generate the four kinds of pairs. For each audio clip, we form one HC pair, one HI pair, one HM pair and several MM pairs since the evaluation metrics are usually applied to compare captions generated by different machine systems. The number of MM pairs for each audio clip range from one to four.

To avoid indistinguishable cases between similar pairs, we adopt a filtering strategy for pair generation. We first generate all candidate pairs and calculate the similarity of each pair using Sentence-BERT embedding. Then we exclude candidate pairs with a similarity higher than 0.9. We randomly sample pairs from the remaining captions to obtain the final generated pairs for comparison. For HC and HM, we randomly sample the pairs. For MM, we randomly sample four pairs if there are more than four candidate pairs, otherwise we use all the pairs left. In case of no pairs left after filtering, we select the pair with the lowest similarity. In this way, we finally obtain human judgments on 1,750 pairs on Clotho and 1,671 pairs on AudioCaps. Higher inter-annotator agreement is achieved on both datasets, indicated by a Fleiss Kappa score of 0.48 and 0.33 respectively. We will refer to them as AudioCaps-Eval and Clotho-Eval. To our best knowledge, this is the first dataset of human judgments on the quality of audio captions. We evaluate the effectiveness of different metrics using this dataset.

We conducted experiments on the two established datasets to compare different metrics’ effectiveness on deciding a better caption out of two candidate sentences. For each pair of candidates, we compute all the metrics for two sentences and consider the one with a higher score as better according to the metric. A decision is considered correct if it agrees with human judgment. We calculate per-category accuracy by dividing correct predictions with total predictions within the categories (HC, HI, HM and MM), and merge them as a total score with micro average. Note that we exclude samples with equal number of opposite human judgments (excluding “not sure”) so that the chosen samples can be considered distinguishable.

Since we may also include reference sentences in comparison, we only choose 4 out of 5 sentences from original references in the calculation of metrics. For an HC pair, we exclude each used reference itself from the five references. For an HI pair or HM pair, we eliminate the first human correct sentence from original references as their common new references. For an MM pair, we form the $C_{5}^{4}$ combinations selecting 4 out of 5 references as their common new references and calculate metrics on each of the 5 derivatives, then the five scores are averaged counting for the final score.

To enable a fair comparison across the pre-trained model based metrics, we choose models of roughly equal size (usually small) and pre-trained on paraphrase tasks (if possible), which are BERTScore (t5-paraphraser), BLEURT (BLEURT-tiny) and Sentence-BERT (paraphrase-TinyBERT-L6-v2). Note here BERTScore (t5-paraphraser) is much larger than the other two models. BLEU₁, BLEU₄, ROUGE_L, METEOR, CIDEr and SPICE are implemented with tools provided by DCASE challenge. ²²2https://github.com/audio-captioning/caption-evaluation-tools

To evaluate the performance of the proposed Error Detector, we randomly sampled 723 machine generated captions, and annotated if the caption has a fluency issue. We then compare the model’s prediction on the overall Error class (threshold=0.9) with our annotations. The model achieved a precision of 96.8, recall of 76.4, and F1 of 85.4.

Table 2 shows our experimental results on AudioCaps-Eval and Clotho-Eval across different metrics. The results are consistent on two datasets for a common explanation. Intuitively, HC and MM pairs pose bigger challenges for all metrics, as they are either equally good or equally bad on describing the corresponding audio clip. Nonetheless, pre-trained model based metrics generally perform much better than other metrics, especially our proposed FENSE which achieves the best overall accuracy on both datasets. BERTScore falls 4-5 percent behind other pre-trained model based metrics despite its biggest model size. On AudioCaps-Eval, N-gram exact match metrics (BLEU, ROUGE) perform 14-18 points worse than Sentence-BERT, while BLEU₁ and BLEU₄ struggle to exceed a random guess accuracy on MM pairs, showing their shortage differentiating two machine generated audio captions. METEOR and CIDEr attain a moderate performance gain beyond N-gram exact match metrics, however, they are still far behind Sentence-BERT. Not surprisingly, SPICE performs the worst, even unable to obtain 50% accuracy on AudioCaps MM and Clotho-Eval HC & MM, which strongly confirms our hypothesis that image captioning specified metrics cannot be adopted in audio caption evaluation.

Table 3 shows the overall accuracy of three representative metrics with and without sentence penalty of our proposed Error Detector (The last row: FENSE). With Error Detector, all the three metrics achieve extra performance gain, even for Sentence-BERT. This suggests that metrics leveraging either N-gram matching or contextual semantic similarity will fail in detecting fluency issues, and Error Detector can help mitigate such issues, thus setting a new SOTA.

Integrating Error Detector into Sentence-BERT metric, we get our proposed metric FENSE. Since table 2 and 3 have shown that FENSE significantly outperforms all the above metrics on the two evaluation datasets, we can use it to evaluate existing audio captioning system mentioned in §2.1. Fig 2 illustrates the judgments over four audio captioning systems made by humans, FENSE and SPICE on Clotho-Eval. Here we choose ATT, FC and RL whose beam search temperature equals to 0.5. On the y-axis we show the fraction of times one system wins another (rated better than another systems under this metric). We can see that NN is voted as the best system by both humans and FENSE, while it’s the worst under SPICE. Humans and FENSE ranked ATT, FC and RL accordingly after NN. However, SPICE fails to reach their consensus again. This indicates the superiority of FENSE, and further confirms our claim that image captioning specified metrics like SPICE are inappropriate to be transferred into audio caption evaluation.