Contrastive Latent Space Reconstruction Learning for Audio-Text Retrieval

Despite the effectiveness of the previous methods, audio-text retrieval is less studied due to the difficulty of discovering discriminative representations for audio clips. Chechik et al. [11] proposed a scalable machine learning method with the passive-aggressive model (PAMIR). Ikawa et al. [12] constructs a latent variable space with the onomatopoeia generation. Elizalde et al. [13] utilize a siamese network to jointly map audio and text features into a shared space. However, these methods are restricted by the form of queries. Andreea Maria et al. [14] exploits Mixture-of-Embedded Experts (MoEE) [17] and Collaborative-Experts (CE) [18] for the purpose of audio-text retrieval involving natural language descriptions. Lou et al. [19] generate audio-text embeddings utilizing pre-trained audio neural networks (PANNs) [20] and employ NetRVLAD pooling [21]. Mei et al. [22] incorporates contrastive learning to increase feature discrimination.

Contrastive learning is an unsupervised paradigm designed to enhance the quality of representations by promoting the closeness of positive pairs and the distinctiveness of negative ones. InfoNCE [23] is proposed to maximize the lower bound of mutual information. NT-Xent [16] aims to promote multi-modal representation learning by building two symmetric contrastive losses.BYOL [24] achieves promising results without negative pairs. Decoupled Contrastive Learning (DCL) [25] improves the effectiveness by removing the positive pairs in the denominator of the contrastive loss. Recently, Huang et al. [26] proposed Model-Aware Contrastive Learning (MACL) to solve uniformity-tolerance dilemma and gradient reduction. These contrastive methods have been proven effective experimentally.

In this paper, The multimodal dataset is written as $\mathcal{D}=\{\mathrm{A},\mathrm{T}\}$, where $\mathrm{A}=[a_{1},a_{2},...,a_{m}]$ and $\mathrm{T}=[t_{1},t_{2},...,t_{m}]$ represent the audio modality and the text modality, respectively. A data batch is denoted as $O=[o_{1},o_{2},...,o_{b}]$, where $b$ is the batch size and $o_{j}=[a_{j},t_{j}]$ is the $j$th audio-text pair. The relevance between two instances is measured by cosine similarity:

where $x$ and $y$ are two vectors.

The proposed CLSR framework is depicted in Fig. 1, which comprises three parts: the feature extraction module, the confidence-aware contrastive module and the modality reconstruction module.

The effectiveness of CLSR is verified on two standard datasets, i.e., AudioCaps [32] and Clotho [33].

AudioCaps comprises approximately 50,000 audio clips, each lasting for 10 seconds, sourced from AudioSet [34]. Among these, 49,274 audio clips have been chosen for the training set, each accompanied by its respective textual description. Furthermore, the validation set comprises 494 audio clips, while the test set contains 957 audio clips. Each audio clip in both sets is paired with five corresponding captions.

Clotho contains audio clips ranging from 15 to 30 seconds, annotated by corresponding textual data. The training set comprises 3,839 audio clips, while both the validation and test sets include 1,045 audio clips each. Additionally, each audio clip is paired with five corresponding textual descriptions.

Following [22], the log Mel-spectrogram is computed using a Hanning window with 1024 points and a hop size of 320 points, resulting in 64 mel bins. The maximum training epoch is set to be 50, with Adam as the optimizer. The learning rate is set as $1e^{-4}$ with a decay rate of $1/10$ per 20 epochs. The initial temperature $\tau_{0}$ is equal to $0.07$, the scaling factor $\gamma$ is set to 1.2, and the dimension of the output embedding is $1024$. $\alpha$ and $\beta$ are set to be 1 and 0.1, respectively. Experiments are carried out using a single NVIDIA Tesla V100 GPU.

The evaluation metric employed is Recall at rank $k$ (R@k), which indicates whether the most relevant content to a query appears at the top rank $k$. The R@k metric is scaled between 0 and 1, and higher values signify better performance, where the top-ranked retrieval result is more likely to be related to the query.

We compare CLSR with several baselines, i.e., MOEE [17], CE [18], ARC [19], and ASE [22]. It should be noticed that MOEE and CE are two frameworks used for text-video retrieval, and, following [35], we adapt it for text-audio retrieval. We evaluate CLSR and these baselines on AudioCaps and Clotho, and the experimental results including R@1, R@5 and R@10, are detailed in Table I. Based on the experimental results, it can be observed that:

• •

  CLSR consistently achieves superior R@k results in both Audio-to-Text and Text-to-Audio tasks across both datasets. CLSR considers enlarging the distance of negative pairs to enhance instance discrimination, contributing to more robust latent embedding. For contrastive learning, to alleviate the limitation of a small batch size, a reweighting strategy is imposed to adaptively adjust weight of different positive pairs.

• •

  In most cases, R@k results on Audio-to-Text task tend to surpass those for the Text-to-Audio task. This trend may be attributed to the rich semantic information in textual data compared to audio data.

For CLSR, multi-modal contrastive learning, semantic consistency and the reconstruction loss are adopted for CMR representation learning. In this subsection, ablation study is conducted to analyze their effects. From CLSR, four variants are derived, i.e., CLSR-s, CLSR-t, CLSR-k and CLSR-m. CLSR-s removes the intra-modal contrastive loss, i.e., the third term and the fourth term in Eq. (6). CLSR-t drops the confidence-aware temperature control strategy. CLSR-k adopts no semantic consistency loss, which is the second term in E.q (11). CLSR-m discards the reconstruction loss. CLSR’s performance is assessed in comparison to these four variants on AudioCaps.

The R@k scores are listed in TABLE II. It can be shown that CLSR generally surpasses these variants. Concretely, CLSR-s achieves low R@k results, which demonstrates the effectiveness of the expanded intra-modal contrastive loss for learning robust latent representations. CLSR-t, CLSR-k achieve better results than CLSR-s, which are still inferior to CLSR. This demonstrate that the adaptive temperature scheme and semantic consistency contribute to improving the retrieval results. The performance of CLSR-m is satisfying, but still lower than CLSR, validating the efficacy of the reconstruction loss.