Video to Music Moment Retrieval

We define VMMR as follows. Consider that we have a set of untrimmed complete music $\mathcal{M}$ and a set of videos $\mathcal{V}$. Given a video query $v\in\mathcal{V}$, the goal is to retrieval a music $m^{*}\in\mathcal{M}$ that conveys a suitable emotional tone to the content in the video query and localize the most appropriate music moment $w^{*}$ in the music $m^{*}$ by providing the start and end time point $\tau^{s}$ and $\tau^{e}$.

To better address the task, we designed a two-stage framework. In the first stage, Music Retrieval, we select the $\mathbf{top}k$ music tracks from the corpus $\mathcal{M}$ that exhibit semantic similarities with $v$. Subsequently, in the second stage, Moment Localization, we pinpoint the most relevant moments from each of the $\mathbf{top}k$ music tracks.

Due to the massive demand for advertising short videos and the cost constraints from businesses, there’s an urgent need for automated and precise music recommendations in ad video production. To train such a model in a supervised manner, it is necessary to have a dataset of corresponding quadruple (short video, music, moment start timestamp, moment end timestamp) data, abbreviated as ($v,m,\tau^{s},\tau^{e}$). While large-scale datasets of videos with paired music are available, it is not easy to find datasets which also contain high-quality annotated moments with start and end time points for the corresponding music tracks. As a result, to tackle this new task, we devise a moment construction pipeline that generates moment timestamps based on video audio and corresponding available music in a weakly supervised mode. And we automatically construct an advertising domain dataset Ad–Moment. In the following part, we discuss the pipeline through steps of data collection, data cleaning, and moment localization.

Raw Data Gathering. To obtain a representative and diverse subset of short videos for advertising, we have authorized downloading videos from short video platforms as candidates. For this study, we gathered nearly 1 million diverse short videos uploaded between Jan 2023 and Sep 2023. Besides the MP4 video and corresponding MP3 background music files, we also downloaded varied metadata, including titles, tags, and protobuf files containing playback information, if available. Specific annotated data instances are illustrated in the supplementary material.

Automated Data Cleaning. As the raw data is quite diverse with varied annotation quality, automated data cleaning is necessary to remove videos that cannot accurately identify BGM music or have meaningless visual content and low view counts. After removing low-quality data, we filtered videos related to the advertising themes based on video and creator tags. At this point, we obtain approximately 80k videos with duration ranging from 2.1 seconds to 753.6 seconds, with a mean value of 29.1 seconds and a median of 23.8. Due to the nature of VMMR, which involves recommending appropriate music moments for short videos, we constrain the background music to consist of high-quality segments from complete musical compositions. Through the manual review of the raw data, we empirically select videos that are associated with only one background music track. Besides, the video duration is restricted from 5 to 50 seconds, while the duration of the complete music is constrained from 30 to 240 seconds. In the end, we select 53,194 short videos, each corresponding to 4,050 complete music tracks.

Ground truth Moment Localization.

After filtering out high-quality advertising short videos with paired music, the next goal is to construct fine-grained annotations for music moment time points. However, this is quite challenging because it is difficult for humans to distinguish differences in music at the millisecond level. To that end, we develop a weakly supervised multi-modal collection pipeline to obtain highly confident timestamp data. It consists of two core steps: Background Sound Extraction and Moment Localization. In the pipeline, it is essential to annotate the music used in a given video. Then, the pipeline will provide precise temporal location information.

Background Sound Extraction. At first, we extract the raw audio from the video and the corresponding complete raw music, which contains the musical rhythms, lyrics, and dialogue. Demucs[31] is a renowned toolbox for Music Source Separation tasks. It enables the separation of the audio into the vocal track and non-vocal track components. And it splits the raw music into vocal and instrumental tracks. In our scenario, we only use the non-vocal component to avoid potential shortcuts presented in the vocal track from the video, which is not permissible in our cross-modal learning task.

Moment Localization. In the next step, we use the non-vocal track, cleared of vocal noise interference, and the instrumental track from the music for timestamp localization. Specifically, we extract the waveform data from the two audio using the torchaudio package and create a sliding window on the instrumental waveform data with a window length matching that of the non-vocal track and a step size of 0.1 seconds. Then, we calculate the cosine similarity between the waveform data features to obtain a similarity ranking list. Finally, we identify the window with the highest similarity as the ground-truth moment corresponding to the raw audio.

Analysis and statistics. To further evaluate the quality of timestamp annotations, we conducted a user study. Specifically, we randomly select GT moments, moments with a 3-second deviation with GT, and random moments from the corresponding complete music videos. These three categories are then matched with the video audio to determine the most accurate corresponding moment. Six individuals were assigned to evaluate the same 200 videos and calculate the average of their results. As shown in Fig. 2a, over $95\%$ of the matches aligned with the ground truth (GT). The word cloud of specific video tags in Fig. 2b shows the video content diversity.

Tab. 1 displays the overall data statistics. We separately allocated 2,000 videos with different music as the validation set and test set, and the remaining data as the training set. The average duration and standard deviation of all the music are 138.9 seconds and 69.6 seconds, respectively. The videos have an average duration of 23.9 seconds with a standard deviation of 10.7 seconds. As shown in Fig. 2c, the video count per music exhibits a clear long-tail distribution, with the most frequently used music appearing in 1,941 videos. Fig. 2d shows that some moments are selected from music beginning. Furthermore, the length of most moments is relatively small compared to the entire music track, indicating a tendency for videos to use segments of the music rather than the full track.

Feature Representation. The initial features of video and music are extracted from pre-trained networks. We will introduce the feature extraction and contextual modeling for video and music separately.

Music Feature. Given a music track, we first pad it to match the longest duration of music in the dataset with zero, which is 240 seconds, and load it by torchaudio package in the sample rate of $16k$. Next, we divide it into $S$ overlapping audio segment $m_{k}=\left\{m_{k}^{i}\right\}^{S}_{i=1}$, where $S$ represents the number of segments obtained from the padding music. Each segment is 10 seconds long, with a window stride of 5 seconds. We encode each music segment $m_{k}^{i}$ into a feature vector $x_{m_{k}}^{i}$ using a pre-trained AST model [7], applied to the audio spectrograms. The music features are extracted offline without fine-tuning the AST model. These features can be formulated as $\mathbf{x}_{m_{k}}=\left\{\mathbf{x}_{m_{k}}^{i}\right\}_{i=1}^{S}$, where $\mathbf{x}_{m_{k}}^{i}\in d^{768}$.

Video Feature. Given an input video query sequence, we extract all frames at a rate of 1 fps, denoted as $v=\left\{v^{i}\right\}_{i=1}^{W}$, where $W$ is the total number of frames in the video. We then use a pre-trained CLIP model [29] to extract visual feature $\mathbf{x}_{v}^{i}$ for each video frame $v^{i}$. The pre-trained CLIP model is not fine-tuned. The video feature are represented as $\mathbf{x}_{v}=\left\{\mathbf{x}_{v}^{i}\right\}_{i=1}^{R}$, where $\mathbf{x}_{v}^{i}\in d^{512}$ and $R\geq W$ is the maximum number of frames in the videos. Any missing frame features will be padded with zeros for videos with shorter durations.

Context Modeling. Due to the difference in dimensions of outputs from the CLIP and AST models, we compress the input features from both modalities to a unified dimension of $d=256$ using a linear projection layer, which not only reduces the number of model parameters but also performs dimensionality reduction for features. We also select $d=256$ as the output dimension for all subsequent encoding modules. We employ the Transformer architecture [34] for our music and video encoders. Transformers are crucial for improving model performance by encoding context sequences and modeling temporal relationships from video and music. Unlike MVPt, we use a masked single-block Transformer followed by mean pooling to aggregate frame or music segment features instead of relying on the [CLS] token. An important point to note is that the Transformer encoder shares parameters between the video and music sides and applies masking operations within each modeling module. The specific architecture is depicted in the supplementary material.

More formally, the key data flow of the visual and musical sides is expressed as follows:

Cross-Modal Contrastive Learning. Given a batch of $B$ (music, video) pairs, the model needs to generate and optimize $B\times B$ similarities. We apply InfoNCE [26], a symmetric cross-entropy loss, over these similarity scores to optimize the model’s parameters,

where $s(\hat{\mathbf{y}}_{m},\hat{\mathbf{y}}_{v})$ is the cosine similarity function, and $\tau$ is a learnable temperature parameter.

The loss $\mathcal{L}$ is the sum of video-to-text loss $\mathcal{L}_{m2v}$ and text-to-video loss $\mathcal{L}_{v2m}$. Notably, because multiple videos may share the same complete music, the positive music sample $\hat{\mathbf{y}}_{m}^{+}$ in $\mathcal{L}_{v2m}$ refers to all music instances corresponding to $v_{k}$ in the batch instead of just $\hat{\mathbf{y}}_{m_{k}}$.

After the retrieval stage, we obtain $k$ candidate musics that are most similar to the given video. The localization stage aims to identify the most relevant moments from each music.

Simply put, we can consider two straightforward approaches to tackle this task. We first use a sliding window to generate multiple window proposals, and then perform similarity retrieval by the model from the first stage. However, this approach suffers from significant efficiency issues, and the retrieval model struggles to distinguish different segments within the same music. Then, we guess that the detected highlights of the music, without considering the video content, might be sufficient for the VMMR task. Nevertheless, experiments in Tab. 4 revealed that not all ground-truth moments used the highlight music segments.

Therefore, as shown in Fig. 3, we designed Music–DETR, drawing inspiration from the DETR [2] architecture in visual object detection, which directly predicts a moment from the music for a given video in an end-to-end manner.

Cross-Modal Fusion. We use the same inputs and context modeling modules as the retrieval model, except for mean pooling, to separately obtain music segment representations $\mathbf{y}_{m}=\{\mathbf{y}_{m}^{i}\}_{i=1}^{S}$ and frame representations $\mathbf{y}_{v}=\{\mathbf{y}_{v}^{i}\}_{i=1}^{R}$. To extract video-related semantic information from the music features, we use a cross-attention Transformer [20] module to construct joint representation $\mathbf{z}_{c}$ for subsequent moment localization. Specifically, the keys and values from the video modality are fed into the music modality’s multi-headed attention block. To ensure that the output cross-modal features are suitable for subsequent music localization, i.e. video-guided music features, we select the music modality as the Query.

After cross-modal feature fusion, we further encode the features to $\hat{\mathbf{z}}_{c}$ using a transformer.

DETR-Decoder. Here, we follow the DETR method [2] in object detection to perform end-to-end temporal moment localization. The moment queries $\mathbf{q}_{m}$ in the decoder are composed of decoder embeddings and randomly initialized learnable embeddings. In our work, the choice of decoder embeddings in moment queries is particularly important. Compared to zero initialization [14, 23] in video moment retrieval, we believe that in multimodal tasks, the video query provides clear guidance for locating the target. Specifically, by mean-pooling the frame embeddings $\mathbf{y}_{v}$ to video embedding $\hat{\mathbf{y}}_{v}$, we then repeat it $Q$ times to serve as the moment queries for $Q$ predictions. This acts as an effective prior, helping the model to better focus on relevant temporal regions during the localization process. This decoder, as a cross-attention transformer, is formulated as follows,

Prediction Heads. Based on the decoder output $\mathbf{E}_{\text{dec}}$, we apply a 3-layer feed-forward network with ReLU [6] activation $f_{\text{loc}}$ to predict the normalized moment center coordinate and width. During training, the best matching moment is selected from the $Q$ ones with the Hungarian algorithm. We also follow the approach from DETR, where a linear layer with softmax $f_{\text{cls}}$ is used to predict class labels. In DETR, this layer is trained by object class labels. In our context, we assign a foreground label to a predicted moment if it corresponds to ground truth, and a background otherwise. To further constrain the relationship between the decoder output and video, we use a linear layer $f_{\text{embed}}$ to obtain a moment embedding aligned with the video representations before the cross-attention. Additionally, to facilitate the learning of the moment, we add the music embedding as a shortcut connection to it.

Loss. Our moment detection loss $\mathcal{L}_{\text{det}}$ measures the discrepancy between the ground-truth and predicted moments. Following DETR, $\mathcal{L}_{\text{det}}$ is computed as a weighted combination of an $L_{1}$ loss and a generalized IoU loss [30]. A cross-entropy loss $\mathcal{L}_{\text{cls}}$ is used to help discriminate moment (foreground) and non-moment (background) segments in a given music. Additionally, to improve the alignment between video and music moment, we use contrastive learning to compute the alignment loss $\mathcal{L}_{\text{align}}$ between the video and moment embeddings. The final loss summarizes the three losses.

Datasets. Our experiments are mainly conducted on Ad–Moment. The dataset is divided at random into three disjoint subsets: 49k videos for training, 2k for validation, and 2k for testing, see Tab. 1.

ReaL also works for VMR. So in addition to Ad-Moment, we adopt the public HIMV dataset [10] commonly used for VMR. While HIMV have links to 200k videos, we downloaded with success only 138,265 videos as many links expired. We term this subset HIMV-138k. It is worth pointing out as no data split is publicly available, performance on HIMV is not directly comparable across papers. For a relatively fair comparison, we randomly select 2k videos for validation and another set of 2k videos for testing, with the retained 134,265 videos for training.

Evaluation criteria. For VMR, we use Recall@k, the fraction of queries that correctly retrieve the targeted music in the top k results (k=1, 5, 10, 25). For MML, we use mean Intersection over Union (IoU), given that the relevant music is already available. For VMMR, we use Recall@K, IoU=m [11], the percentage of test samples that have at least one predicted moment with IoU$>m$ in the top-k predictions (k=1,10,100 and m=0.5).

Implementation details. We perform experiments on 8 V100 GPUs. The mini-batch size is 512. An initial learning rate of 1e-4 is used to train the retrieval stage and 3e-4 for the localization stage. We train the models for $100$ epochs in retrieval and $80$ epochs in localization with the AdamW optimizer [19]. Following CLIP, we employ a cosine schedule [18] with a warm-up proportion of $0.05$. We sample frames from all videos at $1$ fps and resize all frames to $224\times 224$ pixels.

Results on VMR. Tab. 2 shows the VMR results on HIVM. Since previous methods for VMR are not open-sourced, we directly cite their published numbers. YT8M-MV, HIMV-200k and HIMV-138K are all subsets of YouTube8M under the “music video” category. Although this comparison is not entirely fair due to differences in datasets, it is evident that our simple and decoupled method achieves comparable scores even with smaller-size training data. Given that UT-CMVMR uses much larger training data and more features (optical flow and rhythm), our method is mostly comparable.

In Ad–Moment, we also evaluated the performance of the retrieval model in Tab. 3. The ablation study on the input stride size will be discussed later.

Results on MML. We used a sliding window approach with a 1-second stride and video duration as the window length. Multiple windows are selected from the full music track and ranked based on similarity by the retrieval method, ultimately choosing the highest similarity window as the predicted moment. Additionally, when selecting moments from complete music, we employed a highlight detection model [8] to find a highlight fully based on the duration of the video. For music where no highlight can be detected, we simulate the choice of a music editor by simply choosing the segment from the beginning. This also explains why many segments start from the beginning, see Fig. 2d. Our method outperforms the baselines, see Tab. 4.

Results on VMMR. To ensure a fairer comparison, the results for the two baseline methods in the complete VMMR task are calculated based on the best retrieval performance (top line in Tab. 3). Similar to the Music Moment Localization task, ReaL method also shows higher metrics over the straightforward baseline, obtaining $105\%$ ($0.90\rightarrow 1.85$) boost at $\text{R@}100,\text{IoU=}0.5$ in the complete VMMR task.

We provide the ablation study for the framework in terms of module details, learning objectives, and hyperparameters.

Music embedding as auxiliary. With the music embedding shortcut removed, we observe a $7.6\%$ decrease at Mean IoU in the music moment localization subtask. This shows the necessity of music as an auxiliary component.

Video embedding to initialize. We further evaluate the impact of video embedding as initialization of decoder moment queries. By comparison, using video as the init significantly enhances the model’s performance. This is because, in the case of moment retrieval as a multimodal task, the video modality provides clear guidance for locating the target, unlike in object detection. Therefore, relying solely on the random init of learnable embeddings in Fig. 3 is entirely insufficient.

Is $\mathcal{L}_{\text{align}}$ necessary? Removing the $\mathcal{L}_{\text{align}}$ results in about $3\%$ drop at IoU in Tab. 4. The results from the localization subtask and the overall VMMR task indicate that the moment alignment is essential.

Hyperparameters. Tabs. 3, 4 and 5 discuss the effect of the stride size when using a sliding window from the music track input. Performance under different sizes are presented. The VMMR performance is based on the same stride size, ensuring a consistent comparison across varying configurations. A smaller stride represents a higher sampling rate, suggesting that more dense sampling provides clear advantages for the localization task. Through temporal learning, this method demonstrates robustness to different configurations of the input music.

Further analysis for localization task. Considering real-world end-to-end applications, the duration for the predicted moment should closely align with the given video length. To this end, we further analyze the moment length by observing the predicted and ground truth moment. We visualized the number of samples grouped by the duration difference of 1-second intervals between the predicted and the ground truth moments in Fig. 4. Since the input video is constrained to the maximum duration by padding with zeros in the input set, the model indirectly learns the duration information of video during masked context modeling. By average, the mean lines of the predicted and ground truth moments are nearly identical ($22.75\text{s}\rightarrow 22.79\text{s}$), ensuring the potential effectiveness of our approach in real-world applications.