Hear The Flow: Optical Flow-Based Self-Supervised
Visual Sound Source Localization

Given a video consisting of both the audio and visual modality, visual sound source localization aims to find the spatial region in the visual modality that generated the audio. Consider a video consisting of $N$ frames. Let the image corresponding to a video frame be $I$, where $I\in\mathbb{R}^{W_{i}\text{x}H_{i}\text{x}3}$ and $A$ be the spectrogram representation generated out of the audio around the frame, where $A\in\mathbb{R}^{W_{a}\text{x}H_{a}\text{x}1}$. The problem of audio localization can be thought of as finding the region in $I$ that has a high association/correlation with $A$. More formally, this can be written as:

where $\Phi(I;\theta_{i})$ and $\Psi(A;\theta_{j})$ correspond to convolution neural network-based feature extractors associated with visual and audio modalities, and $f_{v}\in\mathbb{R}^{m\text{x}n\text{x}c}$ and $f_{a}\in\mathbb{R}^{m\text{x}n\text{x}c}$ are the corresponding lower dimensional feature maps, respectively. $\omega$ is the function that finds the association between the two modalities, and $P(I,A)$ is the region in the original image space with the source that generated the audio. It is important to note that extrapolating the association in the feature space to the corresponding region of the original image space (i.e $P(I,A)$ ) is trivial. Given the above-mentioned feature maps, one way of finding an association between the feature representations is:

where $GAP(f_{a})$ is the global-average-pooled representation of the audio feature map. S represents the cosine similarity of this audio representation to each spatial location in the visual feature map. Here $m$ and $n$ are the width and height of the feature map. If a binary mask $M\in\mathbb{R}^{m\text{x}n\text{x}1}$ generated from a ground truth indicating positive and negative regions of audio-visual correspondence is available, we can formulate the learning objective in a supervised setting:

For a given sample $k$ (with an image frame $I_{k}$ and audio $A_{k}$) in the dataset, the positive and negative response can be defined as

Here $S_{k\rightarrow k}$ refers to the cosine similarity $S$ from Eq 2 when using $I_{k}$ and $A_{k}$. Similarly, $S_{k\rightarrow j}$ is the cosine similarity when the image and audio are not from the same video. $\langle\cdot,\cdot\rangle$ denotes the inner product. The final learning objective has a similar formulation to [23]:

In most real-world scenarios, the ground truth necessary to generate the binary mask $M$ would be missing. Hence there is a need for a training objective that does not rely on explicit ground truth annotations. One way to achieve this objective is to replace the ground truth mask with a generated pseudo mask as proposed in [6]. The pseudo mask can be generated by binarizing the similarity matrix $S$ based on a threshold. More specifically, given $S_{k\rightarrow k}$ from Eq 2, the pseudo mask can be written as:

where $\epsilon$ is a scalar threshold. $\sigma$ denotes the sigmoid function that maps a similarity value in $S_{k\rightarrow k}$, that is below the threshold to value 0 and above the threshold to 1. $\tau$ is the temperature controlling the sharpness. Additionally, [6] further refines the pseudo mask by eliminating potentially noisy associations. This is done by considering separate positive and negative thresholds above and below the similarity value that is considered reliable. If a value is between these thresholds, it’s considered a noisy association and is subsequently ignored. More formally:

Here $\epsilon_{p}$ and $\epsilon_{n}$ are the positive and negative thresholds, respectively. Once the positive and negative responses are computed, the overall training objective is similar to Eq 4.

In the above approach, it is logical to bootstrap the prediction and perform self-supervised training if the pseudo masks in Eq 5 generated at the initial training iterations resemble that of the ground truth. However, this is not guaranteed since the feature extractors associated with the individual modalities (in Eq 1) are randomly initialized. Therefore, a high or low value in the similarity matrix $S_{k\rightarrow k}$ during the initial iterations of the self-supervised training may not correspond to informative positive or negative regions since the feature extractors are not trained. If a feature extractor is initialized with pretrained weights from a classification task, for example, the visual extractor on ImageNet, the network will often activate towards objects in the image. Considering this characteristic as an object-centric prior, it may be useful for self-supervised sound localization as the most salient objects in a frame are often the ones emitting the sound. However, situations may arise where the source of the audio is not the most salient object in the frame. This would produce sub-optimal associations $S_{k\rightarrow k}$ in the initial iterations, which, when used for self-supervised training as mentioned in Eq 6 would lead to sub-optimal performance. As a result, there is a need to construct more meaningful priors when computing $S_{k\rightarrow k}$ to improve audio-visual associations, subsequently improving self-supervised learning.

Having motivated the need for some meaningful priors that enable better audio-visual associations, we approach the problem from an object detection viewpoint. In earlier object detection methods such as R-CNN [15] and Fast R-CNN [14], a selective search was used as a method to generate region proposals. Selective search provided a set of probable locations where an object of interest may be present. An alternative to selective search-based approaches is a two-stage approach like in [27], using region proposal networks. Most of these region proposal networks have auxiliary training objectives in order to produce regions containing potential objects. Using these objectives to generate potential regions of interest in a self-supervised setting becomes challenging. Furthermore, generating candidate regions using selective search or regular region proposal networks, only based on visual modality, might not be well suited for enforcing priors for a cross-modal task such as visual sound source localization.

As a better alternative, we use optical flow to generate informative localization proposals. Optical flow using frames of a video can efficiently capture the objects that are moving. Most often, these objects are the source of the sound. Capturing optical flow in the pixel space can often be a good prior to improve audio-visual association. Furthermore, since the optical flow tends to focus on the relative motion of objects rather than the salient objects, it can complement the priors of the pre-trained vision model, which tends to focus on the latter. We design a network as shown in Figure 2, which takes in optical flow computed between two adjacent video frames and generates regions in the feature map $f_{v}$ that act as priors to create better audio-visual associations. The localization network is comprised of a cross-attention between the feature representation extracted from image and flow modalities. Given the flow feature representation $f_{f}$ and visual feature representation $f_{v}$, we project these feature representations using separate projection layers to create two tensors $K_{v}$ and $Q_{f}$. $\beta$ is computed as an outer product of the tensors $K_{v}$ and $Q_{f}$ along the channel dimensions. That is, if $K_{v}$ and $Q_{f}$ $\in\mathbb{R}^{m\text{x}n\text{x}d}$, then the resulting $\beta$ $\in\mathbb{R}^{m\text{x}n\text{x}d\text{x}d}$ is computed as below:

The softmax function is applied to the final dimension to normalize the attention matrix. The goal is to compute the attention to be applied to each spacial location, thus yielding a cross attention matrix of size $d\text{x}d$ for each spatial location. We compute another tensor from the visual modality ${V_{v}}$ $\in\mathbb{R}^{m\text{x}n\text{x}d}$. For each spatial location in $V_{v}$, we have a $d$ dimensional representation which we multiply to the corresponding $d\text{x}d$ attention matrix in $\beta$. That is :

Finally $E$ is projected back to produce the final cross-attended proposal prior $E_{p}$ $\in\mathbb{R}^{m\text{x}n\text{x}c}$. In order to impose this prior for performing the audio-visual association, we add $E_{p}$ to the visual feature map $f_{v}$ as shown in Figure 2. The enhanced audio-visual association can be written as:

where $\oplus$ denotes element-wise addition. Once the enhanced audio-visual association is obtained, we use Eq 6 to compute the positive and negative responses. We train the entire network (feature extractors and localization network) end-to-end using the optimization objective mentioned in Eq 4.

For training and evaluating our proposed model, we follow prior work in this area and use two large-scale audio-visual datasets:

For proper comparisons against prior works, we use two metrics to quantify audio localization performance: Consensus Intersection Over Union (cIoU) and Area Under Curve of cIoU scores (AUC) [28]. cIoU quantifies localization performance by measuring the intersection over union of a ground-truth annotation and a localization map, where the ground-truth is an aggregation of multiple annotations, providing a single consensus. AUC is calculated by the area under the curve of cIoU created from thresholds varying from 0 to 1. In our experiments, we show results for cIoU at a threshold of 0.5, denoted by $cIoU_{0.5}$, and AUC scores, denoted by $AUC_{cIoU}$.

In this paper, sound source localization is defined as localizing an excerpt of audio to its origin location in an image frame, both extracted from its respective video clip. For both Flickr SoundNet and VGG Sound, we extract the middle frame of a video along with 3 seconds of audio centered around the middle frame and a calculated dense optical flow field to construct an image-flow-audio pair. For the image frames, we resize images to $224\times 224$ and perform random cropping and horizontal flipping data augmentations. To calculate an optical flow field corresponding to the middle frame, we take the middle frame and subsequent frame of a video $V$, denoted by $V_{t}$ and $V_{t+1}$ respectively, and use the Gunnar Farneback [10] algorithm to generate a 2-channel flow field corresponding to horizontal and vertical flow vectors denoting movement magnitude. We similarly perform random cropping and horizontal flipping of the flow fields, which are performed consistently with image augmentations. For audio, we sample 3 seconds of the video at 16kHz and construct a log-scaled spectrogram using a bin size of 256, FFT window of 512 samples, and stride of 274 samples, resulting in a shape of $257\times 300$.

Following [6], we use ResNet18 backbones as the visual and audio feature extractors. Similarly, we use ResNet18 as the optical flow feature extractor. We pretrain the visual and flow feature extractors on ImageNet and leave the audio network randomly initialized. During training, we keep the visual feature extractor parameters frozen. For all experiments, we train the model using the Adam optimizer with a learning rate of $10^{-3}$ and a batch size of 128. We train the model for 100 epochs for both the 10k and 144k sample subsets on Flickr SoundNet and VGGSound. We set $\epsilon_{p}=0.65$, $\epsilon_{n}=0.4$, and $\tau=0.03$, as described in Eq 6.

In this section, we compare our method against prior works [1, 6, 19, 25, 28, 29, 30] on standardized experiments for self-supervised visual sound source localization. Results of various training configurations are reported in Tables 1 and 2 for the Flickr SoundNet and VGG Sound Source testing datasets, respectively.

As shown in Table 1 and 2, our method, HTF, significantly outperforms all prior methods, creating the new state-of-the-art on self-supervised sound source localization. On the Flickr testing set, we achieved an improved performance of 11.7% cIoU and 4.7% AUC when trained on 10k Flickr samples and 10.6% cIoU and 2.9% AUC when trained with 144k Flickr samples. Similarly, on the VGG Sound Source testing set, we improve by 7.9% cIoU and 2.9% AUC when trained on 10k VGG Sound samples and 5.5% cIoU and 2.0% AUC when trained on 144k samples.

Further, we investigate the robustness of our method by evaluating it across the VGG Sound and Flickr SoundNet datasets. Specifically, we train our model with 144k VGG Sound samples and test on the Flickr SoundNet test set. Comparing against [6, 29, 30], we significantly outperform all methods, as shown in Table 1, which shows our model is capable of generalizing well across datasets. We further investigate our method’s robustness by testing on sound categories that are disjoint from what is seen during training. Following [6], we sample 110 sound categories from VGG Sound for training and test on the same 110 categories (heard) used during training and 110 other disjoint (unheard) sound categories. As shown in Table 3, we outperform [6] on both heard and unheard testing subsets. In addition, we highlight that the performance of the unheard subset slightly outperforms the heard subset, showing our model performs well against unheard sound categories.

Utilizing a self-supervised loss formulation similar to [6], we see that our method significantly outperforms it on both testing datasets across all training setups and experiments. We highlight that these improvements are obtained from incorporating a more informative prior, based on optical flow, into the sound localization objective. In section 4.6, we further investigate the direct influence of incorporating optical flow along with our other design choices.

In Figure 3, we visualize and compare sound localizations of LVS [6] and our method on the Flickr SoundNet and VGG Sound Source test sets. As shown, our method can accurately localize various types of sound sources. Comparing against LVS [6], we examine localization improvements across multiple samples, specifically where sounding objects exhibit a high flow magnitude through movement. For example, in the first column, LVS [6] localizes only a small portion of the sounding vehicle, while our method entirely localizes the vehicle, where a significant magnitude of flow is exhibited. In the fifth column, our method more accurately localizes to the two crowds in the stadium, both of which are sound sources exhibiting movement.

However, it is also important to investigate samples where little optical flow is present. It is possible that a frame in a video exhibits little movement, for example, a stationary car or person emitting noise. In these cases, there is no meaningful optical flow to localize towards. In Figure 4, we see that even in the absence of significant optical flow, our method still localizes on par or better compared to LVS [6]. This reinforces that optical flow is used as an optional prior, where areas of high movement when present, may be used to localize better, but is not required. In the following section, we further investigate the exact effects of introducing priors like optical flow into the self-supervised framework.

In this section, we explore the implications of our design choices with multiple ablation studies. As explained in section 3.2, we explore the need for informative priors to train a self-supervised audio localization network. We introduce optical flow as one of these priors, in addition to pretraining the vision network on ImageNet to provide an object-centric prior. In Table 4, we study the individual effects of each of these design choices, namely adding the flow attention mechanism, ImageNet weights for the vision encoder, and freezing the vision encoder during training.

When training the model without any priors (model 4.a), we see that performance suffers as there is little meaningful information for the self-supervised objective. However, when simply adding the optical flow attention previously described (model 4.d), we see a large performance improvement as the network can now use optical flow to better localize, as moving objects are often the ones exhibiting sound. Similarly, when using ImageNet pretrained weights (Table 4.b), we see a significant performance improvement as the model now has an object-centric prior, where salient objects in an image are often exhibiting sound. When combining both priors (model 4.e), we see even further performance improvements, which show the importance of incorporating multiple informative priors for the self-supervised sound localization objective.

We further explore the effects of freezing the vision encoder during training. As previously mentioned, a network pretrained on a classification task such as ImageNet will often have high activations around salient objects (an object-centric prior). When training in a self-supervised setting, the network may divert from its original weights and instead have a less object-centric focus, which may be suboptimal for sound source localization. When freezing the network in the non-flow setting (model 4.c), we see performance slightly decrease compared to the unfrozen counterpart (model 4.b). However, when freezing the network in the optical flow setting (model 4.f), we see a slight improvement over the flow setting with an unfrozen vision encoder (model 4.e). We infer that enforcing the vision encoder to keep its object-centric characteristics while the flow encoder can reason and attend towards other parts of the image produces a more informative representation, improving localization performance.

Finally, we explore variations of the optical flow encoder to better understand how the optical flow information is being used. We replace the learnable ResNet18 encoder with a single max pooling layer to see if the simple presence of movement is still informative for localizing sounds. As shown in Table 5, when using a simple max pooling layer (model 5.a), we still notice a significant performance improvement over the network without optical flow (models 4.a-c). However, we see a further improvement over the max pooling layer when using a learnable encoder, like a ResNet18 network. While the max pooling layer only captures the presence of movement at a particular location, a learnable encoder allows deeper reasoning of the flow information. For example, the eighth column in Figure 3 shows an optical flow field where the sounding object (tractor) is not moving but rather the environment around it is. In this case, with the max pooling encoder, the network is biased away from the sounding object, whereas a learnable encoder can better reason about the flow in the given frame, improving overall localization performance.