Telling Left from Right: Learning Spatial Correspondence of Sight and Sound

Due to the challenges in obtaining large-scale annotated data for supervised training, many works have proposed leveraging audio-visual correspondences for self-supervised learning [3, 8, 22, 26, 27, 28, 29]. These tasks learn audio-visual representations either by leveraging the shared semantic information [3, 29] or by exploiting temporal correlation [22, 26, 27, 28]. However, they do not exploit spatial correspondence for self-supervision, and most only take mono audio as input.

Inspiring our effort is prior work on audio-visual correspondence tasks for predicting whether visual and audio signals come from the same video [3], and whether visual and audio signals are temporally aligned [22, 27]. These tasks learn audio and visual representations based on correspondence in semantic or temporal information. In contrast, our correspondence task is designed to teach a model to match spatial audio cues to positions of sound sources in the video and exploits the spatial relation.

Audio-visual source separation utilizes visual information to aid in the separation of sound mixtures. Many self-supervised approaches have been proposed to solve this task [10, 13, 15, 27, 32, 36], including mix-and-separate frameworks that combine audio tracks from videos and then train models to separate them using visual information [10, 27, 36, 37]. In order to leverage the visual cues, these separation models learn an audio representation that captures the semantics [37] or temporal patterns [10, 36] of sound in order to match them to the visual frames respectively. Strategies to separate audio without explicit mixtures [13, 15] or to co-segment video at the same time [32] have also been proposed. However, all of these approaches still learn an audio representation based on matching semantics or temporal patterns and do not leverage spatial cues.

Estimating the direction-of-arrival of a sound using multiple microphones has been traditionally tackled using beamforming algorithms such as steered power response [6] that do not learn an audio representation and cannot easily handle the concurrency of multiple sound sources. More recently, methods based on neural networks have been proposed for direction of arrival estimation [1, 19, 23]. These models learn spatial audio representations, but they are trained through strong supervision, while we propose to use self-supervision to learn spatial audio cues by leveraging audio-visual spatial correspondence.

A separate stream of research has focused on localizing sound sources in videos [5, 11, 17, 21], including in self-supervised ways [4, 27, 33]. However, these methods do not exploit spatial audio. Recently, Gan et al. [12] proposed learning a spatial audio representation that localizes vehicles in video using a pretrained vision network as a teacher. In contrast, our representation is learned by leveraging audio-visual spatial correspondence without explicitly modeling the target locations of a teacher.

Several self-supervised approaches have recently been proposed for audio spatialization, which is the task of converting mono audio to spatial audio using a concurrent visual stream to inject the audio with spatial cues [14, 25, 24]. Similar to us, these approaches use spatial audio as a self-supervisory signal. For example, Gao et al. [14] propose a U-Net architecture with a visual stream to convert a mono audio input (generated by downmixing stereo audio) to a stereo output, using the original stereo audio as the target during training. In their model, the visual features provide complementary spatial information that is missing from the mono audio to produce stereo audio. In contrast, our audio representation learns spatial cues directly from stereo audio in order to match the perceived localization of a sound with its position in the video. Also similar to us, Lu et al. [24] proposed a spatial correspondence classifier, but they apply it as an adversarial loss for aiding audio spatialization whereas we propose the spatial audio-visual correspondence task for self-supervised feature learning.

Spatial audio enables listeners to infer the locations of sound sources. In the case of stereo audio, binaural cues such as differences in time of left and right signals arriving (inter-aural time differences) and the difference in levels of the left and right signals (inter-aural level differences) contribute to the perception of sound sources being localized to the left or right [30]. We hypothesize that these binaural cues can be used to learn useful multimodal representations by teaching a model to recognize when the video and audio are not spatially aligned. During training, we provide as input video clips where we flip the order of the channels in the audio stream with probability 0.5, i.e., if the original audio is given by $a(t)=(a_{l}(t),a_{r}(t))$, where $a_{l}(t)$ and $a_{r}(t)$ are the left and right channels as a function of time $t$, the flipped audio is $\tilde{a}(t)=(a_{r}(t),a_{l}(t))$. This transformation switches the inter-aural time and level differences between the audio channels.

Formally, let $D=\{(v,a,y)_{i}\}_{i=1}^{N}$ be our video dataset where $v$ is a visual stream, $a$ is an audio stream, and $y$ indicates whether or not $a$ is flipped with respect to $v$. We train the neural network $f_{w}(v,a)$ with parameters $w$ to maximize a classification cross-entropy objective given by the log-likelihood,

We conjecture that solving the flipping task requires understanding audio-visual spatial correspondence, as the model must match the location of objects in audio signals with the location of objects visual signals.

We illustrate our network in Figure 3(a). The network comprises two streams – a visual stream and an audio stream – which are fused by concatenating features along the time dimension, followed by additional convolutional layers to yield an output representation prior to classification. For vision, our base model uses the public PyTorch implementation of ResNet-18 [16] (note that we only apply spatial and not temporal convolutions). We use frames sampled at 6 Hz and resized to 256 x 256 as input. For audio, our base model uses stacked residual blocks with S&E [20], matching the output temporal dimension to that of the vision network. We use the log-scaled mel-spectrogram of audio sampled at 16 kHz and stack the left and right stereo channels as input. Our two-stream network is comparable to the architectures of previous audio-visual correspondence tasks [3, 22]. However, a key distinction is that our model requires the positions of sound sources detected by the visual sub-network. While spatial pooling of the features from the visual sub-network prior to fusion is suitable for other correspondence tasks, we found it necessary for our task to flatten the visual features along the spatial dimensions without pooling prior to fusion with the audio. Models used for audio spatialization tasks, which also require knowledge of the position of sound sources in the visual frame, process visual features in a similar way [14, 25]. For applications to downstream tasks, we can use features from the audio sub-network, visual sub-network, or the fused representation.

We train and evaluate our models on both our YouTube-ASMR dataset and the FAIR-Play dataset [14]. The latter dataset consists of approximately 2K 10-second video clips of people playing instruments. While this dataset is smaller than YouTube-ASMR, we use it to demonstrate the generality of our approach. For both datasets, we use 3-second clips sampled from full clips, introducing flipped audio examples with probability 0.5. We apply a random crop and shift the color/contrast of the frames for data augmentation. To account for possible left-right biases in the audio and visual information, we apply random left-right flipping of both the video and audio channels (note: flipping both at the same time maintains the audio-visual spatial alignment or lack thereof, and thus does not change the prediction target). For optimization, we used SGD (momentum=0.9) with a learning rate of 0.01 on up to 7M samples for YouTube-ASMR and 800K samples for FAIR-Play using 1-4 GPUs.

Our base model is trained from scratch using ResNet-18 as the visual sub-network architecture as shown in Figure 3(a). To determine if our model can obtain effective visual features with self supervision, we compared against a baseline using ResNet-18 pretrained on ImageNet classification and finetuned on our task (“Pretrained on ImageNet”). To assess the importance of motion features, we trained a model from scratch using MCx as the visual sub-network [35], which uses 3D spatiotemporal convolutions and is designed for video classification (“Motion”). To determine if semantic audio features improve the task performance, we force the model to learn semantic audio features in a third separate stream that only takes mono audio input (“+Mono audio”). This third stream uses the same audio sub-network as stereo audio (with one input channel instead of two) and is similarly fused by concatenating features along the time dimension. Finally, to determine whether the learned audio features additionally benefit from having traditionally computed spatial audio cues, we integrated features computed using Generalized Cross Correlation with Phase Transform [7] to the audio stream by introducing three additional channels in the stereo audio input (“+GCC-Phat”).

We report the test set classification accuracy of the audio-visual spatial correspondence model trained on YouTube-ASMR and FAIR-Play in Table 2. Our models perform well and comparably to the supervised baseline; in fact, the performance on the YouTube-ASMR dataset is comparable to human performance on a sub-sample of 200 clips (about $80\%$, N=2 subjects). This result indicates that our model architecture is well-suited for matching spatial audio cues with sound source locations in the video frames, and also that the spatial audio cues in both datasets are sufficiently rich for learning the visual features of sound sources. We did not obtain gains using the MCx network, suggesting that 3D spatiotemporal convolutional features may not be integral to the task on these datasets. We observed gains using the dual audio model (“+Mono Audio”), indicating that using semantic audio features could potentially aid performance on the pretext task (e.g., when there are multiple sound sources coming from different directions). Finally, GCC-Phat features did not improve the model performance, which suggests that our audio sub-network learned, in a fully self-supervised manner, the spatial localization cues that are traditionally computed using beamforming algorithms. In the downstream task analysis, we use our base model with the ResNet-18 visual sub-network trained from scratch to focus on evaluating audio-visual features learned using spatial audio cues rather than semantic cues.

First-order ambisonics (FOA) extends stereo audio to the 3D setting, with extra channels to capture sound depth and height at time $t$: $a(t)=(a_{w}(t),a_{y}(t),a_{z}(t),a_{x}(t))$, where $a_{w}(t)$ represents omnidirectional sound pressure, and $(a_{y}(t),a_{z}(t),a_{x}(t))$ are front-back, up-down, and left-right sound pressure gradients respectively. FOA is often provided with 360-degree video to give viewers a full-sphere surround image and sound experience. Analogous to the case of stereo audio, we can train a model to detect whether the visual and audio streams are spatially aligned in 360-degree videos. To generate misaligned examples, we propose the transformation, $\tilde{a}(t)=(a_{w}(t),a_{x}(t)\sin{\theta}+a_{y}(t)\cos{\theta},a_{z}(t),a_{x}(t)\cos{\theta}-a_{y}(t)\sin{\theta})$ which rotates the audio about the z-axis by $\theta$.

We use the same model architecture as for field-of-view video and stereo audio (depicted in Figure 3). The main difference is that the input to the audio sub-network has the four FOA channels instead of two stereo channels. To train our model, we use the YouTube-360 dataset, which contains over a thousand 360-degree videos with FOA audio [25]. Our model achieves about 60% classification accuracy on the YouTube-360 test set; see Supplemental for details.

Does the audio sub-network trained on the audio-visual spatial correspondence task learn a strong representation of 360-degree surround sound? We first investigate whether the audio embeddings extracted from held-out data contain 360-degree spatial audio cues. We observe that the audio features are strongly correlated with the directional energy of the sound based on an unsupervised projection using principal component analysis. Similar to Section 5.1, we bin the embeddings based on their first two principal components and project the bin over the horizontal range of the video to track sound sources in the video using our self-supervised audio features. We show qualitative results in Figure 4(c): notice that a moving vehicle is tracked from left to right.

To determine whether the learned spatial audio cues provide a quantitative boost on spatial audio localization tasks, we evaluated our audio sub-network on the Tau Spatial Sound dataset [2]. This dataset consists of 400 minute-long recordings of multiple sound events synthetically up-mixed to 36 different directions along the X-Y plane (every 10 degrees). Our objective is to predict the direction-of-arrival (DOA) of sound events given FOA audio input. We first compared a baseline model trained from scratch with one that was initialized using the weights from our pretrained audio sub-network. We found that our pretrained weights provided a notable boost over the baseline, using the error in the azimuth angle as the evaluation criteria (Table 6). Next, we extracted the embeddings of the Tau Spatial Sound audio clips using our pretrained audio sub-network and performed classification using a linear SVM, providing only one example of an event from each azimuth angle (one-shot learning). Even in this case, we were able to predict the direction of arrival with a mean error of only about 30 degrees. These results suggest that the spatial information extracted using our self-supervised task are useful for acoustic localization.