Curriculum Audiovisual Learning

As the filters of convolution networks have shown the property of class-relevant activation [32, 17], it becomes feasible to discover and disentangle the audio and visual components by analyzing and integrating their channel representations. Concretely, we first employ convolution networks to model each modality then embed the data into feature maps with size $H\times W\times C$, where $H$ and $W$ are the frame size and $C$ is the number of channels. Then, these feature maps are reshaped into a set of vector $\{{x_{1}},{x_{2}},...,{x_{H\times W}}|x_{i}\in R^{C}\}$. Due to the distinct activation of different modal components, some of the reshaped feature vectors in the $C$-dimension channel space should take similar distribution when they describe the same modal components but dissimilar for different components. Hence, we propose to integrate these feature vectors by performing soft K-means clustering [5] in the channel space for each modality.

Formally, the objective function of soft K-means clustering [5] can be formulated as

where $c_{j}\in R^{C},j=1,2,...,k$ is the $j-$cluster center, $k$ is the number of sound-sources in the audiovisual scene, $d\left({{x_{i}},{c_{j}}}\right)$ is the Euclidean distance between current feature point $x_{i}$ and center $c_{j}$ for measuring their similarity. $w_{ij}\in[0,1]$ is the indicator variable, which indicates the degree of assignments and can be achieved by performing softmax function over the distance of $d\left({{x_{i}},{c_{j}}}\right)$, i.e.,

where the hype-parameter $\beta>0$ is called stiffness parameter and controls the scalability of assignments.

Eq. 1 is a minimization problem about two parts, i.e., the assignments and centers. The Expectation-Maximization algorithm can be employed for solving it effectively [21]. In the E-step, we fix the cluster centers $C$ and update the assignment $w_{ij}$ via Eq. 2 . In the M-step, we fix the assignment $w_{ij}$ and re-compute the centers with the updated assignments from E-step, i.e.,

By alternatively executing E- and M-step, we aim to find $k$ centers, each of which should correspond to certain modal component, such as specific object or sound. Meanwhile, the corresponding cluster assignment can be interpreted as a spatial-mask over feature map and indicates the location of sound-source in both modalities, as shown in Fig. 2.

For a given audiovisual scene, although the contained sounds and objects have been described as different clustering centers, it is still difficult to directly perform alignment between them only with supervision at the entire scene level. Fortunately, the pervasive concurrency of sound and sound-maker can help to infer the latent alignment by comparing the matching degree of different sound-object pairs, where the valid pair should possess higher matching degree. Concretely, for each audio clustering center $c_{i}^{a}$, we aim to find proper visual clustering center (i.e., sound-maker) based on their concurrency, which can be formulated as

where $k_{a}$ and $k_{v}$ are the number of audio center and visual centers, respectively²²2As the visual background is usually irrelevant with sound, we set $k_{v}=k_{a}+1$ and use additional visual center to represent it.. By minimizing Eq. 4, each audio center is aligned to the nearest visual center, meanwhile we can also derive the similarity score of sounds and objects in the entire scene, i.e., $S_{av}$.

For an arbitrary scene, the self-supervision signal only confirms whether the audio and visual information are from the same scene (video) or not. To effectively leverage such supervision, we employ the contrastive loss to train the audiovisual network and infer the latent alignment simultaneously, which has shown the property of consistency and robust in two-stream network optimization [19, 20]. Concretely, the contrastive loss is written as

where $a_{i}$ and $v_{j}$ stand for the sound and image from scene $i$ and $j$, respectively. $\delta_{i=j}$ is an indicator of each sound-image pair, i.e., $\delta_{i=j}=1$ if $i=j$, otherwise $0$. In practice, the negative samples of $i\neq j$ are randomly sampled from the training set. Generally, Eq. 5 encourages the audiovisual network to have higher matching confidence for the aligned sound-image pair than the mismatched ones by introducing the hyper-parameter of $margin$.

Usually, the audiovisual scenes in the wild contain different amounts of sound-sources, we find that directly performing audiovisual learning with these data will make the model very difficult to optimize and also lower the alignment performance. To settle this problem, we propose to train the audiovisual model step by step, which is about starting from simple scene then gradually increasing the difficulty level, where the number of sound-sources is considered as the reference for audiovisual scene complexity. Intuitively, for the simple scene with single sound-source, it is easy to visually localize the sound-maker from background then align it to the unique sound, such as the example of accordion in Fig. 1. By contrast, we find that if training with complex audiovisual scenes (e.g., with three sound-sources) from the beginning, the model will get much lower convergence speed and worse results. While, model trained with simple scenes can contribute prior knowledge for distinguishing different sound-makers and sounds, and also provides the reference for alignment . Therefore, the audiovisual learning model can be further optimized with the complex scene, which leads to better results.

In practice, to effectively perform curriculum learning, all the audiovisual data have been sorted from simple to complex before training, according to the number of sound-sources in the scene. For different learning stages, the cluster number is accordingly set to the number of sources, e.g., $k_{a}=1$ and $k_{v}=2$ for the scene with single source. Based on these graded audiovisual data, we can train the audiovisual learning model in a curriculum fashion.

Since the audiovisual scene complexity is crucial for curriculum training, it is worth to learn to model and estimate the number of sound-source in a given scene. Formally, the discrete probability distribution of Poisson for counting data $y_{i}$ is given by $P({Y_{i}}={y_{i}})=\frac{{{e^{-{\lambda_{i}}}}\lambda_{i}^{{y_{i}}}}}{{{y_{i}}!}},{y_{i}}=1,2,...,$ where $\lambda_{i}$ is interpreted as the expected number of events in the interval and ${y_{i}}!$ is the factorial of $y_{i}$. In this task, $y_{i}$ is performed as the number of sound source in the audiovisual scene. Consequently, we propose to model $\lambda_{i}$ as a function of the input sound $a_{i}$ by the audio network, which is written as ${\lambda_{i}}=f(a_{i})$. The function $f(\cdot)$ means the counting network of sound-sources. By taking the negative log-likelihood w.r.t. the Poisson distribution, we can have the Poisson regression loss

where the term of ${\rm{ln}}({y_{i}}!)$ can be ignored, as it is a constant to the model training. After training the counting network, we can estimate the scene complexity by identifying the number that holds the maximal probability.

Considering that the audiovisual learning model has learned to align objects and sounds in the training phase, we can directly identify the potential object which produces given sound by comparing their similarity, i.e.,

For the clustering center of sound-sources $c_{source}^{a}$ , we compare it with all the visual centers $\{{c_{1}^{v}},{c_{1}^{v}},...,{c_{k_{v}}^{v}}\}$ and select the closest one as visual representation of sound source. As the corresponding assignment $w_{\cdot source}$ indicates the correlation between all the visual feature vectors and $c_{source}^{v}$ ,we can reshape it back to the size of $H\times W$ and regard it as the location mask of sound-maker to achieve visual localization. To better visualize the object position, we can also further resize the assignment to the size of input image.

To validate the effectiveness of inferred object representation further, we propose to perform sound separation based on visual guidance. The representative audiovisual separation network in  [10, 12] is adopted, as shown in Fig. 6. Concretely, the separation network takes the visual clustering center (i.e., $c_{i-source}^{v}$) as the sound-maker representation in $i-$scene, and targets to separate its produced sound from the mixed audio signal. Alternatively, we can also use the assignment $w_{\cdot source}$ to point out the location of sound-maker, and regard it as the object mask over the visual feature maps. Then, a sound-maker-awareness max-pooling can be performed over the masked feature maps to obtain robust object representation.

A variant of U-Net [25] is used to perform sound-source separation, similar to [12, 31]. The network takes the spectrogram of $i-$mixed sound $a_{i}^{mix}$ as the input, then encodes it into audio feature maps via stacked convolution layers. The replication and tiling operation is performed over the visual representation to match the size of embedded audio feature maps. Then we concatenate these two modalities, and feed them into stacked up-convolution layer to generate a spectrogram mask. The separation loss is written as

where $g(\cdot)$ means the audiovisual sound separation network and $M_{i}$ is the mask of the spectrogram magnitudes of the target sound $a_{i}$ and mixed sound ${{a}_{i}^{mix}}$ , i.e., ${M_{i}}={{{m^{{a_{i}}}}}\mathord{\left/{\vphantom{{{m^{{a_{i}}}}}{{m^{a_{i}^{mix}}}}}}\right.\kern-1.2pt}{{m^{a_{i}^{mix}}}}}$. With the masked spectrogram, we can use Inversed Short-Time Fourier Transform (ISTFT) to produce separated sound signal w.r.t. specific sound-maker.

In this section, we aim to have an insight into how the curriculum strategy influences the audiovisual learning performance. Concretely, to evaluate the effects of different audiovisual complexities, the original set and the curriculum set of $\mathcal{C}_{1}$ and $\mathcal{C}_{3}$ are selected for training the audiovisual network, respectively. As shown in Fig. 4, it is obvious that the network trained with the simple curriculum of $\mathcal{C}_{1}$ enjoys the fastest convergence and lowest training loss, while the one trained with $\mathcal{C}_{3}$ suffers from the worst performance. Such phenomena indicate that the model performance is significantly affected by the audiovisual complexity, and the simple scene can provide better learning performance.

Further, we want to know what the model can benefit from pre-curriculum, i.e., the effects of curriculum initialization. In Fig. 4, we show the training accuracy of audiovisual model on the $\mathcal{C}_{3}$ set, which are initialized from random and the model trained with the $\mathcal{C}_{2}$ set, respectively. Surprisingly, the model initialized from $\mathcal{C}_{2}$ enjoys the great advantages compared with the random one. Curriculum learning indeed helps to accelerate and improve the audiovisual learning performance.

The unimodal representation learned by audiovisual model is also influenced by the curriculum strategy. To assess such influence, we propose to perform acoustic scene classification by viewing the trained audiovisual model as a feature extractor. The ESC-50 dataset is chosen for evaluation and we follow the same pre-processing and train/test split as [17]. Table 1 show the comparison results, where the sound-source-level alignment is Eq. 5 and the scene-level alignment is directly comparing the audio and visual representation without clustering, similar to [1]. In Table 1, we can summarize these results into three points. First, learning with simple curriculum can provide more proper initialization. Second, sound-source matching better utilizes the audiovisual concurrency than scene-level matching, especially in the complex scene. Third, direct video-level matching in the complex scene may deteriorate the pre-trained network. This is probably because the chaotic audiovisual correlation could confuse the scene matching objective, but it was ignored before.

In this section, we evaluate the performance of audiovisual complexity estimation. Table 2 shows the Poisson regression results when training on AudioSet-Balanced-Train and testing on AudioSet-Evaluation, where the number of sound source ranges from 1 to 5. Compared to the chance results, the audio Poisson regression network has a great superiority in both accuracy and Mean Average Error (MAE). Moreover, the results can be further improved by adopting the network pre-trained with audiovisual learning objective, i.e., Eq. 5. Intuitively, if the network has been trained with higher level curriculum, e.g., $\mathcal{C}_{2}$, it will better estimate the number of sound-sources in the audio modality. Such evidences show that complex audiovisual data can be effectively modeled by self-supervised learning method, especially when adopting the curriculum learning strategy.

Table. 6 shows the sound separation results. We can find that our model shows superior performance over most compared methods, even some of them adopt additional visual knowledge. For example, AV-MIML [11] and Sound-of-Pixels [31] use ImageNet pre-trained model as the visual extractor. Besides, we also show the results of CAVL-$\mathcal{C}_{1}$. Such results are obtained by directly viewing the audio assignment in the audiovisual learning model as the spectrogram mask. However, as the audio assignment is achieved over the embedded feature maps, it is too coarse to perform fine-grained spectrogram prediction. Hence, such method does not work for sound separation.