AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization

Given a video sequence $\mathbf{S}$ of duration $T$, a set of non-overlapping video segments $\mathcal{V}=\{V^{(1)},V^{(2)},\dots,V^{(T)}\}$, and synchronized audio segments $\mathcal{A}=\{A^{(1)},A^{(2)},\dots,A^{(T)}\}$ of duration $t$ are extracted. Each video segment ${V}^{(i)}=\{{{v_{k}}^{(i)}}\}_{k=1}^{P}$ consists of $P$ image frames and the corresponding audio segment ${A}^{(i)}=\{{{a_{m}}^{(i)}}\}_{m=1}^{Q}$ consists of $Q$ samples, respectively. If the $i^{th}$ audio-video segment pair represents an event, it is labeled either as event $(e^{(i)}=1)$ or background $(e^{(i)}=0)$. Along with the generic event/background label, each segment of the entire video is labeled with a particular event category. Hence, the set of one-hot encoded labels¹¹1One-hot encoding changes the dimension over vector space over $\mathbb{R}$. for the video sequence across $C$ categories is denoted by, $\mathbf{Y}=\{y^{(i)}\}_{i=1}^{T}\in\mathbb{R}^{T\times C}$. For example, let consider $\mathbf{Y}=\{b,b,b,e_{2},e_{2},e_{2},e_{3},e_{3}\}$ where $b$ represents background and $e_{2},e_{3}$ denote an event of class $2$ and $3$, respectively. Here, we utilize the class labels ($y^{(i)}$) to generate event label ($e^{(i)}$) as $\{0,0,0,1,1,1,1,1\}$ to distinguish between event $(1)$ and background $(0)$.

We extract positive and negative audio-image pairs from the target dataset where a positive pair corresponds to the same AVE and a negative one represents a mismatch. Initially, we start with the pre-trained Audio and Image encoders from AudioCLIP [12]. Afterwards, we intiate fine-tuning on the extracted audio-image pairs utilizing the InfoNCE loss $L_{\text{InfoNCE}}=L_{I}+L_{A}$, where $L_{I}$ represents the image-to-audio matching loss and $L_{A}$ represents the audio-to-image matching loss. $L_{I}$ is given by

$$L_{I}=-\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{B}I_{ij}\text{log}\frac{\text{exp}(({\mathbf{z}_{i}^{I}}^{\text{T}}\mathbf{z}_{j}^{A})/\tau)}{\sum_{k=1}^{B}\text{exp}(({\mathbf{z}_{i}^{I}}^{\text{T}}\mathbf{z}_{j}^{A})/\tau)}$$

(1)

where $B$ denotes total number of audio-image pairs in a batch, $\mathbf{z}_{i}^{A}$, $\mathbf{z}_{i}^{I}$ represent normalized audio and image features of $i^{\text{th}}$ pair, respectively, $I_{ij}$ represents the identity matrix element with $I_{ii}=1,\forall i=j$ and $I_{ij}=0,\forall i\neq j$, and $\tau$ is a trainable temperature. Similarly, we construct the audio-to-image matching loss $L_{A}$.

The fine-tuned audio and image encoders are deployed to extract the features from the whole video sequence $S$. To generate feature map $\mathbf{v}^{i}$ from each video segment $\mathcal{V}^{(i)}$ containing $K$ image frames, we take the mean of feature maps $\mathbf{z}_{k}^{I}\ \text{for}\ k=1,\dots,T$. Afterwards, all feature maps from $T$ video segments are concatenated to generate the video feature ${s_{v}}$ of a particular sequence $S$. Similarly, audio feature of each segment $\mathbf{a^{i}}$ are concatenated to generate audio feature $\mathbf{s_{a}}$ of a video sequence, and thus:

	$\displaystyle\mathbf{s_{v}}=\mathbf{v}^{1}\oplus\mathbf{v}^{2}\oplus\dots\oplus\mathbf{v}^{T}$		(2)
	$\displaystyle\mathbf{s_{a}}=\mathbf{a}^{1}\oplus\mathbf{a}^{2}\oplus\dots\oplus\mathbf{a}^{T}$		(3)

where $\oplus$ denotes feature concatenation and $T$ denotes the number of segments in a sequence.

For better discrimination of the local feature variations particularly at the event transition edges, it is required to fuse multi-modal features over short temporal windows. However, the general context of the entire video is essential for better event classification. The proposed Multi-Window Temporal Fusion (MWTF) module effectively solves this issue by incorporating multi-domain attention over various temporal scales of the entire video, an approach that addresses the impact of both local and global variations (Figure  3).

Initially, to re-scale the video feature representation ($\mathbf{s_{v}}$) in accordance with the audio representation ($\mathbf{s_{a}}$) for early fusion, we adopt the Audio-Guided Video Attention (AGVA) module presented before [24, 37, 7]. The re-scaled video feature $\mathbf{s_{v,a}}\in\mathbb{R}^{T\times 1024}$ and corresponding audio feature $\mathbf{s_{a}}\in\mathbb{R}^{T\times 1024}$ are processed with separate bidirectional long short term memory (BiLSTM) layers for generating $\mathbf{v}\in\mathbb{R}^{T\times 256}$ and $\mathbf{a}\in\mathbb{R}^{T\times 256}$, respectively. Afterwards, temporal aggregation of audio-visual features is carried out to generate $\mathbf{F}\in\mathbb{R}^{T\times 512}$.

In the MWTF module, we incorporate $N$ sub-modules that operate on different timescales depending on the window length, $\{w_{i}\in\mathbb{R}\}_{i=1}^{N}$. The basic operations in each sub-module are divided into three stages: split, fusion, and aggregation. In the split phase of the $i^{th}$ sub-module, the aggregated feature $\mathbf{F}$ is segmented into $c_{i}$ blocks based on window length $w_{i}(w_{i}\times c_{i}=T)$ that generates $\{{F_{i}^{(k)}\in\mathbb{R}^{w_{i}\times 512}}\}_{k=1}^{c_{i}}$, and thus,

$$F_{i}^{(k)}=\mathbf{F}[w_{i}(k-1):w_{i}k];\ \forall k\in\{1,2,\dots,c_{i}\}$$

(4)

In addition, it is possible to use varying window lengths in a sub-module totaling the number of time steps $T$.

Following the split action, the multi-domain attention guided fusion operation is carried out on each $k^{th}$ feature block $F_{i}^{(k)}$ of the $i^{th}$ sub-module. The multi-domain attention operation is illustrated in Figure 4. Considering the two-domain distribution of each block $F_{i}^{(k)}$, we introduce joint temporal (TA) and feature attention (FA) mechanisms by reusing the weights of similar transformations.

Firstly, each block of $F_{i}^{(k)}\in\mathbb{R}^{w_{i}\times 512}$ features is transformed to generate query vector $Q_{i}^{(k)}\in\mathbb{R}^{w_{i}\times d}$, key vector $K_{i}^{(k)}\in\mathbb{R}^{w_{i}\times d}$, and value vector $V_{i}^{(k)}\in\mathbb{V}^{w_{i}\times d}$, such that,

	$\displaystyle Q_{i}^{(k)}=\mathbf{W_{2}}(\sigma_{1}(\mathbf{W_{1}}(Norm(F_{i}^{(k)})))$		(5)
	$\displaystyle K_{i}^{(k)}=\mathbf{W_{4}}(\sigma_{1}(\mathbf{W_{3}}(Norm(F_{i}^{(k)})))$		(6)
	$\displaystyle V_{i}^{(k)}=\mathbf{W_{6}}(\sigma_{2}(\mathbf{W_{5}}(Norm(F_{i}^{(k)})))$		(7)
	$\displaystyle\forall i\in\{1,\dots,N\},\ k\in\{1,\dots,c_{i}\}$

where $\mathbf{W_{1}}\in\mathbb{R}^{d\times 256}$, $\mathbf{W_{2}}\in\mathbb{R}^{d\times d}$, $\mathbf{W_{3}}\in\mathbb{R}^{d\times 256}$, $\mathbf{W_{4}}\in\mathbb{R}^{d\times d}$, $\mathbf{W_{5}}\in\mathbb{R}^{d\times 256}$, $\mathbf{W_{6}}\in\mathbb{R}^{d\times d}$, and $\sigma_{1}(\cdot)$, $\sigma_{2}(\cdot)$ denote the $tanh$ and $Relu$ activation, respectively.

Afterwards, we process each query $Q_{i}^{(k)}$ and key $K_{i}^{(k)}$ vectors on the temporal and feature domains to generate $\beta_{t,i}^{(k)}\in\mathbb{R}^{w_{i}\times w_{i}}$ and ${\beta_{f,i}}^{(k)}\in\mathbb{R}^{d\times d}$, respectively, such that

	$\displaystyle\beta_{t,i}^{(k)}=(QK^{T})_{i}^{(k)}/\sqrt{d}$		(8)
	$\displaystyle\beta_{f,i}^{(k)}=(KQ^{T})_{i}^{(k)}/\sqrt{w_{i}}$		(9)
	$\displaystyle\forall i\in\{1,\dots,N\},\ k\in\{1,\dots,c_{i}\}$

Then, we generate the temporal attention map $A_{t,i}^{(k)}\in\mathbb{R}^{w_{i}\times w_{i}}$ by applying $softmax$ over the rows ($axis=1$) of $\beta_{t,i}^{(k)}\in\mathbb{R}^{w_{i}\times w_{i}}$. In addition, the feature attention map $A_{f,i}^{(k)}\in\mathbb{R}^{d\times d}$ is generated by applying $softmax$ over the columns ($axis=0$) of $\beta_{f,i}^{(k)}\in\mathbb{R}^{d\times d}$.

These multi-domain attention maps are sequentially applied on each axis of $V_{i}^{(k)}$ to generate the modified feature map $\mathcal{F}_{i}^{(k)}\in\mathbb{R}^{w_{i}\times d}$ by,

	$\displaystyle\mathcal{F}_{i}^{(k)}=A_{t,i}^{(k)}(V_{i}^{(k)}A_{f,i}^{(k)})$		(10)
	$\displaystyle\forall i\in\{1,\dots,N\},\ k\in\{1,\dots,c_{i}\}$

Finally, in the aggregate phase, the modified feature maps of each $i^{th}$ block in a sub-module are temporally concatenated to generate $G_{i}\in\mathbb{R}^{T\times d}$ such that,

$$G_{i}=(\mathcal{F}_{i}^{(1)}||\mathcal{F}_{i}^{(2)}||\dots||\mathcal{F}_{i}^{(c_{i})});\ \forall i\in\{1,\dots,N\}$$

(11)

where ‘$||$’ denotes the feature concatenation along temporal axis.

Afterwards, all the modified feature maps from each sub-module are concatenated along channel axis maintaining temporal relation to generate $\mathcal{O}\in\mathbb{R}^{T\times Nd}$ as,

$$\mathcal{O}=G_{1}\oplus G_{2}\oplus\dots\oplus G_{c_{i}}$$

(12)

where $\oplus$ denotes feature concatenation along channel axis.

As the background class represents miss-aligned audio-visual pairs from all other classes, it often becomes difficult to distinguish them from event classes in case of subtle variations. To enhance the contrast between the event-segments and backgrounds, we introduce supervised event-guided temporal attention (EGTA) over the event region. Following EGTA, we refine the contrasted event segments with an additional stage of single window fusion for better discrimination of the event categories.

To generate EGTA $\alpha_{att}\in\mathbb{R}^{T\times 1}$, the fusion vector $\mathcal{O}$ is passed through a BiLSTM module as,

	$\displaystyle\gamma=BiLSTM(Norm(\mathcal{O}))$		(13)
	$\displaystyle{\alpha_{att}}=\sigma_{3}(\mathbf{W_{7}}(\gamma))$		(14)

where $\gamma\in\mathbb{R}^{T\times d^{\prime}}$, $\mathbf{W_{7}}\in\mathbb{R}^{1\times d^{\prime}}$, and $\sigma_{3}(\cdot)$ represents the $sigmoid$ activation function.

Afterwards, in the refining phase, we apply the EGTA mask ${\alpha_{att}}$ over the fusion vector $\mathcal{O}$ to generate $\mathcal{O^{\prime}}\in\mathbb{R}^{T\times Nd}$ by,

$$\mathcal{O}^{\prime}=\mathcal{O}\odot(\alpha_{\text{att}}\textbf{\em 1})$$

(15)

where $\textbf{\em 1}\in\mathbb{R}^{1\times Nd}$ denotes a broadcasting vector with all ones, and $\odot$ represents the element-wise multiplication.

As a last step, we incorporate a single window fusion with window length $w=T$ to generate $\mathcal{O}_{w}\in\mathbb{R}^{T\times d}$ by refining event-concentrated vector $\mathcal{O^{\prime}}$. Finally, we obtain the final event category prediction ${y}^{p}\in\mathbb{R}^{T\times C}$ after applying another sequential BiLSTM layer as,

$${y^{p}}=softmax(\mathbf{W_{8}}(BiLSTM(Norm(\mathcal{O}_{w}))))$$

(16)

where $\mathbf{W_{8}}\in\mathbb{R}^{C\times d}$ for $C$ categories of AVE.

To guide the event-attention $(\alpha_{att})$ for refinement, we use an event label loss $\mathcal{L}_{e}$. Also, for the multi-class prediction $y^{p}$, an event category loss $\mathcal{L}_{c}$ is incorporated. The total loss ${L}$ is obtained by combining $\mathcal{L}_{e}$ and $\mathcal{L}_{c}$ as,

$\displaystyle{L}=\lambda_{1}\overbrace{\mathcal{L}_{BCE}{(\alpha_{att},E)}}^{\mathcal{L}_{e}}+\lambda_{2}\overbrace{\mathcal{L}_{CE}{(y^{p},y)}}^{\mathcal{L}_{c}}$

(17)

where $\lambda_{1}$, $\lambda_{2}$ denote weighting factors, ${E}\in\mathbb{R}^{T\times 1}$ denotes the binary event label, and ${y}\in\mathbb{R}^{T\times C}$ denotes the one-hot encoded multi-class event categories over the time-frame.

Due to the sequential nature of AVEs, it is expected that they have high locality over the entire video frame. Therefore, AVEs are typically clustered together and isolated non-AVEs can be viewed as anomalies. We exploit this property to filter the generated event prediction ${y^{p}}$ during inference for obtaining the final prediction $y_{f}$. Here, we consider a window length $W$ to represent the minimum number of consecutive predictions required for considering any change as an anomaly. As such, all non-matching $y^{p}$ values are corrected according to the prevailing one.

AVE-CLIP is compared against several state-of-the art methods in Table 1. The multi-modal approaches outperform the uni-modal ones by a great margin. This is expected given the richer context provided by the multi-modal analysis over the uni-modal counterpart. The multi-modal audio-visual fusion strategies play a very critical role in the AVE localization performance. Traditionally, various audio-visual co-attention fusion schemes have been explored to enhance the temporal event features that provide comparable performances. Recently, Lin et al. [17] introduced a transformer based multi-modal fusion approach that incorporates instance-level attention to follow visual context over consecutive frames. However, the proposed AVE-CLIP architecture achieves the best performance with an accuracy of 83.7% that outperforms the corresponding transformer based approach by 6.9%. Moreover, the AVE-CLIP provides 5.9% higher accuracy compared to the best-performing co-attention approach proposed by Zhou et al. [37].

To analyze the effect of individual modules in proposed AVE-CLIP, we carried out a detailed ablation study over the baseline approach. The final AVE-CLIP architecture integrates the best performing structures of the building blocks.

The qualitative performance of the proposed AVE-CLIP is demonstrated in Figure 5 for two audio-visual events. For comparative analysis, we have shown the performance of the PSP model ([37]) as well. In the first event, the AVE represents a moving helicopter. Though the helicopter is visible in the first frame, it is a background event due to the absence of the flying helicopter sound. Only the middle three frames capture the AVE through audio-visual correspondence. Our proposed method perfectly distinguishes the helicopter event whereas PSP ([37]) fails at the challenging first frame. The second event representing a person playing the violin is very challenging given that the violin is hardly visible. Though the sound of violin is present throughout, the image of violin is visible in only few frames that represents the AVE. The PSP ([37]) method generates some incorrect predictions at event transitions. However, the proposed AVE-CLIP perfectly distinguishes the event frames, which demonstrates its effectiveness for generalizing local variations. Furthermore, AVE-CLIP achieves better performance in many challenging cases that demands different scales of temporal reasoning throughout video.