Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning

With the development of the deep learning [7, 8, 9, 10, 11] in recent years, audio-visual zero-shot learning has gained significant attention due to its potential applications in various domains such as violence detection [12], aerial scene recognition [13], speech recognition [14, 15] and video classification [16, 17, 18, 19, 20, 21, 22, 23, 24, 25]. MARBLE [26] provides a comprehensive benchmark for evaluating AI in music understanding, addressing the need for deep music representations, large-scale datasets, and a universal community-driven standard. IcoCap [27] improves video captioning by using easily-learned image semantics to diversify video content, helping captioners focus on relevant information. Finsta [28] enhances video-language models by using fine-grained spatio-temporal alignment with scene graph structures, improving performance on various tasks without needing to retrain from scratch. Chen et al. [29] propose Co-Meta Learning, which improves self-supervised speaker verification by leveraging complementary audio-visual information and updating network parameters using a disagreement strategy and meta learning. MAViL [30] uses three forms of self-supervision to learn audio-visual representations, achieving state-of-the-art performance in audio-video classification and enhancing both multimodal and unimodal tasks. SEEG [31] generates semantic-aware gestures by decoupling semantic-irrelevant information and leveraging semantic learning, outperforming other methods in semantic expressiveness and evaluations on various benchmarks. Hong et al. [4] incorporate a novel loss function that aligns video and audio features in the hyperbolic space, along with exploring the use of multiple adaptive curvatures for hyperbolic projections. Gowda et al. [32] discuss the issue of invalidation of the zero-shot setting in action recognition due to the overlap between classes in the pre-training and the evaluation datasets. They also highlight similar issues in few-shot action recognition and provide their splits for future evaluation in the field. Narayan et al. [33] incorporate a feedback loop from a semantic embedding decoder to refine the generated features iteratively. These synthesized features, along with their corresponding latent embeddings, are then transformed into discriminative features and utilized for classification, reducing ambiguities between categories. Wu et al. [34] propose the Dual Attention Matching (DAM) module, which spans longer video durations to better model high-level event information and captures local temporal details using a global cross-check mechanism. MA [35] focus on improving weakly-supervised audio-visual video parsing by using cross-modal correspondence and contrastive learning to generate reliable event labels and address audio-visual asynchrony. Wu et al. [36] propose the switchable LSTM framework to manage the generation and retrieval of nouns from external knowledge. Our proposed knowledge slots are primarily used for cross-modal fusion and semantic reasoning across different types of data (audio and visual). In contrast, the external knowledge in [36] is specifically tailored for enhancing language models by incorporating external visual knowledge for novel object captioning. Yang et al. [37] provides a comprehensive framework for multiple knowledge representations, which is crucial for understanding the integration of different modalities in ZSL. Yan et al. [38] introduce a semantics-guided approach for zero-shot learning, which aligns closely with the temporal-semantic integration. Li et al. [5, 39] first demonstrate the potential of SNN in audio-visual zero-shot learning. By efficiently extracting temporal information from different modalities using SNN, they achieved significant improvements in ZSL. However, due to the output heterogeneity between SNN and Transformer, their model’s performance on seen classes tends to decline. Therefore, how to relieve this challenge is the key to release the potential of audio-visual ZSL.

Spiking Neural Networks (SNNs) are a biologically-inspired models that have time-evolving states [40, 41]. Unlike traditional neural networks that use continuous-valued activations, SNNs communicate through discrete spikes, which are analogous to action potentials in biological neurons. Each neuron in SNN receives input from neighboring neurons and generates a spike when the combined signals surpass a certain threshold. The precise timing of these spikes is crucial as it corresponds to the timing of action potentials in biological neurons. Spikes are transmitted between neurons through synapses, which have weights that determine the strength of the connections. The correlation between pre- and post-synaptic spikes is commonly used to train an SNN by adjusting the synaptic weights. In SNNs, information is encoded in the precise timing of spikes, allowing them to capture the temporal dynamics of data. The timing of spikes carries important information about the input, and the interactions between neurons are determined by the arrival times of these spikes. Currently, a significant number of researchers have been studying the intrinsic nature of SNNs, including attention mechanisms [42, 43, 44], deep SNNs [45, 46, 47, 48], and simulations of biological visual pathways [49]. In addition to these investigations, SNNs have found wide-ranging applications in various fields, such as image classification [50, 51], speech recognition [52, 53], object detection [54, 55], and multimedia learning [56, 5].

Unlike existing SNNs in the field of multimodal learning, we have specifically designed our SNN for the audio-visual domain. Firstly, we recognize the importance of temporal encoding by leveraging the information from different time steps in the SNN. We propose a time step factor (TSF) to dynamically fuse the outputs from different time steps. Additionally, to reduce the spiking noise in the SNN output and enhance the model’s robustness, we introduce a global-local pooling (GLP) to improve the overall performance and stability of the SNN by combining the max and average pooling operations.

Our SNN network consists of three convolution SNN blocks, each comprising a convolution operation layer followed by a LIF-based layer [58]. The LIF model consists of an integration phase, where the neuron accumulates input currents, and a firing phase, where the neuron generates a spike and resets its membrane potential. Specifically, the dynamics of a LIF neuron can be described by the following equation:

where $\tau_{m}$ is the membrane time constant, $\boldsymbol{V}(t)$ is the membrane potential at time $t$, $R$ is the membrane resistance, and $\boldsymbol{I}(t)$ is the current input at time $t$. To compute the input of the $i$-th LIF neuron $\boldsymbol{I}_{i}(t)$, we calculate the convolution operation and batch normalization with the output of the previous layer $\boldsymbol{P}(t)$ as:

where $\boldsymbol{W}_{p}$ represents the weight matrix, $\mathrm{BN}(\cdot)$ represents the batch normalization and $\mathrm{CONV}(\cdot)$ represents the convolution operation. When the membrane potential reaches a threshold value $v_{th}$ would generate a spike, and the membrane potential is reset to a reset potential $V_{rest}$.

To optimize the distribution of input features before processing by the LIF neurons, we propose the GLP as shown in Fig. 3. The max pooling operation captures the global maximum value of the input features, which represents the overall variation in the input distribution. The average pooling operation calculates the average value of the input features, which highlights the importance of local salient regions. By combining the outputs of max and average pooling, the GLP module provides guidance for generating the input features based on the global and local characteristics. The overall process can be written as follows:

where $\boldsymbol{P}_{max}$ and $\boldsymbol{P}_{avg}$ are corresponding to the max and average pooling features, and $\beta$ is the learnable items. Indeed, the relationship between the output of the SNN and the corresponding time steps is crucial. Effectively utilizing the outputs from different time steps can significantly influence the final performance. A common method is to assign equal weights to each time step and compute the average output across all time steps to obtain the final result. However, this method overlooks the diversity among different time steps. To deal with this, we propose the TSF which adjust the weights of SNN outputs in different time steps dynamically. By considering the importance of each time step, the SNN can effectively capture the temporal dynamics and encode relevant information at different time scales. The final output of the spatial-temporal SNN can be summarized by the following equations:

where $max(\cdot)$ returns the largest item of the input and $c\in(a_{t},v_{t})$. This dynamic adjustment of weights allows the SNN to adaptively emphasize the contributions of different time steps based on their significance, leading to obtain more fine-grained representation of temporal sequences.

To reduce the spiking noise, we dynamically adjusting the threshold of the LIF neurons based on the current output of the SNN and the pooling matrix. Specifically, we use the entropy of the current SNN output to represent the amount of information contained in the features. If the information content is rich, it indicates a more informative scene, and we need to increase the threshold to suppress spike noise. The threshold adjustment for the audio and visual modalities of the SNN can be expressed as follows:

where $\mathcal{N}(\cdot)$ represents the normalization operation.

In this paper, we utilize both the spatial-temporal SNN and LSR module to extract temporal and semantic information, respectively. However, these two types of network outputs have significantly different data distributions: one is a binary sequence, while the other is a floating-point feature. It poses a challenge to effectively fuse these outputs while preserving the complex and high-level interactions. A powerful solution for feature fusion that has been recently proposed is bilinear modeling, which can encode fully parameterized bilinear interactions. First, the semantic and temporal features in each modality need to be projected into embedding vectors as $\boldsymbol{R}_{a}\in\mathbb{R}^{d_{as}}$, $\boldsymbol{R}_{v}\in\mathbb{R}^{d_{vs}}$, $\boldsymbol{S}_{a}\in\mathbb{R}^{d_{at}}$ and $\boldsymbol{S}_{v}\in\mathbb{R}^{d_{vt}}$, respectively. The bilinear model in visual pipeline can be written as:

where $\boldsymbol{\mathcal{T}_{a}}\in\mathbb{R}^{d_{as}\times d_{at}\times K_{a}}$ and $\boldsymbol{\mathcal{T}_{v}}\in\mathbb{R}^{d_{vs}\times d_{vt}\times K_{v}}$ represent the full tensors and $\times_{i}$ represents the $i$-mode product. However, the parameters in full tensor $\boldsymbol{\mathcal{T}_{c}}$ could be very large. Here, we propose the temporal-semantic tucker fusion to factorize the full tensor $\boldsymbol{\mathcal{T}_{c}}$ following [59]. The decomposition of full tensor $\boldsymbol{\mathcal{T}}$ could be defined as :

where $\boldsymbol{\mathcal{G}}$ is the core tensor, $\boldsymbol{U}^{(s)}\in\mathbb{R}^{d_{s}\times n_{a}}$, $\boldsymbol{U}^{(t)}\in\mathbb{R}^{d_{t}\times n_{v}}$ and $\boldsymbol{U}^{(k)}\in\mathbb{R}^{K\times n_{k}}$ are the factor matrices. We could utilize the tensor decomposition to factorize the full tensor $\boldsymbol{\mathcal{T}_{a}}$ and $\boldsymbol{\mathcal{T}_{v}}$, and rewrite the Eq. (10) as follows:

We can perform bilinear interaction to capture the complex relationships between the temporal and semantic information, and then project them into a lower-dimensional representation. This process is defined as as:

where $\widetilde{\boldsymbol{R}_{a}}=\boldsymbol{R}_{a}^{\top}\boldsymbol{U}^{s}_{a}\in\mathbb{R}^{n_{a}\times s_{a}}$, $\widetilde{\boldsymbol{R}_{v}}=\boldsymbol{R}_{v}^{\top}\boldsymbol{U}^{s}_{v}\in\mathbb{R}^{n_{v}\times s_{v}}$, $\widetilde{\boldsymbol{S}_{a}}=\boldsymbol{S}_{a}^{\top}\boldsymbol{U}^{t}_{a}\in\mathbb{R}^{n_{a}\times t_{a}}$ and $\widetilde{\boldsymbol{S}_{v}}=\boldsymbol{S}_{v}^{\top}\boldsymbol{U}^{t}_{v}\in\mathbb{R}^{n_{v}\times t_{v}}$.

After integrating the temporal and semantic features from different modalities, we propose a cross-modal transformer to further reason about the implicit feature correspondences within each modality. We establish residual connections between the two modalities, to capture the complementary information between them. Layer normalization is applied to mitigate the impact of feature variations. The cross-modal transformer contains a stack of standard transformer layers to obtains a joint temporal-semantic representation. The cross-modal transformer block in two modalities are shared weight which can be summarized as follow:

where $\mathrm{MHCA}(\cdot)$ represents the multi-head cross attention. The ultimate goal of our model is to predict the text category based on the audio-visual inputs. To project the joint audio-visual features into the same space as the text features, we construct the projection and reconstruction layers. The projection layer maps the audio-visual features to align with the text feature. the reconstruction layer helps to preserve the relevant information while discarding any noise or irrelevant details that may have been introduced during the projection. Both the projection and reconstruction layers have a similar structure, consisting of two linear layers $f_{3}^{m}:\mathbb{R}^{s_{in}*h_{emb}}\rightarrow\mathbb{R}^{s_{in}*h_{hid}}$ and $f_{4}^{m}:\mathbb{R}^{s_{in}*h_{hid}}\rightarrow\mathbb{R}^{s_{in}*h_{out}}$, followed by dropout regularization with rate $d_{proj}$. The final audio-visual joint feature embeddings can be obtained as:

where $\boldsymbol{Pro}_{av}(\cdot)$ is the projection function. The final textual labeled embedding $\boldsymbol{\mathcal{F}}_{tex}$ is obtained through the word projection layer $\boldsymbol{Pro}_{tex}$. The architecture of the $\boldsymbol{W}_{tex}$ is similar with $\boldsymbol{Pro}_{av}$ with dropout rate $d_{text}$.

The STFT is tranined using a Nvidia V100S GPU. The audio and visual embeddings are extracted using pretrained SeLaVi [57]. In STFT, we set $a_{in}=512$, $h_{emb}=512$, $h_{hid}=512$, $h_{out}=300$ and $h_{proj}=64$. In VGGSound, UCF, ActivitiNet datasets, the dropout rates are corresponding to $d_{enc}=(0.20,0.25,0.10)$, $d_{dec}=(0.25,0.20,0.15)$, and $d_{text}=(0.1,0.1,0.1)$, respectively. The cross-modal transformer is constructed with 8 heads, the dimension of each head is 64. We select Adam as training optimizer. STFT is trained 60 epochs with 0.0001 learning rate. To update parameters more effectively, STFT using the combination of triplet loss $\mathcal{L}_{t}$, projection loss $\mathcal{L}_{p}$ and reconstruction loss $\mathcal{L}_{r}$.

In this study, we conducted experiments and evaluated the proposed models using three benchmark datasets: ActivityNet, VGGSound, and UCF101. These datasets were chosen to provide a diverse range of audio-visual data and cover various domains, enabling a comprehensive evaluation of the proposed models’ performance. The statistics of these datasets are as follows: 1). ActivityNet contains a wide variety of human activities along with the corresponding videos. The dataset consists of approximately 200 different activity classes and more than 20,000 videos with an average duration of about 2 minutes per video. 2). UCF101 dataset consists of more than 13,000 videos collected from YouTube, with an average duration of around 7 seconds per video. The videos cover a wide range of human actions in various contexts and provide a challenging dataset for action recognition algorithms. 3). VGGSound dataset consists of more than 200 different classes and includes thousands of audio clips obtained from online sources.

In Table II, we demonstrate the superiority of proposed STFT compared to state-of-the-art (SOTA) methods. On the VGGSound dataset, STFT achieves significant improvements over TCaF with a 37.2% increase in HM and a 35.9% increase in ZSL scores. On the UCF101 dataset, STFT achieves an HM of 32.58 and a ZSL score of 29.72. Compared to the best current method MDFT [39], STFT achieves a 3.9% improvement in HM but experiences a slight decrease in ZSL. It’s worth noting that both our STFT and MDFT models employ SNN as temporal encoders. However, due to the different output data distributions between SNN and the Transformer used in MDFT, MDFT’s performance on Seen Classes is not satisfactory. To address this issue, we propose the temporal-semantic tucker fusion, which achieves a 15.7% improvement on Seen Classes. On the ActivitiNet dataset, STFT obtains an HM score of 15.38 and a ZSL score of 12.91, surpassing AVCA’s HM score of 12.13 and ZSL score of 9.13. Compared to AVMST, STFT achieves a 21% improvement in HM and a 24.5% improvement in ZSL. Overall, our STFT model demonstrates superior performance compared to existing methods on various evaluation metrics across the three datasets.

While the proposed method shows significant improvements on the VGGSound-GZSL dataset compared to others, it is important to note that the MDFT method requires data enhancement to convert RGB images to events, inherently introducing additional computational complexity. MDFT utilizes an Event Generative Model (EGM) to convert RGB images into event streams, eliminating background scene bias and capturing motion information. However, this conversion adds computational overhead, requiring the processing of high-resolution image data to generate events and the use of Spiking Neural Networks (SNNs) to handle the sparse event data efficiently. In contrast, our proposed method avoids this complexity by directly modeling the audio-visual data without converting RGB images to events. This design choice allows our method to maintain competitive performance across different datasets while being more computationally efficient.

Moreover, we observed a slight decline in ZSL performance on the UCF101 dataset. This decline can be attributed to the fixed rank constraint used in the semantic-temporal Tucker fusion module, which may not fully capture the complex temporal dynamics of the UCF101 dataset. To address this issue, we suggest dynamically adjusting the rank constraint based on the singular values of the input data in the future. Additionally, significant variations in activity patterns within the UCF101 dataset may introduce redundancy at higher time steps, negatively impacting ZSL performance. We suggest reducing redundancy through an optimized temporal encoding process and exploring different configurations of the time-step factor to enhance temporal feature integration.

As shown in Table X. “#params” represents the number of parameters, “GFLOPs ” represents the computational cost during training. Overall, our model exhibits strong performance in both the number of parameters and computational costs, while ensuring high classification effectiveness. Compared to MDFT, our model reduces the parameters by approximately 32%, and reduces the GFLOPS from 5.62 to 4.27.

We expand our methodology by incorporating features from audio and video classification networks into our model training and evaluation process. Specifically, we use C3D [65], a network pre-trained on the Sports1M [66] dataset for video classification, to extract visual features. For audio feature extraction, we use VGGish [67], pre-trained on the Youtube-8M [68] dataset. To create a unified feature representation for each video, we average these extracted features over time, resulting in a 4096-dimensional visual feature vector and a 128-dimensional audio feature vector.

To adjust the audio features derived from the Youtube-8M pre-trained network, we make changes to the dataset splits for VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL. We remove the test unseen classes that overlap with the Youtube-8M dataset, leading to modified dataset splits named VGGSound-GZSL^cls, UCF-GZSL^cls, and ActivityNet-GZSL^cls. More details on these adjustments are available in Table VIII.

Table XI presents the comparative results of our STFT model against the baseline models, using the audio and video classification features mentioned above. The STFT model shows exceptional performance across all datasets compared to the baselines. For example, in the VGGSound-GZSL^cls dataset, STFT achieves a Harmonic Mean (HM) of 10.08% and a Zero-Shot Learning (ZSL) accuracy of 8.79%, outperforming the TCaF model, which records an HM of 8.77% and a ZSL accuracy of 7.41%. In the UCF-GZSL^cls scenario, STFT reaches an HM of 51.14% and a ZSL accuracy of 49.74%, surpassing both AVCA and AVGZSLNet^cls, which show lower HMs and ZSL accuracies. Likewise, on ActivityNet-GZSL^cls, AVCA outperforms AVGZSLNet^cls in terms of HM and ZSL accuracy. These results highlight STFT’s consistent superiority over competing models, attributing this success to the innovative integration of our temporal-semantic tucker fusion module, which effectively combines SNN and Transformer outputs for improved multi-scale fusion and performance.

We use t-SNE visualization to demonstrate the advantages of the proposed STFT in exploring intrinsic correlations within multimodal data, as shown in the Fig. 4. In the UCF101 dataset, STFT actively clusters features from the same parent category and separates features from different parent categories. For example, features such as “basketball” and “basketballdunk,” which belong to the same parent category “sport,” are brought closer, while features like “PlayingCello” and “HandstandWalking” from the “instrument” category are separated. These visualizations illustrate how our method explores correlations between different types of data.

We demonstrate the qualitative comparison results with recent SOTA method MDFT in Fig. 5. MDFT focus on decoupling motion and background information, while our model coupling the outputs of SNN and Transformer effectively. In this paper, we address the challenges of time steps and spiking redundancy in SNN, and the output heterogeneity between SNN and Transformer. In sports classes with frequent changes in motion information, STFT demonstrate superiorities compared with MDFT due to the less spiking redundancy.

Although our model has demonstrated SOTA performance in HM on three benchmark datasets, we observed a slight decrease on ZSL in UCF101. This may caused by the fixed rank constraint. A potential solution could be dynamically setting the rank constraint of the current temporal-semantic tucker fusion based on the singular values of the input information. This adaptive adjustment strategy of rank constraint could potentially improve the ZSL performance.

The proposed Spiking Tucker Fusion Transformer (STFT) is designed for scalability, effectively handling larger datasets and complex audio-visual sequences. The STFT uses a temporal-semantic fusion module based on Tucker decomposition, enabling multi-scale fusion of SNN and Transformer outputs. This design ensures the number of parameters remains manageable, maintaining computational efficiency even with larger datasets. Efficient data loading and batching strategies are used to handle larger datasets, ensuring memory constraints are not exceeded and performance is maintained. Additionally, the STFT adapts to different audio-visual sequence complexities through dynamic adjustments, such as the TSF for synthesizing temporal information and GLP for reducing spike noise and enhancing robustness. These components enable the model to effectively manage sequence complexity, ensuring robust performance across diverse scenarios. Overall, the STFT’s design and components enhance its scalability and applicability in real-world scenarios involving large datasets and complex audio-visual sequences.