Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition

Each training sample consists of a video clip paired with corresponding audio, visual narration and audio narration. The former is provided in the dataset we use [10], while we use Pengi [14] to generate the latter. While we adopt the approach from [10] of using separate frozen encoders for each modality to extract base features, our method differs by incorporating the audio modality and utilizing distinct encoders for each modality, instead of a single encoder for fused appearance and motion features. Specifically, we train (from scratch) separate encoders $f_{appearance}$, $f_{motion}$, and $f_{audio}$ to derive domain-generalizable features $ap$, $m$, and $a$ for appearance, motion, and audio. Additionally, we train a separate encoder $f_{text}$ extracts the features of visual narration $t$ and audio narration $\hat{t}$. See Fig. 2. The action prediction $\hat{y}$ is generated by the classifier $h$ based on the fused multimodal embedding. The cross-entropy loss $\mathcal{L}_{c}$ is computed between the true and predicted action labels, $y$ and $\hat{y}$.

Before training, we calculate consistency ratings for each video sample using a LLM [11], assessing the degree to which the audio and visual narrations correspond semantically. This information is used to control fusion. Computing consistency between audio and visual content through text captures abstract concepts that raw features might miss. While features emphasize low-level details, text-based evaluations ensure that the audio and visual content correspond at a deeper, conceptual level.

We use the following prompt obtain consistency ratings:

Rate the consistency between two narrations from the same video out of 100. The first narration describes the visual aspect, and the second describes the audio. Consider how well the audio narration overlaps with and complements the visual narration. Output only the percentage score.

The consistency ratings $r$ are then used as weights to modulate the contribution of the audio embeddings $a$ before the concatenation of appearance, motion, and audio features, as shown in Fig. 2. Specifically, only during training, the audio embeddings are scaled by the consistency rating $r$, such that $a=a*r$. This consistency-weighted audio approach ensures that audio information with strong semantic alignment to the visual content exerts a greater influence on the final prediction. This approach enhances the quality of audio representations during training, minimizing the impact of noisy or irrelevant audio cues and improving the overall robustness.

Aligning visual features with narrations enriches the model with domain-invariant, human-like understanding, enhancing its ability to generalize across domains [13]. As discussed previously, we generate audio narrations that are specifically tied to the audio content. The LLM in audio captioner Pengi [14] mimics human-like understanding, providing audio-specific, semantically rich descriptions. Since Pengi uses a separate encoder for audio feature extraction, its generated audio narrations are initially not aligned with our model’s audio features $a$. Thus, in our approach, we align audio features $a$ with these audio narration features $\hat{t}$, while aligning appearance features $ap$ and motion features $m$ with visual narration features $t$ using contrastive learning.

Given a batch of samples $\mathcal{B}=\{(ap_{i},m_{i},a_{i},t_{i},\hat{t}_{i})\}^{B}_{i=1}$, we frame the alignment as noise contrastive estimation [18]. Specifically, $\mathcal{L}_{{ap}\rightarrow{t}}$ treats the appearance $ap_{i}$ as the anchor, with other narrative texts serving as negatives, and minimizes:

$$\mathcal{L}_{{ap}\rightarrow{t}}=-\dfrac{1}{|\mathcal{B}|}\sum^{B}_{i}log\dfrac{exp(s(ap_{i},t_{i})/\tau)}{\sum^{B}_{j}exp(s(ap_{j},t_{j})/\tau)}$$

(1)

where $s(\cdot,\cdot)$ is the cosine similarity and $\tau$ is a learnable temperature. In a similar manner, $\mathcal{L}_{{t}\rightarrow{ap}}$ uses the text $t_{i}$ as the anchor, with other appearance features as negatives. These losses are then combined to create the appearance-text alignment loss $\mathcal{L}_{ap,t}=\mathcal{L}_{{t}\rightarrow{ap}}+\mathcal{L}_{{ap}\rightarrow{t}}$. Likewise, motion-text alignment $\mathcal{L}_{m,t}$ and audio-text alignment $\mathcal{L}_{a,\hat{t}}$ are computed, and all these alignment losses are aggregated into a total alignment loss $\mathcal{L}_{align}=\mathcal{L}_{ap,t}+\mathcal{L}_{m,t}+\mathcal{L}_{a,\hat{t}}$.

To form the overall training objective, we combine the alignment loss with the cross-entropy classification loss:

$$\mathcal{L}=\mathcal{L}_{c}+\lambda\mathcal{L}_{align}$$

(2)

where $\lambda=0.1$ is used to weight the alignment loss.

In Table 1, the mean percentage drops in parentheses indicate the extent of domain shifts across various domains, comparing modality settings where the training set either contains samples that share location and/or scenario with the test domain, or share neither. The mean performance drop for motion features is 25.8%, while audio exhibits a drop of 32.7%. In contrast, appearance shows a significantly higher shift (54.8%). This aligns with our hypothesis that temporal dynamics—such as consistent patterns of movement and continuity of sound—remain more stable across different environments and scenarios. In contrast, spatial semantics represented by appearance features, are more variable due to differences in objects, backgrounds, and other visual elements that can vary significantly from one domain to another. Moreover, the multimodal approach, which integrates audio, motion, and appearance, demonstrates a reduced shift with a mean drop of 42.8% compared to appearance alone (54.8%). Our analysis underscores the critical role of incorporating audio and motion features for robust domain generalization.

Table 2 compares our proposed method, the state-of-the-art CIR [10] approach, and a baseline model on the ARGO1M dataset. The baseline is trained solely with cross-entropy loss, whereas CIR enhances its performance through cross-instance reconstruction guided by text. Notably, although the original CIR results did not include audio, we have extended their approach by incorporating audio features and multiple trained encoders $f$ for each modality to ensure a fair comparison with our method. Our method consistently outperforms CIR across all domain splits, except for Co-JPN, achieving up to a 1.7% improvement and a 1.2% higher average performance overall. Additionally, our method outperforms the baseline by an average of 4.8%.

Table 3 illustrates the impact of different modality settings on performance. In the ‘Ap-Mo’ setting used by CIR [10], appearance and motion features are fused early and processed through a single encoder. We propose an alternative approach in the ‘Ap, Mo’ setting, where separate encoders ($f_{appearance}$, $f_{motion}$) are employed to learn distinct domain-generalizable features, particularly because motion exhibits greater resistance to domain shifts. As a result, our ‘Ap, Mo’ outperforms ‘Ap-Mo’. Furthermore, the integration of audio with separate encoders in ‘Ap, Mo, Au’ yields the best overall performance.

Table  4 evaluates the effectiveness of each component of our approach. Starting with the baseline (trained using only cross-entropy), we observe that weighting the audio embeddings $a$ by the consistency ratings $r$ enhances the model’s performance, owing to the more reliable audio representations. Aligning all modalities with narration (‘$B+\mathcal{L}_{vt}+\mathcal{L}_{mt}+\mathcal{L}_{at}$‘) further improves performance. Notably, when we align audio features with their corresponding audio narrations $\hat{t}$ using $\mathcal{L}_{a\hat{t}}$ the model outperforms the version with $\mathcal{L}_{at}$ where audio features are aligned with the original visual-based narrations $t$. Finally, the combination of alignment losses and the consistency-weighted audio approach in our final method ‘Ours‘ achieves the best performance across all domains.

Conclusion. We proposed a novel multimodal framework where motion, audio, and appearance improve domain generalization in first-person action recognition. We demonstrated that audio and motion features exhibit greater resilience to domain shifts than appearance features. By aligning audio features with audio-specific narrations and applying consistency-weighted audio during training, our method enhanced the robustness of action representations.