Adversarial Cross-Domain Action Recognition with Co-Attention

Fig. 2 shows the overall architecture of TCoN. During training, given a source and target video pair, we first partition each source video into $K_{s}$ segments and each target video into $K_{t}$ segments uniformly. We use $v_{ij}^{s}$ to denote the $j$-th segment in the $i$-th source video, and $v_{i^{\prime}j^{\prime}}^{t}$ is similarly defined for the target video. Then, we generate features $f_{ij}^{*}$ ($*\in(s,t)$) for each segment with a feature extractor $G_{f}$, which is a 2D or 3D CNN, i.e., $f_{ij}^{*}=G_{f}(v_{ij}^{*})$. Next, we use the co-attention module to calculate the co-attention matrix between this source and target video pair, from which we further derive the source and target attention score vectors, as well as the target-aligned source feature $\hat{f}^{t}$. Then, the discriminator module performs distribution matching given the source, target, and target-aligned source features. Finally, a shared classifier $G_{y}$ accepts the source and target features together with their attention scores, to predict the labels $\hat{y}_{i}^{*}$. For label prediction, we follow the standard practice, where we first predict per-segment labels and then weighted-sum segment predictions by attention scores.

Co-attention was originally used in NLP to capture the interactions between questions and documents to boost the performance of question answering models (?). Motivated by this, we propose a novel cross-domain co-attention mechanism to capture correlations between videos from two domains. To the best of our knowledge, this is the first time that co-attention is explored under the cross-domain action recognition setting.

The goal of the co-attention module is to model relations between source and target video pairs. As shown in Fig. 3, given a pair of source and target videos $V_{i}^{s}$ and $V_{i^{\prime}}^{t}$, and their segment features $f_{ij}^{s}|_{j=1}^{K_{s}}$ and $f_{i^{\prime}j^{\prime}}^{t}|_{j^{\prime}=1}^{K_{t}}$, we use $k$ to index video pairs, i.e., $pair(k)=(i,i^{\prime})$, which denotes that video pair $k$ is composed of the source video $V_{i}^{s}$ and the target video $V_{i^{\prime}}^{t}$. We first calculate a self-attention score vector $\mathbf{a}_{i}^{ss}$ and $\mathbf{a}_{i^{\prime}}^{tt}$ for each video:

$$a_{ij}^{ss}=\frac{1}{K_{s}-1}\sum_{\bar{j}\neq j}\langle f_{i\bar{j}}^{s},f_{ij}^{s}\rangle,$$

(1)

$$a_{i^{\prime}j^{\prime}}^{tt}=\frac{1}{K_{t}-1}\sum_{\bar{j}^{\prime}\neq j^{\prime}}\langle f_{i^{\prime}\bar{j}^{\prime}}^{t},f_{i^{\prime}j^{\prime}}^{t}\rangle,$$

(2)

where $a_{ij}^{ss}$ and $a_{i^{\prime}j^{\prime}}^{tt}$ are the $j$-th and $j^{\prime}$-th element of $\mathbf{a}_{i}^{ss}$ and $\mathbf{a}_{i^{\prime}}^{tt}$, respectively. $\langle\cdot,\cdot\rangle$ denotes the inner-product operation. Self-attention score vectors measure the intra-domain importance of a segment within a video. After we obtain these self-attention score vectors, we then derive each $(j,j^{\prime})$-th element of the cross-domain similarity matrix $A_{k}^{st}$ as $a_{jj^{\prime}}^{st}=\langle f_{ij}^{s},f_{i^{\prime}j^{\prime}}^{t}\rangle$. Note that for clarity, we drop the pair index $k$ for $a_{jj^{\prime}}^{st}$. Each element $a_{jj^{\prime}}^{st}$ measures the cross-domain similarity between segment pair $v_{ij}^{s}$ and $v_{i^{\prime}j^{\prime}}^{t}$. Finally, we calculate the cross-domain co-attention score matrix ${A}_{k}^{co}$ by

$${A}_{k}^{co}=\left(\mathbf{a}_{i}^{ss}(\mathbf{a}_{i^{\prime}}^{tt})^{T}\right)\odot A_{k}^{st},$$

(3)

where $\odot$ represents element-wise multiplication. As can be seen from this process, in the co-attention matrix ${A}_{k}^{co}$, only the elements corresponding to those segment pairs $v_{ij}^{s}$ and $v_{i^{\prime}j^{\prime}}^{t}$ with high $a_{ij}^{ss}$, $a_{i^{\prime}j^{\prime}}^{tt}$ and $a_{jj^{\prime}}^{st}$ would be assigned high values, which means that only key segment pairs that are also common in both domains will be paid high attention to. Thus, this co-attention matrix effectively reflects the correlations between source and target video pairs. Also, key segments (those with only high $a_{ij}^{ss}$ or $a_{i^{\prime}j^{\prime}}^{tt}$) or common segments (those with only high $a_{jj^{\prime}}^{st}$) are not ignored, but paid less attention to. Only segments that are neither important nor common are discarded as they are essentially noise and do not help with the task.

For each segment, we derive attention scores from the above co-attention matrix by averaging its co-attention scores with all segments in the other video. We generate the ground-truth attention $\mathbf{a}_{i}^{s}$ and $\mathbf{a}_{i^{\prime}}^{t}$ for $V_{i}^{s}$ and $V_{i^{\prime}}^{t}$ as follows,

$$\mathbf{a}_{i}^{s}=\frac{1}{M_{i}^{s}}\sum_{i\in pair(k)}\sum_{row}A_{k}^{co},$$

(4)

$$\mathbf{a}_{i^{\prime}}^{t}=\frac{1}{M_{i^{\prime}}^{t}}\sum_{i^{\prime}\in pair(k)}\sum_{column}A_{k}^{co},$$

(5)

where $\sum_{row}$ and $\sum_{column}$ are summation with respect to rows and columns, respectively. $M_{i}^{s}$ is the number of related pairs to $V_{i}^{s}$ and $M_{i^{\prime}}^{t}$ is defined similarly. All the attention vectors sum to $1$. Since we should not assume access to source videos during testing time, we further use an attention network $G_{a}$, which is a fully-connected network, to predict attention scores for target videos:

$$\hat{a}_{i^{\prime}j^{\prime}}^{t}=G_{a}(f_{i^{\prime}j^{\prime}}^{t}),$$

(6)

where $\hat{a}_{i^{\prime}j^{\prime}}^{t}$ is the $j^{\prime}$-th element of the predicted attention. We further calculate the loss for the attention network with supervision from the ground-truth attention:

$$C_{a}=\frac{1}{N_{t}}\sum_{i^{\prime}=1}^{N_{t}}L_{a}(\hat{\mathbf{a}}_{i^{\prime}}^{t},\mathbf{a}_{i^{\prime}}^{t}),$$

(7)

where $L_{a}$ is the regression loss. With the source and target attention, the final classification loss is thus:

	$\displaystyle C_{y}=$	$\displaystyle\frac{1}{N_{s}}\sum_{i}^{N_{s}}L_{y}(\sum_{j=1}^{K_{s}}a_{ij}^{s}G_{y}(f_{ij}^{s}),y_{i}^{s})$		(8)
		$\displaystyle+\frac{1}{N_{t}}\sum_{i^{\prime}}^{N_{t}}L_{y}(\sum_{j^{\prime}=1}^{K_{t}}\hat{a}_{i^{\prime}j^{\prime}}^{t}G_{y}(f_{i^{\prime}j^{\prime}}^{t}),y_{i^{\prime}}^{t}),$

where $L_{y}$ is the cross-entropy loss for classification. We train the classifier using source videos with ground-truth labels. Similar to previous work (?), we also use target videos that have high label prediction confidence as training data, where the predicted pseudo-labels serve as the supervision. This helps preserve the $\lambda$ term in the error bound derived in Theorem 1 in (?). Note that the total number of source and target video pairs is quadratic to the size of dataset, which is very large. For efficiency and co-attention with higher quality, we only calculate co-attention for video pairs with similar semantic information, i.e., video pairs with similar label prediction probability (which acts as a soft label).

Fig. 4 illustrates the video-level discriminator we employ to match the distributions of target-aligned source and target video features, and the segment-level discriminator we use to match target-aligned source and source segment features.

For each pair $k$ of video $V_{i}^{s}$ and $V_{i^{\prime}}^{t}$, the target-aligned source segment feature ${\hat{f}_{kj^{\prime}}^{t}}|_{j^{\prime}=1}^{K_{t}}$ is calculated as follows,

$$\hat{f}_{kj^{\prime}}^{t}=\sum_{j=1}^{K_{s}}(A_{k}^{co})_{jj^{\prime}}f_{ij}^{s},$$

(9)

where $(A_{k}^{co})_{jj^{\prime}}$ is the $(j,j^{\prime})$-th element of $A_{k}^{co}$. Note that after we get $A_{k}^{co}$, we further normalize each column of it with a softmax function to keep the norm of target-aligned source features same as source features. Each target-aligned source segment feature is thus derived by weighted-summing source segment features, where the weight is the co-attention score between the target segment feature and each source segment feature. Thus, each target-aligned source segment feature preserves the semantic meaning of the corresponding target segment and falls on the source distribution. We concatenate segment features ${\hat{f}_{kj^{\prime}}^{t}}|_{j^{\prime}=1}^{K_{t}}$ as $\hat{F}_{k}^{t}$ and ${f_{i^{\prime}j^{\prime}}^{t}}|_{j^{\prime}=1}^{K_{t}}$ as $F_{i^{\prime}}^{t}$ in temporal order, which then naturally form as action features. Furthermore, $F_{i^{\prime}}^{t}$ and all corresponding $\hat{F}_{k}^{t}$ are strictly temporally aligned since segments at the same time step express the same semantic meaning.

Then, we can derive the domain adversarial loss to match video distributions of target-aligned source features and target features. To further ensure the target-aligned source features to fall in the source feature space, we also match the segment distributions of target-aligned source features and source features. The loss for the video-level $C_{dv}$ and segment-level $C_{ds}$ discriminators are defined as follows,

$$C_{dv}=\frac{1}{N_{t}}\sum_{i^{\prime}}L_{d}(G_{d}^{v}(F_{i^{\prime}}^{t}),d_{v})+\frac{1}{N_{st}}\sum_{k}L_{d}(G_{d}^{v}(\hat{F}_{k}^{t}),d_{v}),$$

(10)

	$\displaystyle C_{ds}$	$\displaystyle=\frac{1}{N_{s}K_{s}}\sum_{i,j}L_{d}(G_{d}^{seg}(f_{ij}^{s}),d_{seg})$		(11)
		$\displaystyle+\frac{1}{N_{st}K_{t}}\sum_{k,j^{\prime}}L_{d}(G_{d}^{seg}(\hat{f}_{kj^{\prime}}^{t}),d_{seg}),$

where $N_{st}$ is the number of all pairs and $L_{d}$ is the binary cross-entropy loss. The video domain label $d_{v}$ is $0$ for target-aligned source features and $1$ for target features. The segment domain label $d_{seg}$ is $0$ for target-aligned source segment features and $1$ for target segment features.

We perform optimization in the adversarial learning manner (?). We use $\theta_{f}$, $\theta_{a}$, $\theta_{y}$, $\theta_{d}^{v}$ and $\theta_{d}^{seg}$ to denote the parameters for $G_{f}$, $G_{a}$, $G_{y}$, $G_{d}^{v}$ and $G_{d}^{seg}$ and optimize:

$$\min_{\theta_{f},\theta_{a},\theta_{y}}C_{y}+\lambda_{a}C_{a}-\lambda_{d}(C_{dv}-C_{ds}),$$

(12)

$$\min_{\theta_{d}^{v},\theta_{d}^{seg}}C_{dv}+C_{ds},$$

(13)

where $\lambda_{a}$ and $\lambda_{d}$ are trade-off hyper-parameters.

With the proposed Temporal Co-attention Network that contains an attentive classifier as well as a video- and a segment-level adversarial networks, we can simultaneously align the source and target video distributions while also minimize the classification error within both source and target domains, thus address the cross-domain action recognition problem in an effective way.

For TSN, C3D and TRN, we train on source and test on target directly. For shallow learning methods, we use deep features from the source pre-trained model as input. For the hybrid model, we apply domain discriminator in CDAN to segment features and the consensus domain discriminator output of all segments as final output. For DAAA and $\text{TA}^{3}\text{N}$, we use their original training strategy. For TCoN, since the target attention network is not well-trained at the beginning, we use uniform attention at the first few iterations and plug it in when the loss of the attention network is lower than a certain threshold. To train TCoN more efficiently, we only calculate co-attention for segment pairs within mini-batchs.

We implement TCoN with the PyTorch framework (?). We use the Adam optimizer (?) and set the batch size to 64. For TSN and TRN-based models, we adopt the BN-Inception (?) backbone pre-trained on ImageNet (?). The learning rate is initialized to 0.0003 and decreases by $\frac{1}{10}$ every 30 epochs. We adopt the same data augmentation technique as in (?). For C3D-based models, we strictly follow the settings in (?) and use the same base model (?) pre-trained on Sports-1M dataset (?). We initialize the learning rate for the feature extractor to 0.001 while 0.01 for classifier since it is trained from scratch. The trade-off parameter $\lambda_{d}$ is increased gradually from $0$ to $1$ as in DANN (?). For the number of segments, We do grid search for each dataset in [1, minimum video length] on a validation set. Please refer to supplementary material for the actual numbers.

The classification accuracies on three datasets using TSN-based TCoN are shown in Table 1. We can observe that the proposed TCoN outperforms all baselines on all datasets. In particular, TCoN improves previous methods with the largest margin on Jester, where temporal information is much more important, and temporal misalignment is more severe. Both CDAN and DAAA use segment features and minimize the Jensen-Shannon divergence of segment feature distributions between domains. The higher accuracy of TCoN demonstrates the importance of temporal alignment in distribution matching. We also notice that the Flow model consistently outperforms the RGB model for TCoN, indicating that TCoN well utilizes temporal information.

We also compare with DAAA (?) under their experiment setting with the C3D backbone. From Table 2, we can observe that TCoN outperforms DAAA on both tasks, which further suggests the efficacy of the proposed co-attention and distribution matching mechanism.

Moreover, we compare with the state-of-the-art cross-domain action recognition method $\text{TA}^{3}\text{N}$ (?) on two datasets they used, namely UCF50-Olympic_Sports and $\text{(UCF101-HMDB51)}_{\text{2}}$ (which is slightly different from $\text{(UCF101-HMDB51)}_{\text{1}}$ in 3 out of the 12 shared classes), as well as Jester. And we use the same backbone TRN (?) and input modality (RGB) as theirs. The results from Table 3 show that on the datasets they used, TCoN outperforms $\text{TA}^{3}\text{N}$ on 3 out of 4 tasks and on par with it on the other. Moreover, TCoN achieves better performance on Jester, which again corroborates that TCoN can well handle not only the appearance gap but also the action gap.

To test whether our model is robust when the two domains do not share the same action space during training, we conduct experiments on $\text{(HMDB51}\rightarrow\text{UCF101)}_{\text{all}}$, where we train TCoN on data from all classes in HMDB51, but only test on the shared classes between two domains (it is impossible to predict those non-overlapping target classes). The results from Table 4 show that TCoN still outperforms its baselines, suggesting the robustness of it in this case.