Multi-modal Instance Refinement for Cross-domain Action Recognition

In action recognition, early works use 2D/3D convolution [9, 8] for feature extraction only in a single modality, i.e., RGB. Later, optical flow of video segments is used as auxiliary training data which carries more temporal and motion information compared with RGB [21]. Therefore, current popular CNN-based methods adopt a two-stream 3D convolutional neural network structure [19, 1] for feature extraction which could utilize the information contained in multiple modalities and model the temporal information. Most recently, vision transformer [6] based approaches have excelled CNN-based methods on many benchmarks. MVD [22] builds a two-stage masked feature modeling framework, where the high-level features of pretrained models learned in the first stage will serve as masked prediction targets for student model in the second stage. Although these methods show promising performance in a supervised manner, we are going to focus on action recognition under the setting of UDA.

Though both RGB and optical flow have been studied for domain adaptation in action recognition, there are only a limited number of works attempted to conduct multi-modal domain adaptation [26, 10, 17]. Munro and Dame [17] propose MM-SADA, a multi-modal 3D convolutional neural network with a self-supervision classifier between modalities. It uses Gradient Reversal Layer (GRL) [7] to implement domain discriminator within different modalities. Kim et al. [10] apply contrastive learning to design a unified framework using transformer for multi-modal UDA in video understanding. Xu et al. [26] propose a source-free UDA model to learn temporal consistency in videos between source domain and target domain. Similar to [17], our work adopts a multi-modal 3D ConvNet for feature extraction and utilizes domain adversarial learning, but we focus on a different task by incorporating deep reinforcement learning into our action recognition framework to eliminate the effect of negative transfer.

Deep reinforcement learning has been applied to various tasks in computer vision [23, 27]. Wang et al. [23] design a reinforcement learning-based two-level framework for video captioning, in which a low-level module recognizes the original actions to fulfill the goals specified by the high-level module. ReinforceNet [27] incorporates region selection and bounding box refinement networks to form a reinforcement learning framework based on CNN to select optimal proposals and refine bounding box positions. Recently, several works apply reinforcement learning to action recognition [5, 25]. Dong et al. [5] design a deep reinforcement learning framework to capture the most discriminative frames and delete confusing frames in action segments. Weng et al. [25] improve recognition accuracy by designing agents that learn to produce binary masks to select out interfering categories. All these methods adopt deep reinforcement learning to refine negative frames in the action segment within the same domain, while our method uses deep reinforcement learning to refine negative action segments across domains to handle negative transfer in cross-domain action recognition.

For multiple modalities, a feature fusion layer is applied after the action classifiers for summing the prediction scores from different modalities as shown in Fig. 2(a). For input $\mathcal{X}$ with multiple modalities, we have $\mathcal{X}=(\mathcal{X}^{1},\mathcal{X}^{2},\ldots,\mathcal{X}^{K})$, where $\mathcal{X}^{k}$ represents the input from the $k$-th modality. Therefore, we can define the classification loss as follows:

where $y$ represents the class label, $C^{k}$ is the action classifier of the $k$-th modality, $F^{k}$ denotes the feature extractor of the $k$-th modality and $x^{k}$ represents source action segments from the $k$-th modality which are labeled.

We visualize the overall workflow of instance refinement module in Fig. 3. In each modality, we select negative instances from the $\mathcal{i}$-th batch of action segments $\mathcal{F}^{s}_{i}$ and $\mathcal{F}^{t}_{i}$ in source and target domain, respectively. We divide a batch into several sub batches, namely, candidate set $\mathcal{F}_{C}$ for iterating the agents over more episodes. Therefore, we can have a total number of $\mathcal{E}$ candidate sets in a batch. Each episode is responsible for selecting out $E$ negative samples, thus the terminal time of an episode is defined as $E$. Take time $e$ in the selection process as an example, S-agent takes an action $\mathcal{A}^{s}_{e}$ by observing current state $\mathcal{S}^{s}_{e}$. Then, the current state is updated as $\mathcal{S}^{s}_{e+1}$ since an action segment has been selected out. In the meantime, S-agent would receive a reward $\mathcal{R}^{s}_{e}$ for taking action $\mathcal{A}^{s}_{e}$. After arriving at terminal time $E$, S-agent has done selection for this episode and the candidate set $\mathcal{F}_{C}$ is optimized as $\hat{\mathcal{F}_{C}}$. Then, the batch $\mathcal{F}^{s}_{i}$ would become $\hat{\mathcal{F}^{s}_{i}}$ at the end of the last episode, and similarly, we can reach a $\hat{\mathcal{F}^{t}_{i}}$ for $\mathcal{F}^{t}_{i}$ . We give detailed illustrations on State, Action, Reward and DQN Loss in the following part of this section.

• •

  State. Agents make selections on the level of candidate set and take feature vectors of all the action segments inside the candidate set as state. In this case, the state $\mathcal{S}^{s}_{k}$ of S-agent in the $\mathcal{k}$-th candidate set $\mathcal{C}^{s}_{k}$ could be defined as $\mathcal{S}^{s}_{k}=[\mathcal{f}^{s}_{k,1},\mathcal{f}^{s}_{k,2},\mathcal{f}^{s}_{k,3},\ldots,\mathcal{f}^{s}_{k,\mathcal{N}_{c}}]\in\mathbb{R}^{d\times\mathcal{N}_{c}}$ where $\mathcal{f}^{s}_{k,n}$ is the feature vector of $\mathcal{F}^{s}_{i,k.n}$ that has $d$ dimensions and $\mathcal{N}_{c}$ is the number of action segments inside a candidate set. Once an action segment $\mathcal{f}^{s}_{k,n}$ is selected out from $\mathcal{S}^{s}_{k}$ , it will be replaced by a $d$-dimensional zero vector to keep the state shape unchanged. This is the same for T-agent where $\mathcal{S}^{t}_{k}=[\mathcal{f}^{t}_{k,1},\mathcal{f}^{t}_{k,2},\mathcal{f}^{t}_{k,3},\ldots,\mathcal{f}^{t}_{k,\mathcal{N}_{c}}]\in\mathbb{R}^{d\times\mathcal{N}_{c}}$ is the state and $\mathcal{f}^{t}_{k,n}$ is the feature vector of target action segment.

• •

  Action. For a candidate set of size $\mathcal{N}_{c}$, we can have $\mathcal{N}_{c}$ actions to perform in each episode. Therefore, we can define the set of actions that can be taken by S-agent as $\mathcal{A}^{s}=\{1,2,\ldots,\mathcal{N}_{c}\}$ and T-agent as $\mathcal{A}^{t}=\{1,2,\ldots,\mathcal{N}_{c}\}$, which represents the index of the action segment that is to be selected out. The aim of the DQN agents is to maximize the accumulated reward of the actions taken. We define the accumulated reward at time $e$ as $\mathcal{R}_{e}=\sum_{t=e}^{E}\gamma^{t-e}r_{e}$, where $\gamma$ is the discount factor and $r_{e}$ represents the instant reward at time $e$. In DQN, we define a state-action value function to approximate the accumulated reward as $Q(\mathcal{S}_{e},\mathcal{a}_{e})$, where $\mathcal{S}_{e}$ denotes the state and $\mathcal{a}_{e}$ denotes the action taken at time $e$. For both modalities, $\mathcal{S}_{e}\in\{\mathcal{S}_{e}^{s},\mathcal{S}_{e}^{t}\}$ and $\mathcal{a}_{e}\in\{\mathcal{A}^{s},\mathcal{A}^{t}\}$. As shown in Fig. 3, DQN outputs a set of q-values corresponding to each action and chooses the optimal action which has the maximum q-value to maximize accumulated reward. This policy can be defined as follows:

  $$\hat{\mathcal{a}_{e}}=\max_{\mathcal{a}_{e}}Q(\mathcal{S}_{e},\mathcal{a}_{e}).$$

  (2)

• •

  Reward. Rewards given to agents are based on actions taken and the relevance of selected action segments to the opposite domain. To measure the relevance, we use the prediction results from domain classifier $D$. The domain logits of an action segment are processed by a sigmoid function and the relevance measure $\Delta(\mathcal{f})$ is defined as:

  $$\Delta(\mathcal{f})=\begin{cases}Sigmoid(D(\mathcal{f})),&\mathcal{f}\in\hat{\mathcal{F^{s}}}\\ 1-Sigmoid(D(\mathcal{f})),&\mathcal{f}\in\hat{\mathcal{F^{t}}}.\end{cases}$$

  (3)

  In Eq.3, we unify the relevance measure in both source and target domains by defining the domain label of source to be 0 and target to be 1. Then, the predefined threshold $\tau$ and the relevance measure $\Delta(\mathcal{f})$ can be compared to give rewards to agents according to the criterion defined below:

  $$r=\begin{cases}1,&\Delta(\mathcal{f})<\tau,\\ -1,&otherwise.\end{cases}$$

  (4)

  This criterion is quite intuitive for an agent to recognize if it has made the right selection. Besides, we can set different thresholds for agents in different domains as $\tau^{s}$ and $\tau^{t}$ in source and target, respectively.

• •

  DQN Loss. For a DQN, the target output is defined as:

  $$y_{e}=r_{e}+\gamma\cdot\max_{\mathcal{a}_{e+1}}Q(\mathcal{S}_{e+1},\mathcal{a}_{e+1}|\mathcal{S}_{e},\mathcal{a}_{e})$$

  (5)

  where $y_{e}$ represents the temporal difference target value of the Q function $Q(\mathcal{S}_{e},\mathcal{a}_{e})$. Based on this, the loss of DQN can be defined as:

  $$\mathcal{L_{q}}=\mathbb{E}_{\mathcal{S}_{e},\mathcal{a}_{e}}[(y_{e}-Q(\mathcal{S}_{e},\mathcal{a}_{e}))^{2}].$$

  (6)

  Then, we can have the overall deep Q-learning loss defined as follows:

  $$\mathcal{L_{dqn}}=\sum^{K}_{k=1}(\mathcal{L_{q}^{s}}+\mathcal{L_{q}^{t}})_{k}$$

  (7)

  which is the sum of losses from S-agents and T-agents from all modalities.

We realize feature alignment across domains in an adversarial way by connecting the domain discriminator with a GRL as shown in Fig. 2(b). We apply a domain discriminator in each modality rather than using a single discriminator for all modalities after late fusion since aligning domains in a combined way might lead the network to focus on a less robust modality and lose the ability to generalize to other modalities [17]. Then, we can define our domain adversarial loss as:

where $d$ is the domain label, $D^{k}$ is the domain discriminator for the $k$-th modality, $F^{k}$ is the feature extractor of the $k$-th modality, and $x^{k}$ denotes the action segments from source domain or target domain of the $k$-th modality.

With the losses defined in previous sections, we can have an overall loss function:

For DQN agents, we use experience replay [13] and $\epsilon$-greedy strategy [16] during training. An experience replay pool to store actions, states, rewards, etc. is established for every agent, which ensures that data given to them is uncorrelated. The $\epsilon$-greedy strategy introduces a probability threshold of random action $\epsilon$ to control whether an action is predicted by DQN agent or just randomly selected. This helps to balance the exploitation and exploration of an agent. The strategy is implemented as follows:

where $\lambda$ is a random variable. If $\lambda$ is larger than $\epsilon$, the action would be predicted by the agents, or the action would be randomly chosen from the pool of actions.

For all the experimental results, we follow [17] to report the average top-1 accuracy on target domain over the last 9 epochs during training. In the meantime, the experimental results of our model trained with only source data are reported as a lower limit. Also, we report results of supervised learning on target domain as the upper bound.

We have a total of 6 domain adaptation settings based on D1, D2 and D3. We make a comparison between our method, several baseline methods and two state-of-the-art methods under every domain setting in Table 1. On average, our method outperforms all other state-of-the-art methods. MMIR has an overall performance improvement of $5.8\%$ compared with Source-only, $4.5\%$ compared with MMD [14], $4.1\%$ compared with AdaBN, $4.0\%$ compared with MCD [18], $1.0\%$ compared with MM-SADA [17] and $0.3\%$ compared with Kim et al. [10].