EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2022: Team HNU-FPV Technical Report

Given a RGB stream of video frames $\left\{\bm{v}^{s}_{1},\bm{v}^{s}_{2},...\right\}$ and a optical flow stream of video frames $\left\{\bm{v}^{t}_{1},\bm{v}^{t}_{2},...\right\}$, the model will extract the spatio-temporal features of the two different video streams. For the local feature extraction, the model takes a glance at each frame in the video with the corresponding glancer network ${f}_{\text{G}}$. Then the cheap and coarse feature will be fed into the corresponding policy network $\pi$ to select the area that contributes the most to the task:

where $\tilde{\bm{v}}^{s}_{n}$, $\tilde{\bm{v}}^{t}_{n}$ are the selected patches of RGB video frames and optical flow video frames of the $n^{th}$ frame. And the selected patches $\tilde{\bm{v}}^{s}_{n}$, $\tilde{\bm{v}}^{t}_{n}$ will be fed into the corresponding focuser network ${f}_{\text{F}}$ to extract the local feature maps $\bm{e}^{s}_{\text{L}}$, $\bm{e}^{t}_{\text{L}}$:

Finally, the global spatio-temporal features $\bm{e}^{s}_{\text{G}}$, $\bm{e}^{t}_{\text{G}}$ and audio feature $\bm{e}^{a}_{\text{G}}$ extracted from the global branch. Note that our model considers the global features corresponding to the audio modalities, which are not represented in the figure for the sake of simplicity. Then the global features are concatenated with the local spatio-temporal features $\bm{e}^{s}_{\text{L}}$, $\bm{e}^{t}_{\text{L}}$ extracted from the local branch are concatenated to serve as the input final feature $\bm{e}$ to the video domain adaptation:

After the global and local spatio-temporal features are obtained, it will be more beneficial for the model to perform efficient domain alignment. In the video domain-adapted training of global-local features from the source and target domains, we adapt an existing video domain adaptation method for action recognition tasks, \ie, TA3N [1]. As shown in Figure 2, model first aligns frame-level features from the source and target domain inputs through the adversarial discriminators $\hat{G}_{sd}$ and generates the corresponding domain loss ${L}_{sd}$. At the same time, the frame-level features of the input are modeled in the temporal relation module of TA3N, and these relation features are aggregated to obtain the video-level features. In aggregating these relational features, the domain attention mechanism is added to pay more attention to the alignment of local temporal features that have larger domain discrepancy. In the domain attention mechanism, the adversarial discriminators $\hat{G}^{n}_{rd}$ are used to align the relational features from the source and target domains, and the corresponding domain loss ${L}^{n}_{rd}$ is generated. Then, the adversarial discriminators $\hat{G}_{td}$ are also used to align the video-level features from the source and target domains, and the corresponding domain loss ${L}_{td}$ is generated. Finally, the model classifies the video-level features through two corresponding classifiers ${f}^{v}_{\text{C}}$ and ${f}^{n}_{\text{C}}$, and generates the predicted verb classification and noun classification.

Spatio-temporal feature extraction. Since the network of spatial feature extraction and temporal feature extraction are the same in parameter settings, the following description will not distinguish between spatial and temporal feature extraction. For the global feature of spatio-temporal feature extraction, we use RGB, flow and audio features provided by the organizers that were extracted with Temporal Binding Network (TBN)  [4] pretrained in the source domain. And we follow the model setting in  [7] to extract the local feature of the spatio-temporal feature. We also adopt MobileNet-V2 (MN2) [5] and ResNet-50 (RN) [3] as the glancer network ${f}_{\text{G}}$ and focuser network ${f}_{\text{F}}$, respectively. And the same policy network is used to select the image patch that contributes most to the task from the input video frames by the differentiable bilinear interpolation. The network parameters are learned with SGD optimizer with momentum 0.9 and weight decay $5\times{10}^{-4}$. For each network of the local branches, the initial learning rates of ${f}_{\text{G}}$, ${f}_{\text{F}}$ and $\pi$ are set to $0.005$, $0.01$, and $1e-4$, respectively. For the video frames that are input into the model, we adopt the same processing method as  [7] and set the size of the selected image patch to $176\times 176$.

Video domain adaptation. After obtaining the spatio-temporal features of the source and target domains, TA3N [1] is used to align the input features and generate the prediction results of the model. The network parameters are also learned with SGD optimizer with momentum 0.9 and weight decay $5\times{10}^{-4}$. During training, the parameters in the spatio-temporal feature extraction are freezed. The initial learning rate is set at 3e-3 and decayed by a factor of 0.1 at epochs 10 and 20.

Table 1 shows the action recognition effect of the model on the target validation set under different input and hyper-parameter settings. The table shows that the accuracy can be improved under the same hyperparameter setting by using the local spatio-temporal branch to extract the local feature. We tried two groups of models trained under different hyper-parameters, and their performance on the target test set is shown in Table 2. Our proposed method performs favorably against TA3N by $0.93\%$ in the top-1 action accuracy. In our final submission, we use RGB, Flow and Audio modalities, and the shared feature dimension of the model is set as 2048. The number of input frames of the glancer network and focuser network is set as 8 and 12, respectively. And the visualization results of the image patches selected from the test set by the proposed method are shown in Figure 3. Each line shows a number of image patches selected from consecutive video frames by the spatial local module of the method. It should be noticed that the spatial local module is fixed after training with source domain data. It can be seen that the model can also be well applied to the videos of the target domain.