A Novel Self-Knowledge Distillation Approach with
Siamese Representation Learning for
Action Recognition

Given a set of $N$ training samples is denote as $\{(x^{(i)},y^{(i)})\}_{i=1}^{N}$ as a labelled source dataset from $K$ classes where $x^{(i)}$ is a video, and $y^{(i)}$ is a $K$-dimensional one-hot vector as its hard label (i.e., the action). For a mini-batch input $B=\{(x^{(i)},y^{(i)})\}_{i=1}^{n}$, we apply data augmentation operators (e.g. flip, contrast adjustment, etc.) for each $x^{(i)}$ in $B$. Let $x_{1}^{(i)},x_{2}^{(i)}=\mathcal{T}(x^{(i)})$ denote the two randomly augmented views from the original video $x^{(i)}$ where $\mathcal{T}(\cdot)$ is the set of data augmentation operators. Note that $x_{1}^{(i)},x_{2}^{(i)}$ have the same the label $y^{(i)}$. The two views $x_{1}^{(i)},x_{2}^{(i)}$ are passed into an encoder network $f_{\theta}$, where $\theta$ denotes the set of parameters of $f$. Let $\mathbf{r}_{1}^{(i)}$ and $\mathbf{r}_{2}^{(i)}$ correspond to the output vectors of the network $f_{\theta}$ with the input being $x_{1}^{(i)},x_{2}^{(i)}$ (i.e., $\mathbf{r}_{1}^{(i)}=f_{\theta}(x_{1}^{(i)})$ and $\mathbf{r}_{2}^{(i)}=f_{\theta}(x_{2}^{(i)})$). The encoder $f_{\theta}$ shares weights between the two views. A $FC$ layer is utilized to generate the logit prediction from $\mathbf{r}_{1}^{(i)}$ and $\mathbf{r}_{2}^{(i)}$ as follows:

where $\mathbf{p}_{1}^{(i)}$ and $\mathbf{p}_{2}^{(i)}$ denote the $K$-dimensional logit prediction vector and each dimension represents the logit value for the $k^{th}$ class (with $k=1,2,...,K$). Let $\mathbf{p}^{(i)}=\mathbf{p}_{1}^{(i)}\oplus\mathbf{p}_{2}^{(i)}$ where $\oplus$ corresponds to the add operator. Similar to other self-knowledge distillation methods, SKD-SRL also performs the knowledge distillation through the soft label as follows:

where $D_{KL}$ denotes the Kullback-Leibler (KL) divergence function, $\tau$ is the temperature scaling parameter. $\mathbf{r}_{1}^{(i)}$ and $\mathbf{r}_{2}^{(i)}$ also are passed into a projector MLP network $g_{\xi}$. We have $\mathbf{z}_{1}^{(i)}=g_{\xi}(\mathbf{r}_{1}^{(i)})$ and $\mathbf{z}_{2}^{(i)}=g_{\xi}(\mathbf{r}_{2}^{(i)})$. A predictor MLP $q_{\mu}$ is utilized to transform one vector $\mathbf{z}_{1}^{(i)}$ ($\mathbf{z}_{2}^{(i)}$) and match it to the other vector $\mathbf{z}_{2}^{(i)}$ ($\mathbf{z}_{1}^{(i)}$) by minimizing their negative cosine similarity, $\xi$ and $\mu$ denote the set of parameters of the network $g$ and $q$, respectively. Denoting the two output vectors as $\mathbf{v}_{1}^{(i)}=g_{\mu}(\mathbf{z}_{1}^{(i)})$ and $\mathbf{v}_{2}^{(i)}=g_{\mu}(\mathbf{z}_{2}^{(i)})$. We utilize the similarity loss function as follows:

where $stopgrad$ is the stop-gradient operator [21]. This means that $\mathbf{z}_{1}^{(i)}$ and $\mathbf{z}_{2}^{(i)}$ are treated as a constant in this term. $D_{sim}$ is the negative cosine similarity function as follows:

where $\left\lVert\cdot\right\rVert_{2}$ is $\ell_{2}$-norm. By integrating Eq. 2 and Eq. 3, we can construct the final optimization objective for the entire network as follows:

where $\mathcal{L}_{CE}$ is the standard cross-entropy loss, $\alpha$ and $\beta$ are the loss weights for the distillation loss $\mathcal{L}_{KL}$ and the similarity loss $\mathcal{L}_{sim}$, respectively. The pseudo-code of SKD-SRL is illustrated in Algorithm. 1.

Network Architecture. For the encoder network $f_{\theta}$, we consider two state-of-the-art CNN architectures including the 3D ResNet-18, and the 3D ResNet-50 in [9]. For the projector network $g_{\xi}$ which maps the representation vectors $\mathbf{r}_{1}$ and $\mathbf{r}_{2}$ to vectors $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$, we instantiate $g_{\xi}$ just a single linear layer of size = 2048. The predictor network $q_{\mu}$ has two MLP layers where the first hidden layer is applied BN and ReLU, and the output layer does not have BN and activation function. Due to the predictor’s input and output dimension as 2048, we set the predictor’s hidden layer’s dimension as 512 following the instruction in [21]. We leave to future work the investigation of optimal $g_{\xi}$ and $q_{\mu}$ architectures.

Data Augmentation. Given an input video, our data augmentation method takes two steps. We first trim two clips with $T$ continuous frames from the raw video. Each frame in the clip is scaled the shorter edge of the frames in the clip to 128, and the other edge is calculated to maintain the frame original aspect ratio. A random cropping window with dimensions of $112\times 112$ is generated and applied to all frames. We then randomly choose data augmentation operators to apply for each clip to generate the two views from the original clip. The list of all augmentation operators are used in our work including flip, contrast adjustment, brightness adjustment, hue adjustment, Gaussian blur and channel splitting [30]. The probability chosen for each augmentation operator is set to 0.5.

UCF101: includes 13,320 action instances from 101 human action classes. The average duration of each video is about 7 seconds.

HMDB51: is a small dataset including 6,766 videos from 51 human action classes. The average duration of each video is about 3 seconds.

Kinetics400: is a large dataset that has 400 human action classes [33]. The videos were temporally trimmed and last around 10 seconds and 200–1000 clips for each action. The total has 306,245 videos in Kinetics400.

Implementation Details. All networks are trained from scratch and optimized by stochastic gradient descent (SGD) with a momentum of 0.9 and an initial learning rate of 0.01. The weight decay is set to $5\times 10^{-4}$. The input of the encoder network is a video clip with 16 frames, each frame has $112\times 112\times 3$ of dimension and normalized to be [-1, 1]. We use the mini-batch of 32 clips per GPU, and training is done in 200 epochs. The learning rate is dropped by 10$\times$ after 10 epochs if the validation accuracy not improving. For our method, the temperature $\tau$ is set as 10, and the loss weights $\alpha$ and $\beta$ are set as 0.1 and 1, respectively.

To examine the effectiveness of the proposed SKD-SRL method, we have compared our approach to the baseline (independently training) with cross-entropy loss on the standard datasets with both the ResNet-18 and ResNet-50 networks.

As shown in Table. I, the SKD-SRL method outperforms the baseline method for both the large and small-scale datasets regardless of the backbone networks. Specifically, the SKD-SRL method increases the accuracies by 23.3% and 12.7% for the 3D ResNet-18 and 3D ResNet-50 networks, respectively, on small-scale datasets such as the UCF101 dataset. In the large-scale dataset, i.e., the Kinetics400 dataset, the SKD-SRL method increases the accuracies by 12.5% and 14.3% for the 3D ResNet-18 and 3D ResNet-50 networks, respectively. From these results, we found that our approach can significantly improve generalization and performance compared to independently training only with hard labels.

Table II compares our SKD-SRL method with other distillation mechanisms. As expected, the student performance in distillation approaches improves compared to independent training (i.e., baseline). Moreover, the Self-KD method shows better top-1 accuracy than the conventional KD (+5.3% accuracy). This shows that although there is no teacher network, Self-KD still significantly improves performance via its self-teaching and self-learning mechanism. Meanwhile, the proposed SKD-SRL method outperforms 3.1% compare to Self-KD. It demonstrates that our approach enhances the generalization capability of the single network by combining Self-KD with Siamese representation learning.

In this part, we evaluate SKD-SRL on the Kinetics400 dataset with two backbone networks, including 3D ResNet18 and 3D ResNet-50. As shown in Table. III, the proposed SKD-SRL method has obtained state-of-the-art performance while utilizing fewer frames (16 vs 32, 64) and lower frame resolutions (112 vs 224). In particular, the SKD-SRL method uses RGB frames without the optical flow, which significantly reduces the cost of optical flow calculation and model training on the optical flow domain. Moreover, our proposed SKD-SRL has better performance with a shallower model than other state-of-the-art methods (3D ResNet-50 vs 3D ResNet-101, DenseNet-169).