VideoMix: Rethinking Data Augmentation for Video Classification

There are many augmentation strategies for static image classification tasks. Horizontal flipping, random resizing, and cropping have been used in training ImageNet classifiers and are now considered the standard set of augmentation strategies [32]. There have been regional dropout methods [6, 41] which remove random regions of an image to enhance robustness and generalization. Other single-image augmentation strategies include RandAugment [5] and AutoAugment [4]. They consider the combination of extensive pixel-level image augmentation types, such as rotation, shear, translation, and color jittering. RandAugment and AutoAugment train classifiers with above operations via random selection and learned policy, respectively. Augmentation strategies that combine more than one image include Mixup [40] and CutMix [39]. Mixup averages the RGB values of two images and the ground truth labels to create new samples. CutMix [39] has improved upon regional dropout by filling in image patches from other images in the dropped-out region, thereby maximizing the pixel efficiency during training. The labels are mixed among the source images as in Mixup. We discuss and experiment with the above static-image augmentation strategies on video classification tasks in our analysis.

There do exist a few attempts to apply data augmentation strategies on videos. On the spatial side, the standard single-image augmentation strategies for image classification have been considered: horizontal flipping and random cropping [35]. Along the temporal axis, a widely-used augmentation strategy is to randomly sub-sample a shorter video clip from the full sequence. However, there has been an overall lack of extensive studies on video augmentation methods. This work contributes the first studies on the impact of video augmentation strategies on the generalization, localization, and transfer learning capabilities.

Under certain application scenarios, it is difficult to directly manipulate the input videos. For example, features may have already been extracted from the original frames and the raw frames are unavailable because of the storage limits or legal issues [1]. As we will cover in the experiments of temporal action localization, Temporal VideoMix is a good alternative that improves the localization ability of a video classifier.

To see the effect of VideoMix on preventing overfitting and stabilizing the training process, we compare validation loss and validation accuracy of SlowOnly-34 models during the training over Mini-Kinetics action recognition dataset in Figure 2. We confirm that VideoMix enables video models to obtain lower validation loss and higher validation accuracy than the baseline. After about 200 training epochs, the baseline performance saturates and the loss does not decrease further. Applying VideoMix lets the model overcome the barrier and improve further beyond this point. The training samples generated by VideoMix allow the video classifiers to generalize better.

We expect VideoMix to let an action classifier simultaneously recognize multiple actions present in the mixed videos. To verify that this is achieved, we visualize the spatio-temporal attention of a video on the synthetic video generated by VideoMix with the class activation maps (CAM) [42]. We use a Kinetics-400 pre-trained SlowOnly-50 model. We extend the original CAM proposed for static images to its spatio-temporal version that generates the $T$ sequential score maps. Detail description of spatio-temporal CAM and more examples are in Appendix A. Figure 3 shows VideoMix samples and corresponding class activation maps with respect to the two action classes, “playing harmonica” and “passing American football”. The CAM results show that VideoMix guides a model to see multiple actions at once (e.g., the CAM for “playing harmonica” highlights the player’s mouth and hands, and the CAM for “passing American football” emphasizes the kid’s hands and the football object). Furthermore, VideoMix reduces scene contexts of videos, as the background scene of “passing American football” are partially removed, and also hides some object cues, as the football of “passing American football” and the hands holding a harmonica of “playing harmonica” are blocked in some frames, which leads to learning more robust and generalized cues beyond the object and scene for action recognition.

Kinetics-400 [17] is a widely used large-scale action recognition benchmark consisting of 240k training videos and 20k validation videos in 400 human action classes. The performances are evaluated with the top-1 and top-5 accuracies. Note that about 10% of the Kinetics-400 videos are not available on YouTube to be downloaded. It is not possible to reproduce the exact accuracies reported in the original paper; we train the action classifier over the available subset and treat this result as the baseline.

We follow the original training recipes of the baseline architectures in [7]. We train models from scratch using the stochastic gradient descent optimizer for 250 epochs with batch size $64$ and initial learning rate $0.1$, which is decayed by cosine annealing. For a training video clip, $64$ consecutive frames are randomly sampled from a video and $T$ frames are sub-sampled with $\tau$ temporal stride as an input for video models. For every model, random resize crop and random horizontal flip are applied on training video clips as standard augmentations. For evaluation, we use $3\times 10$ view ensembles as in [7], where $10$ clips are uniformly sampled along the temporal dimension from the entire video sequences and $3$ spatial regions are uniformly sampled along the longer side of the frames.

We use the SlowOnly, SlowFast [7], and interaction preserved CSN (ip-CSN) [34] to show the impact of VideoMix on the video classification task. Every model is based on the ResNet architecture [13]. We denote the specific ResNet type with the suffix “-(#depth)”. We also denote each video model with frame length $T$ and temporal stride $\tau$ in the trailing bracket $(T\times\tau)$. For example, SlowOnly-50 ($4\times 16$) is based on the ResNet-50 architecture, considers $T=4$ input frames sub-sampled from the original $64$ frames with temporal stride $\tau=16$. SlowFast-50 ($8\times 8$) takes two separate input streams, the slow and fast pathways, each with $8$ and $32$ total number of input frames ($T$) with temporal strides ($\tau$) $8$ and $2$, respectively.

We evaluate VideoMix on Kinetics-400 with SlowOnly and SlowFast networks [7] as the base network architectures. SlowFast combines two branches: the slow branch for static spatial features and the fast branch for dynamic motion features. SlowOnly only has the slow branch that is similar to the ResNet [13] architecture with 3D convolutional kernels. The experimental results are shown in Table 4. The accuracies in the table are reproduced results³³3The original paper has reported the top-1 accuracies for SlowOnly-50 ($4\times 16$), SlowOnly-50 ($8\times 8$), and SlowFast-50 ($8\times 8$) as $72.6$, $74.8$, and $77.0$, respectively. The difference is due to the 10% unavailable videos in Kinetics-400 and smaller batch sizes due to GPU limitations.. We also report the inference cost (GFlops) of a single view (a temporal clip with spatial crop) as well as the number of views for the prediction of a single video. We observe that VideoMix consistently improves the accuracy of baseline models. VideoMix achieves the top-1 accuracy of $72.7\%$, $74.9\%$, and $76.6\%$ for SlowOnly-50 ($4\times 16$), SlowOnly-50 ($8\times 8$), and SlowFast-50 ($8\times 8$) with improvements of $+\mathbf{0.9}\%$, $+\mathbf{1.3}\%$, and $+\mathbf{0.7}\%$, respectively. We also show that VideoMix-augmented SlowFast recognizer achieves a competitive performance ($76.6\%$) against other methods such as I3D [2] ($72.1\%$), Two-Stream I3D [2] ($75.7\%$), and Nonlocal-ResNet50 [36] ($76.5\%$) which require more computational costs (GFlops) than the SlowFast architecture.

We also evaluate VideoMix on Mini-Kinetics as shown in Table 5. We observe that VideoMix improves the performance of various baseline architectures: ip-CSN-50 [34], SlowOnly-50 ($4\times 16$), SlowOnly-50 ($8\times 8$), and SlowOnly-50 ($8\times 8$) with $\mathbf{75.9}\%$ ($\mathbf{+1.1}\%$), $\mathbf{76.0}\%$ ($\mathbf{+1.6}\%$), $\mathbf{79.2}\%$ ($\mathbf{+1.7}\%$) and $\mathbf{81.9}\%$ ($\mathbf{+2.4}\%$), respectively.

We conduct ablation studies on the Mini-Kinetics dataset. SlowOnly-34 is used as the running baseline, where the BasicBlock is used as in ResNet-34 [13]. The results are shown in Table 6.

We first examine the impact of the mixture area hyperparameter $\alpha\in\{0.2,0.5,$ $1.0\text{(ours)},2.0\}$. VideoMix at various $\alpha$ values exhibits stable performances, uniformly outperforming the vanilla baseline. VideoMix is not sensitive to the hyperparameter $\alpha$. When we reduce the chance of applying VideoMix on a minibatch to $prob$=$0.5$ (default is $prob$=$1.0$), the top-1 accuracy drops by $0.6$ percent point. The simple strategy of applying VideoMix on every video in the batch is a better choice. When we increase the number of mixing videos (“#videos”) more than two, the performance significantly drops, which indicates that two videos are enough for VideoMix and mixing more than two videos may hinder the training convergence.

Temporal VideoMix and Spatio-temporal VideoMix are not as effective as the VideoMix (Spatial VideoMix), with only $+0.4\%$ and $+1.5\%$ boosts over the baseline. Per-frame VideoMix independently applies VideoMix at every frame, ruining the temporal continuity of the VideoMix operation. It shows an even worse accuracy of 74.8%, lower than the original model. Temporal continuity is an important ingredient for the video augmentation.

More ablation studies to see the effect of dataset sizes and applying other data augmentation methods together with VideoMix are provided in Appendix B.

Something-Something-V2 dataset [9] contains 169k training and 25k validation videos with 174 action classes. We evaluate the performances with top-1 and top-5 accuracies. Something-Something-V2 is known for the fine-grainedness of actions, the diversity of contexts. It poses new challenges for action recognition not covered by Kinetics [17].

We use SlowOnly-50 ($8\times 8$) and SlowFast-50 ($8\times 8$) models. The models are pre-trained on Kinetics-400 with the standard training strategy, and the final fully-connected layer is replaced with the new one with $174$ output dimensions. The entire models are then fine-tuned for the Something-Something-V2 dataset for $40$ epochs with the batch size $64$ and learning rate $0.01$, which is decayed by a factor of 10 after $26$ and $33$ epochs. Other implementation details are in Appendix C.

We investigate how well VideoMix improves the generalizability of action recognition models in the challenging benchmark beyond the Kinetics. To separate the fine-tuning effects of VideoMix, it is applied only during the fine-tuning stage. The pretrained model is the same as the baseline. Table 7 shows the results. We observe that VideoMix improves the top-1 accuracies of SlowOnly-50 ($8\times 8$) and SlowFast-50 ($8\times 8$) by $+\mathbf{0.9}\%$ and $+\mathbf{0.8}\%$ against the baselines, respectively. VideoMix is effective on Something-Something-V2 as well.

We conduct weakly supervised temporal action localization (WSTAL) task on THUMOS’14 dataset [14]. THUMOS’14 dataset originally consists of 13,320 trimmed videos for training and 2,584 untrimmed videos for validation with 101 action categories. We follow previous WSTAL methods’ setting [20, 22, 27]. We train WSTAL models with the 20 class subset of the untrimmed videos, which consists of 200 untrimmed videos without temporal annotations. The temporal localization performance of a model is evaluated by 212 untrimmed videos with temporal annotations. WSTAL on THUMOS’14 dataset is a challenging task since the length of untrimmed video could be quite long (up to 26 minute) and multiple actions could exist in a video.

For training, we first extract I3D [2] features from the training videos as done in [20]. We sample $750$ video segments from a training video and RGB frames and optical flows are separately fed into dual-path I3D network. Each RGB and optical flow frame results in $1024$-dimensional feature, thus the dimension of extracted feature for a video is $750\times 1024$ for RGB input, and $750\times 2048$ for both using RGB and optical flow input (RGB+flow). The WSTAL model, which consists of two $3\times 3$ convolutional layers followed with LeakyReLU activation and a $1\times 1$ convolutional layer, takes the extracted features and predicts its class label. Since the network is trained with pre-computed features, we do not consider spatial VideoMix, but utilize temporal VideoMix on the extracted features. Implementation details are in Appendix D. For evaluation, temporal class activation mapping (T-CAM) [42, 22] is utilized to localize action instances along the temporal dimension. We threshold T-CAM below 50$\%$ of the highest value, and all 1-dimensional continuous segments are considered as action instance proposals as in [29]. Evaluation metric is average precision (AP) of intersection over union (IoU) thresholds from $0.1$ to $0.9$, and we report their mean value (mAP) as in [29, 22, 20].

Table 8 shows the WSTAL performances with and without optical flow features. We compare VideoMix against the baselines including Hide-and-Seek [29], which has been reported to improve weakly supervised object localization on static images and WSTAL on videos. We also compare against Mixup [40]. We observe that VideoMix improves the temporal localization accuracy mAP of baseline by $+\mathbf{0.9}\%$ and $+\mathbf{1.5}\%$ with and without optical flows, respectively. VideoMix also outperforms the Hide-and-Seek and Mixup in both scenarios showing its superior temporal localization ability. We also conducted VideoMix with a stronger baseline, W-TALC [24], and confirmed that VideoMix improves the performance of W-TALC from 31.1 to 32.3 (+1.2) mAP, which is at the state-of-the-art level.

AVA v2.2 dataset [10] consists of 235 training and 64 validation videos of human actions in videos. Each video is 15 minutes long, and the action locations are densely annotated as bounding boxes in space and time. We follow the protocol in [10] to train and evaluate the detection of 60 human action classes. We use the mean average precision (mAP) metric to measure the performance of video action detection using a frame-level intersection-over-union (IoU) threshold $0.5$.

Our detector is based on the Faster R-CNN [25] architecture, which is modified as in [7] to adapt to the video action detection task. The spatial stride of the final convolutional block is reduced from $2$ to $1$ to increase the feature map size. The 2D RoIAlign layer [12] is replaced by the 3D RoIAlign. SlowOnly-50 ($8\times 8$) and SlowFast-50 ($8\times 8$) have been used as the backbone network for the detection framework. We use the human bounding box proposals provided by [7] computed by an off-the-shelf human detector fine-tuned on AVA persons.

We initialize two detectors with the weights pretrained on Kinetics-400 with or without VidoeMix. We apply the same fine-tuning strategy afterwards to separate the effect of VideoMix on pretraining. We train detectors for $20$ epochs using the SGD optimizer with initial learning rate $0.1$ decayed by factor $0.1$ at $10$ and $15$ epoch. The spatial dimension of the shorter side is resized to 256 pixels while maintaining the aspect ratio. $64$ consecutive frames are extracted for training. Further details are in Appendix E.

Table 9 shows the performances of our detector on the AVA benchmark. Pretraining the detector with VideoMix improves the performance of SlowOnly-50 and SlowFast-50 to $\mathbf{20.4}$ ($+\mathbf{1.3}$) and $\mathbf{24.9}$ ($+\mathbf{1.7}$) mAP, respectively. Switching the pretrained weights to the VideoMix version introduces the gain in detection performance for free. The weights will be published in the future.