Dancing with Still Images:
Video Distillation via Static-Dynamic Disentanglement

Video Temporal Redundancy. Most current works are dedicated to developing and improving methods for compressing image datasets in terms of quantity. Video data have an additional temporal dimension as its major difference from images, suggesting the presence of temporal redundancy. Video temporal redundancy study has a long history [14, 9]. Researchers have long observed the significant temporal redundancy, which has diverse causes. For example, videos inherently exhibit substantial similarity between adjacent frames, leading to low temporal information utilization during data usage. Thus, we focus on, analyze, and exploit the temporal compression for video distillation.

Precondition for Temporal Correspondence. To study the temporal compression, we are categorizing the temporal correspondence for the distillation matching (e.g. gradient/trajectory/distribution matching) between real and synthetic videos. We can put all scenarios into one formulation, which covers all possible methods from the temporal aspect:

Fig. 2(a) gives an example of compressed video distillation. Though the existing distillation methods usually produce irregular and “freeform” patterns that are cryptic for humans, on video data, we intuitively desire an algorithm that obeys temporal consistency. Otherwise, the video algorithm could degrade due to the loss of correct temporal information. The temporal consistency is ensured by two preconditions: the real and synthetic frames sampled for distillation should be in the correct order and follow a uniform flow of time. More specifically, the orderedness is:

That is, all frame sequence pairs are comparable and associated with $\leq_{T}$ and both real and synthetic sequences are in the correct time order. The matching strategy in Fig. 2(b) does not meet orderedness as the two matching pairs at right are in the wrong temporal order.

All the real frame sequences have the same length, as the same to synthetic frame sequences. This ensures that the synthetic video we learn follows a uniform flow of time, e.g., the matching strategy in Fig. 2(c) leads to non-uniform FPS, and the synthetic frames at left will learn a much smaller FPS than the frames at right. We also justify the uniformity with experiments in the supplementary.

These two preconditions narrow the searching space of temporal condensation strategies, enabling us to systematically categorize and analyze the compressed video distillation. In Fig. 3(a), the naive algorithm may match all synthetic frames to real frames. To compress the temporal dimension, we can either reduce the number of frames (Fig. 3(b)) or reduce the correspondence by temporal segmentation (Fig. 3(c)). Note that we do not force orderedness and uniformity within each frame sequence pair for matching, since the distillation matching algorithms themselves could drive the synthetic data to follow the consistency.

To take a step further, we propose Segmented Matching and Interpolation framework for the quantitative comparison and analysis of temporal condensation (Fig. 3(e)). Given a target synthetic frame number $L_{syn}$, to approach flexible compression rate, we distill $N_{real}$ real frames to $N_{syn}$ frames and interpolate the frames to our target $L_{syn}$. Specifically, the real video and synthetic video are segmented evenly for the pairwise distillation matching, which is the only valid strategy for temporal consistency. The interpolation enables the compression of synthetic video. This paradigm covers most video distillation methods from the perspective of temporal compression, and the extent of compression can be parameterized by the four factors:

(1) Number of Independent Synthetic Frames ($N_{syn}$) is the size of the trainable synthetic frames. These frames are learned with dataset distillation algorithms and will be interpolated to the target length of synthetic video $L_{s}$. Smaller $N_{syn}$ indicates a larger temporal compression rate.

(2) Number of Real Frames ($N_{real}$) is the size of real frames for distillation matching. Larger $N_{real}$ implies a larger receptive field for synthetic video during distillation.

(3) Number of Segments ($K$). We cut the real and synthetic videos into the same number of segments and apply distillation between the pairs of real and synthetic segments. Larger $K$ could reduce the training time since the distillation algorithm receives smaller segments with fewer frames.

(4) Interpolation Algorithm ($\mathcal{I}$) interpolates the $N_{syn}$ independent synthetic frames to the required synthetic video length. Our algorithms are detailed in Sec. 3.3.

So the level of temporal compression can be uniquely determined by a quadruplet $(N_{syn},N_{real},K,\mathcal{I})$. We show some examples in Fig. 4 with different combinations. In the following section, we will compare and analyze these four axes and put forward some empirical conclusions to drive future studies on video distillation.

To investigate the effects of different temporal compression levels, we conduct comprehensive experiments with different $N_{syn}$, $N_{real}$, $K$, and $\mathcal{I}$. In the following experiments, the DM [30] algorithm is adopted and we use ConvNet (3-layer convolutional network from [31]) with one layer GRU head [5] as our backbone network.

Comparison of $N_{syn}$ and $N_{real}$ values. We first compare the models with different $N_{syn}$ (number of independent synthetic frames) and $N_{real}$ (number of real frames) and visualize in Fig. 5. We implement the segmented matching segment number $K=1$. The performance is Fig. 5(a) shows that: (1) distilling the video to still image could yield decent accuracy (over 17%), (2) larger $N_{real}$ and $N_{syn}$ brings minor performance gain (less than 3%), (3) the model degrades when $N_{real}<N_{syn}$ (left-top of the figure), potentially due to insufficient temporal information in the real data. Fig. 5(b) and (c) show that larger $N_{real}$ and $N_{syn}$ lead to significant memory consumption, e.g. model with $N_{real}=N_{syn}=16$ takes $16\times$ GPU memory than a $N_{real}=N_{syn}=1$ model. And models with large $N_{real}$ also take more training time. And we can also read Fig. 5 diagonally to fix the per-frame receptive field $N_{real}/N_{syn}$, and basically larger ratio leads to better performance.

Comparison of $K$ values (segments). We study the effects of $K$ in Fig. 6. The segmentation could notably reduce memory consumption (up to 40%) while maintaining the training speed. However, the efficiency is achieved at the cost of performance as the segmentation decreases the “receptive field” of each synthetic frame.

Comparison of Interpolation Algorithms $\mathcal{I}$. $\mathcal{I}$ is critical to compressed video distillation, especially when $N_{syn}$ is small. We use various interpolation methods: (1) Duplication is simply copying the learned synthetic frames to the required length, or namely nearest interpolation. e.g. a 2-frame video $\left[f_{1},f_{2}\right]$ can be interpolated to 4-frame video $\left[f_{1},f_{1},f_{2},f_{2}\right]$. (2) Linear interpolation generates intermediate frames by blending adjacent reference frames. The frame $f_{t}$ at time $t$ will be the weighted sum of nearest reference frames $f_{t_{1}}$, $f_{t_{2}}$ according to their temporal distance $t_{2}-t$ and $t-t_{1}$. e.g. a 2-frame video $\left[f_{1},f_{2}\right]$ can be interpolated to 4-frame video $\left[f_{1},\frac{2f_{1}+f_{2}}{3},\frac{f_{1}+2f_{2}}{3},f_{2}\right]$. (3) Parametric interpolator is a pre-trained interpolation network on the real video dataset. For each video data with $L_{syn}$ frames, we evenly sample $N_{syn}$ frames and duplicate them to length $L_{syn}$. We train a TimeSformer [2] model $\varphi$ on these real data and it learns to recover the original video from the duplicated one. The pretrained model $\varphi$ can be utilized for interpolation, e.g. a 2-frame video $\left[f_{1},f_{2}\right]$ can be interpolated to 4-frame video $\varphi(\left[f_{1},f_{1},f_{2},f_{2}\right])$.

We compare the three interpolators in Tab. 1. The simple duplication method outperforms linear interpolation. The parametric method performs the best, especially on larger frame numbers, as it encodes dataset-specific inductive bias to compensate for the loss of temporal information.

Discussion. With the comparison of the four dimensions of temporal compression, we have some fundamental observations that could offer direction for our further study of video distillation: (1) Temporal compression confers significant advantages to dataset distillation and static images could encode more than little knowledge for video datasets; (2) Segmented distillation could reduce the distillation cost, but significantly sacrifice the model performance (3) Parametric interpolation could compensate for the loss of temporal dynamic information in the video.

The results in Sec. 3.3 indicate that static information in videos is more critical to the distillation task, given the limited capacity of small synthetic data. Hence, we use a $N_{syn}=N_{real}=K=1$ setting to learn a static memory with only one frame. The specific distillation process involves selecting one frame randomly from each video segment to form an image dataset in each epoch. The DC [31] method is then applied for gradient matching on a convolutional network. Since a new image dataset is created in each epoch, the “image” we distill has ideally observed all video frames and distilled the static memory from them.

The choice of dynamic memory can be diverse. In this paper, our dynamic memory is represented as multiple frames of single-channel images. We use a network $\mathcal{H}$ which takes static memory and dynamic memory as input and outputs video clips. At this stage, we fix the static memory and use different matching methods (performance matching, distribution matching, and parameter matching) to simultaneously update the network $\mathcal{H}$ and dynamic memory.

We use a concrete formula to explain our paradigm. We refer $\mathcal{A}(\mathcal{T}_{syn},\mathcal{T})$ as a matching loss of distillation where $\mathcal{T}_{syn}$ is synthetic dataset and $\mathcal{T}$ indicates origin dataset $\{(x_{i},y_{i})\}_{i=1}^{|\mathcal{T}|}$, $x_{i}\in\mathbb{R}^{f_{i}\times c\times h\times w}$ , $y_{i}\in\{0,1,\dots,C-1\}$. Given a frame selection method $\mathcal{B}(\mathcal{T},N)$ selecting $N$ frames from $\mathcal{T}$ to obtain a dataset with $x_{i}\in\mathbb{R}^{N\times c\times h\times w}$, we summarize our paradigm in Alg. 1.

We adopt small video datasets UCF101 [19] and HMDB51 [11], and large-scale Kinetics [3] and Something-Something V2 [8] in our experiments. UCF101 [19] consists of 13,320 video clips in 101 action categories while HMDB51 [11] consists of 6849 video clips in 51 action categories. Kinetics [3] is a collection of video clips that cover 400/600/700 human action classes while SSv2 [8] covers 174 motion-heavy classes. To evaluate the distillation algorithms on more diversified data scales, and considering the efficiency of experiments and the clarity of model comparisons, following the scale of pioneering work on image distillation [25], we build a miniaturized version of UCF101, named MiniUCF, including 50 most common classes from the UCF101 dataset. This miniaturization enables rapid iterations of our method and facilitates observing relatively significant changes in performance. We report the top-1 classification accuracy for MiniUCF and HMDB51, and the top-5 classification accuracy for Kinetics400 and SSv2.

The baseline methods involve: (1) coreset selection methods (random selection, Herding [26] and K-Center [18]) following the implementation for image distillation in DC [31]. (2) direct adaptation of the common image distillation methods (DM [30], MTT [4], FRePo [32]) to the video distillation task. (3) image distillation method (DC [31]) with frame duplication for a “boring videos” proposed by us, namely “Static-DC”.

Data. For MiniUCF and HMDB51, the videos are sampled to 16 frames with sampling interval 4 dynamically, $i.e.$. the frames indices vary in different epochs. Following the setup in C3D [21], each of these frames is cropped and resized to 112x112. For Kinetics-400 and Something-Something V2, the videos are sampled to 8 frames before the distillation, and the frames are cropped to 64x64. We only use horizontal flipping with a 50% probability for data augment.

Static Learning. We use DC [31] to distill static memory with random real frame initialization. We utilize a 4-layer 2D convolutional neural network for distillation (ConvNetD4) and perform an early stop in the distillation training when the loss converges. Interestingly, our experiments indicate that static memory is not necessarily trained to full convergence, as dynamic memory compensates for it.

Dynamic Finetuning. In dynamic fine-tuning, we adopt distillation methods in various types, including DM [30], MTT [4], and FRePo [32] to evaluate the broad applicability of our paradigm. Dynamic memory is initialized with random noise. We use a small 3D CNN (referred to as MiniC3D) for distillation. The $\mathcal{H}$ network used for combining static and dynamic memory is also a MiniC3D. For more details, please refer to the supplementary.

Fair Comparison between the baseline and our method. We rigorously ensure that the total storage space for static memory, dynamic memory, and the $\mathcal{H}$ network is smaller than the corresponding IPC (Instance Per Class). Specifically, on DM and MTT, we use no more than 82% of the storage space corresponding to the baseline, which amounts to 2 static memory with 2 dynamic memory for every instance. On FRePo, we use no more than 42% of the storage space corresponding to the baseline, which means 1 static memory with 1 dynamic memory for every instance.

Evaluation of Distilled Dataset. Naturally, we evaluate how well our synthetic data performs on architectures used to distill it. When evaluating data distilled by FRePo, the results should be considered as a reference only due to the label learning conducted by the method itself and the use of a different optimizer. We also evaluate how well our synthetic data performs on different architectures from the one used to distill it on the MiniUCF, 1 instance per class task.

Hyper-Parameters. Considering the numerous parameters involved in the experiments, we detail the parameter settings for all experiments in the supplementary.

We show the results of our small-scale experiments in Tab. 2 and large-scale in Tab. 3. The full dataset indicates the accuracy of the network trained on the full dataset.

Comparison to Coreset Method. Following image distillation, we compare our method with coreset selection methods on MiniUCF and HMDB51 in Tab. 2. In the majority of cases, our approach outperforms the coreset selection. Regarding coreset methods, we also observe: (1) On average, the herding method proves to be the most effective coreset approach. (2) With an increase in IPC, the performance of herding exhibits a notable improvement. These conclusions align with earlier experiments [31] in image distillation, lending credibility to our findings.

Comparison to Other Methods. We compare our final method with other methods we proposed in Fig. 1. Among the three methods we proposed, Static-DC (Fig. 1(b)) exhibits the poorest performance. Compared to both the coreset method and Static-DC, the naively adapted method (Fig. 1(a)) shows a significant improvement. This underscores the applicability of existing image distillation techniques to videos and the effects of dynamic information in video understanding. In comparison, our final method (corresponding to Fig. 1(c)) achieves a remarkable superiority over all other methods through static and dynamic disentangle learning. In most cases, our method could enhance the current distillation methods while requiring less storage. However, the performance of our method on Kinetics-400 is not as strong as on the other two smaller datasets. This is because Kinetics-400 itself has a much larger number of categories and samples compared to the other two smaller datasets, which increases the difficulty and cost of distillation. Since our $\mathcal{H}$ network is shared, having 400 categories sharing one $\mathcal{H}$ network in Kinetics-400 might cause interference during learning. Allocating different $\mathcal{H}$ networks to different categories or using a more complex $\mathcal{H}$ network could potentially offer better improvement in our method. However, this approach would impose a greater training burden and significantly reduce the practical value of distillation, so we do not conduct larger-scale experiments.

Cross Architecture Generalization. We also show the result of cross-architecture generalization in Tab. 4. The experimental results indicate that the data obtained by static-dynamic disentanglement performs much better on other networks compared to the naively adapted method.

Ratios of Static and Dynamic. We report results for MiniUCF 1 Instance per class task with different static and dynamic memory ratios in Tab. 5. We can find that increasing the quantity of static and dynamic memory both improves the scores. However, as their numbers increase, the time and GPU memory required for Static Learning and the training convergence time for Dynamic Fine-tuning under the same computational power will also increase (Tab. 5). Considering training efficiency and effectiveness while ensuring the storage does not exceed the corresponding IPC, a balanced choice is to use 2 static with 2 dynamics for every instance.

Impact of Video Dynamics on Distillation. Actions in the video exhibit varying degrees of dynamics. To explore the impact of video dynamics, we categorized all classes of MiniUCF into two groups (relatively static, highly dynamic) based on their level of dynamics, calculated by the average Hamming distance between inter-frame features for each class, and compared their test accuracy on networks trained with distilled data (Fig. 8). We observe that (1) data distilled by the Static-DC method is more sensitive for static classes, which aligns with our expectations as this method generates data that lacks dynamic information, akin to a “boring video”. (2) The naively adapted MTT, in comparison, can distill more useful information but still shows higher scores for static classes than dynamic classes. (3) MTT+Ours, however, demonstrates better distillation of dynamic information compared to the previous methods and exhibits significant improvements on dynamic classes.

To observe the temporal changes in the distilled videos, we sampled frames from the videos obtained using different methods and visualized their inter-frame differences. We show two examples in Fig. 9 and more in the supplementary. Although visually abstract, we can still conclude that the distilled videos indeed exhibit temporal variations.

We utilize a pre-trained four-layer 2D convolutional network to extract features from individual frames. Subsequently, we compute the Hamming distance between the features of consecutive frames. Then, we derive the average inter-frame Hamming distance for each video segment, allowing us to ascertain the average inter-frame Hamming distance for each class. We classify 50% of the classes with smaller average inter-frame Hamming distances into the static group, while the remaining 50% with larger distances are designated as the dynamic group. In Suppl. Tab. 6, we highlight the static and dynamic groups using distinct colors.

We show more detailed results in Suppl. Tab. 8. We can observe that both increasing static memory and dynamic memory concurrently enhance the accuracy of static and dynamic classes. Additionally, we note that when the disparity in quantity between static memory and dynamic memory becomes larger, there is an imbalance in the accuracy of the static group and dynamic group. The reason for this lies in the random pairing of dynamic memory and static memory during sampling. If there is an excess of dynamic memory, multiple instances of dynamic memory may end up learning the same content during training, essentially augmenting the static information. Therefore, we recommend maintaining a 1:1 ratio between static memory and dynamic memory when utilizing our paradigm.

Our distilled video data could also generalize beyond the classification task. We show some visualized optical flow extraction following [1] in Fig. 10.

We show more visualized inter-frame differences of MTT [4] and MTT+Ours for MiniUCF IPC=1 in Fig. 12 and Fig. 13 (last pages).

In this section, we provide a detailed introduction to models used in the experiments, including MiniC3D, CNN+GRU, and CNN+LSTM. Additionally, we compare their performance with C3D[21] on the UCF101[19] and HMDB51[19] classification tasks.

The temporal analysis experiments in Sec. 3.3 are conducted with DM [30] algorithm and CNN+GRU model as detailed before. 16 frames are sampled from each video with temporal stride 12, and we set the target synthetic video length also 16. For a fair comparison of time and space complexity, all the experiments are run on one NVIDIA V100 GPU (16GB), 8 cores of Intel Xeon 5218 CPU, and 20 GB memory. The learning rate for synthetic images is set to 1.0 and that for network updating is set to 0.01. The models are trained for 10,000 iterations with a real batch size of 64, which is observed as enough for the training convergence in our experiments.

In the experiments, we initially fine-tune the parameters for methods without our paradigm (naively adapted methods) to achieve the best possible results. To ensure a fair comparison, we strive to maintain consistency in all other parameter settings, making adjustments only to the learning rates associated with the unique dynamic information and $\mathcal{H}$ in the methods with our paradigm.