Efficient Meta-Tuning for Content-aware Neural Video Delivery

In this section, we present our method to significantly accelerate the training of content-aware SR models. Our method adapts the meta-learned model to the first chunk of the video, and sequentially adapts partial parameters of the previous model to the following chunks. The partial parameters are selected by gradient masking and the challenging patches are extracted for adaption. The pipeline of our method is illustrated in Fig. 1. We first introduce Efficient Meta-Tuning (EMT) to sequentially deliver the models from a meta-learned model in Sec. 3.2. Then we propose a novel challenging patch sampling strategy to further accelerate EMT in Sec. 3.3.

Following former works [19, 29], we uniformly divide the input video into chunks, and train the corresponding content-aware SR model for each chunk. [29] proposes to apply deep super-resolution networks to video delivery by training one model for each chunk from scratch. [19] presents a method to compress all the models by one shared model and a few private parameters. However, both methods train the content-aware SR models from scratch, resulting in huge computational cost. Since neighboring chunks are temporally consistent, fine-tuning is much more reasonable compared with training from scratch. Precisely, finding a generic initial model that can not only generalize over diverse video chunks but also adapt rapidly to any specific video chunk, plays a key role in fine-tuning. In order to obtain a better initialization, we adopt MAML [9] to train a meta-learned model. Although MAML has been applied to Zero-Shot SR [21, 24] and video frame interpolation [6], it has never been studied in neural video delivery to the best of our knowledge. Compared with random initialization or pretrained initialization, a meta-learned model has better transferability. To obtain the content-aware SR models, we sequentially fine-tune partial parameters of the previous model. In contrast to fine-tuning the whole model, our method can compress the parameters of all models into one shared meta-learned model and a few partial parameters.

Meta-Learned Initialization We take one chunk as a specific task and aim to learn a SR model that can adapt to various chunks. Specifically, we first pretrain the SR model $f_{\theta}$ on DIV2K [1], then we utilize meta learning to optimize the pretrained parameters as illustrated by Alg. 1. This step enables the SR model to converge to a transferable point, which can be rapidly fine-tuned. In order to build a variety of tasks, we collect several video sequences and uniformly divide them into video chunks. Totally, we obtain $N$ chunks for meta-learning, and each chunk is set as the task $t_{i}$. The collected dataset is denoted as $D_{N}$. We apply bicubic downsampling to the frames and generate LR-HR pairs $(I_{LR}^{i},I_{HR}^{i})$. Our goal is to optimize $f_{\theta}$ according to each LR-HR pair by minimizing the L1 loss as shown in Eq. 1.

During the inner loop (Line 4-7), we conduct one or more gradient updates for the task $t_{i}$ in each iteration. The temporary model for task $t_{i}$ is denoted as $f_{\theta i}$. During each inner gradient update, the task-specific parameters are updated according to Eq. 2, where $\alpha$ is the inner learning rate.

As for the outer loop (Line 9-10), we evaluate the loss of $f_{\theta i}$ on each $t_{i}$ and sum up the losses of all tasks to update the SR model $f_{\theta}$. For one outer gradient update, it considers the gradients from all tasks. The outer update can be formulated as Eq. 3, where $\beta$ is the outer learning rate.

Partial Model Adaption The meta-learned model $f_{\theta}^{m}$ is shared by all the video chunks. To obtain the content-aware SR models for the input video, we adapt the meta-learned model to the first chunk, and sequentially fine-tune the partial parameters of previous adapted model. Formally, we denote the content-aware SR model of ${j^{th}}$ chunk as $f_{\theta}^{j}$. For the first chunk, we rapidly fine-tune the meta-learned model $f_{\theta}^{m}$ to obtain the adapted model $f_{\theta}^{1}$. We use the gradient masking to select ${p_{1}}\%$ most significant parameters before adapting $f_{\theta}^{m}$. As for other chunks, we fine-tune previous model $f_{\theta}^{j-1}$ on ${j^{th}}$ chunk, delivering the adapted model $f_{\theta}^{j}$. We also adopt the gradient masking to select ${p_{2}}\%$ most significant parameters before fine-tuning $f_{\theta}^{j-1}$. It has to be noted that we adopt different percentages for the first chunk and other chunks in this work. Our partial model adaption requires much fewer epochs for both the first chunk and other chunks. Therefore, the computational cost can be greatly reduced compared with training from scratch under the same performance.

Gradient Masking In order to compress the parameters of content-aware models, we design a simple yet effective strategy to find the ${p}\%$ most significant parameters. Given a reference model $f$, we need to find a fraction of parameters before fine-tuning $f$. Specifically, we adopt a few iterations to update $f$, obtaining a temporal model $\widehat{f}$. Afterwards, we calculate $|{\theta_{\widehat{f}}}-{\theta_{f}}|$ and choose the ${p}\%$ parameters that vary most, delivering the parameter mask $M({\theta_{f}})$.

Once we collect the $p\%$ most significant parameters, we can fine-tune the reference model $f$ and simply update the significant parameters. In this manner, our method only needs to store $p\%$ private parameters of the reference model. When fine-tuning for the first chunk, we choose the meta-learned model $f_{\theta}^{m}$ as the reference model. For ${j^{th}}$ chunk of the input video, we choose the previous adapted model $f_{\theta}^{j-1}$ as the reference model.

Previous adapted SR model already possesses the ability to super-resolve current chunk due to the temporal consistency. Thus it can achieve satisfying results on the majority of regions. However, some hard regions are still challenging for the previous adapted model. Since SR networks are fine-tuned on sampled LR-HR patch pairs, we further present a strategy to sample the ${r\%}$ most challenging patches for EMT. Our strategy is highly efficient and brings negligible additional cost. Inspired by the video codec, it first locates the positions of challenging patches in I-frames, then propagates the positions to in-between frames as shown in Fig. 2.

Formally, we denote the frames of input video as ${I_{1}},...,{I_{T}}$, where $T$ is the number of frames. The I-frames are denoted as ${I_{{k_{1}}}},...,{I_{{k_{M}}}}$, where ${k_{1}},...,{k_{M}}$ are the indices of $M$ I-frames. We also denote the chunk indices of $M$ I-frames as ${c_{1}},...,{c_{M}}$. In order to localize the challenging patches for ${I_{{k_{m}}}}$, we run the inference of previous adapted model $f_{\theta}^{{c_{m}-1}}$ to super-resolve the downsampled I-frame $I_{{k_{m}}}^{LR}$:

We can calculate the PSNR between $I_{{k_{m}}}^{SR}$ and ${I_{{k_{m}}}}$ in terms of all possible patches as illustrated by Fig. 2. The time of calculating the PSNR map for 1080P frames is 0.1 seconds. Therefore, it is time-consuming to produce PSNR maps for all frames. For instance, when dealing with a 45-second video, it usually takes around 135 seconds to generate PSNR maps for all frames. Instead, we localize the positions of ${r\%}$ (r=20) most challenging patches in ${I_{{k_{m}}}}$, and then extract the patches from frames between ${I_{{k_{m}}}}$ and ${I_{{k_{m+1}}}}$ according to the coordinates. As the frames between two I-frames are temporally consistent, the localized positions at I-frame can also choose reasonable patches on the in-between frames. Since I-frame is very sparse within a video, the computational cost is negligible. For a 45-second video, the total time of position localization and patch extraction is 0.7 seconds. However, the results of EMT using challenging patches are the same as results using all frames to some extent. In this way, the training efforts of EMT focus on challenging patches, resulting in faster convergence.

For meta-learning, we conduct two gradient updates for each individual task in the inner loop. After updating on all sampled tasks, we conduct one outer gradient update on the SR model. We randomly select training patches with a resolution of $144\times 144$ and set mini-batch size as $16*n$, where n is the number of sampled tasks. Particularly, we set n to 15 and each task consists of 50 frames. We set inner learning rate $\alpha=0.5e-5$ and outer learning rate $\beta=1e-3$. We adopt Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$. ESPCN serves as the default architecture, x2 is utilized as the default scaling factor and PSNR is the default metric.

For fine-tuning, we conduct experiments on two video lengths, including 45 seconds and 2 minutes. The batch size is 16 and learning rate is $1e-4$. We set ${p_{1}}\%$ as 20% and ${p_{2}}\%$ as 1% to compress the parameters as default. We design three settings of our method, including S, M, and L. S and M settings adopt 0.1 epoch and 3 epochs for fine-tuning respectively. As for L setting, we alter ${p_{1}}\%$ to $100\%$. We conduct all the experiments on NVIDIA V100 GPUs.

In this section, we compare our method against the baseline [29] and two codec standards. The baseline uniformly divides a video into chunks and trains one SR model for each chunk from scratch with 300 epochs. We denote the baseline as $C_{1-n}$. For the two commercial codec standards H.264 and H.265, we use ffmpeg with libx264 codec and libx265 codec to compress the HR videos to lower bit-rate while maintaining the resolution. The compressed videos are of the same storage size as our method (LR videos and SR models). We report three variants of our method under S, M, and L settings. Under the L setting, we aim to show the potential of meta-tuning by updating all the parameters for the first chunk. As shown in Tab. 1, our method achieves better performance with less time and parameters compared with baseline [29]. Our results with 0.1 epoch already outperform the baseline with 300 epochs. In terms of parameter compression, given a video with n chunks, we compress all models $n*P$ to $1*p_{1}\%P+(n-1)*p_{2}\%P$. In comparison with H.264 and H.265, our results outperform H.264 and H.265 in most cases as shown in Tab. 2. We also show the qualitative comparison in Fig. 3. As can be seen, our method can restore better details compared with codec standards.

In this section, we compare our work with other neural video delivery methods. CaFM [19] uses content-aware models to compress parameters and achieve competitive performance. However, its joint training strategy takes huge computational cost. Therefore, CaFM is not practical for delivering long videos. SRVC [13] also sequentially delivers the content-aware SR models by fine-tuning previous model. However, they fail to generalize on various architectures and have to train from scratch for a new video. Deep Video Compression (DVC) [20] is an end-to-end DNN-based video compression method. We also compare our method with (DVC) at four different bitrate-distortion trade-off operating points $\lambda\in\{256,512,1024,2048\}$ (DVC1, DVC2, DVC3, DVC4). In Fig. 4 and Tab. 3, we demonstrate the advantages of our method in terms of accuracy, training time, and storage. In Fig. 4, we calculate the average PSNR and storage cost on 45s videos from VSD4K [19]. Under the same storage, our method outperforms other methods at most circumstances. Though CaFM achieves promising results, it takes a huge computational cost. As shown in Tab. 3, we demonstrate the trade-off between accuracy and training time. Our method shows competitive performance while maintaining low computational cost.

Variants of EMT We intend to evaluate the contribution of each component in EMT. Firstly, we compare our method with pretrained initialization, which is denoted as $P_{1-n}$. Similar to our meta-learned initialization, $P_{1-n}$ first initializes the SR model on DIV2K, and is normally finetuned on the meta-learning dataset $D_{N}$. We replace the meta-learned model of EMT by the normally finetuned model to obtain the results of $P_{1-n}$. To be mentioned, $P_{1-n}$ still utilizes gradient masking and challenging patch sampling for a fair comparison. Therefore, we are able to evaluate the effectiveness of meta-learned initialization. To evaluate the effectiveness of Challenging Patch Sampling (CPS), we remove the step of patch sampling in our full pipeline and the results are denoted as $MT_{1-n}$. As shown in Tab. 4, in order to achieve the same PSNR, $P_{1-n}$ takes extra cost compared with our meta-learned initialization. Meanwhile, our method outperforms $MT_{1-n}$ in regard to efficiency, demonstrating the effectiveness of CPS.

Variants of Meta Learning During meta-learning, we randomly sample 15 tasks and each task contains 50 frames. We also study the effect of different numbers of tasks and frames. For the number of tasks, we compare the results of 10, 15 and 20. For the number of frames, we evaluate the results of 10, 50 and 150. As shown in Tab. 5, all the variants achieve similar results, and the setting of 15 tasks with 50 frames already achieves competitive performance.

Variants of Gradient Masking We conduct extensive experiments to explore the performance under different variants of gradient masking. Since our method uses different portions of parameters for fine-tuning the first chunk and other chunks. We report the results of ${p_{1}}\in\left\{{10,20,30,100}\right\}$ and ${p_{2}}\in\left\{{0.5,1,5,10}\right\}$. As can be seen from Tab.6, we empirically set ${p_{1}}=20$ and ${p_{2}}=1$ as our default setting since it can already achieve satisfying results.

Generalization of Various Scaling Factors We evaluate the generalization ability of EMT using ESPCN as the backbone across scaling factors x2, x3 and x4. As shown in Tab. 7, EMT outperforms $C_{1-n}$ under various scaling factors, demonstrating the generalization ability of EMT across various scaling factors.

Generalization of Various Efficient Backbones In this part, we present the results of our method using different efficient SR backbones. We evaluate our method on 45 seconds videos and adopt three additional efficient backbones, including SRCNN [8], FSRCNN [8], and EDSR with one residual block in body (EDSR-1). As shown in Tab. 8, our method also generalizes well to different efficient backbones, validating the generalization ability of our method.