MetaNeRV: Meta Neural Representations for Videos
with Spatial-Temporal Guidance

In implicit neural representation for video, the video $V$ is estimated and parameterized by a neural function $f_{\theta}$ with $\theta$ as its weights (learnable parameters). A typical example of $f_{\theta}$ is a NeRV (Chen et al. 2021). We consider a more general class of $f_{\theta}$ where the model fits video by mapping the inputs to high embedding space and upscaling by the MLP layers. Specifically, the basic NeRV uses Positional Encoding  (Tancik et al. 2020) as its emz2bedding function, while the E-NeRV separately embeds the spatial-temporal context and is fused by the transformer  (Vaswani et al. 2017).

where $\gamma$ is frequency positional encoding, and $F$ is an MLP network encoder of the timestamp into a vector. Here $F_{\mathbf{f}}$ stands for a network that fuses the other information into the temporal embeddings like spatial information. $\mathbf{e}$ stands for the final embedding of the temporal or spatial-temporal information, and the input of the next upscaleed block.

The NeRV-based model uses MLP blocks as upscaled blocks, and the final embedding is upscaled to the output image size by stacking multiple layers of upscaled blocks. Suppose we have L layers, each of which performs an upscale operation. For layer $(l)((1\leq l\leq L))$, the upscaling operation can be expressed as:

where $(f_{l-1})$ is the output of the previous layer (for the first layer, the temporal embedding vector $\mathbf{e}$) and $W_{l}$ and $b_{l}$ are the weight and bias of layer $(l)$. $(\text{Upscale})$ can be any appropriate upscaling function, such as convolution, fully connected layers, etc. Here the NeRV block is shown in Fig. 9.

In each layer, feature transformations and nonlinear activations can be performed in addition to upscaling operations. This can be expressed as:

where $(\sigma)$ is a nonlinear activation function such as ReLU (Glorot, Bordes, and Bengio 2011) and GeLU (Hendrycks and Gimpel 2016).

After the (L) layer of processing, we obtain a representation $(f_{L})$ that matches the image aspect size. This representation can be considered as a feature map where each layer contains image information related to the input time (t).

In summary, the whole process can be formulated as:

here, $(f_{L})$ is the final output feature map, associated with the input time t and has the same spatial dimension as the target image. e is the temporal embedding function, $(W_{l})$ and $(b_{l})$ are the weight and bias of the layers, $\sigma$ is the nonlinear activation function, and $(\text{Upscale})$ is the upscaling function.

We apply gradient descent to fit the neural network. Let $\theta_{0}$ denote the initial network weights before any gradient steps are taken, and let $\theta_{i}$ denote the weights after $i$ steps of optimization. Basic gradient descent is applied following the rule:

where $\beta$ is a learning rate parameter with an optimizer, to track the gradient moments over time to redirect the optimization trajectory. Given $m$ optimization steps, different initial weights $\theta_{0}$ will lead to different final weights $\theta_{m}$ and loss $L(\theta_{m})$.

We adopt a combination of L1 and SSIM loss as our loss function for network optimization, which calculates the loss of overall pixel locations of the predicted image and the ground-truth image. Loss function as follows:

where $\alpha$ is the hyper-parameter to balance the weight for each loss component, and $F_{n}$ is the responding video frame of frame index $t$. The total number of video frames is $N$.

We adopt E-NeRV (Li et al. 2022) with $12.49M$ parameters as our baseline. We follow the same settings as in E-NeRV, like activation choice, as shown in Fig. 9. We set $d=d_{t}=196$ for spatial and temporal feature fusion and $d_{0}=128$ for temporal instance normalization. We set all the positional encoding layers in our model identical to E-NeRV’s positional encoding formulated, and we use $b=1.25$ and $l=80$ if not otherwise denoted.

In this work, we implemented the proposed MetaNeRV and other NeRV-based models  (Chen et al. 2021; Li et al. 2022; Chen et al. 2023; Lee et al. 2023) using the PyTorch framework. And we adopted their original implementations for training NeRV, E-NeRV, HNeRV, and FFNeRV.

In this section, we provide a detailed introduction to the application of spatial-temporal guidance in meta-learning methods. The algorithm 1 details the progressive training approach with a NeRV-based model. The network parameters are initially set with predefined values, denoted as $\theta$. The training commences with the first frame $F_{1}$ and progressively incorporates subsequent frames until all $N$ frames have been utilized. In the outer-loop iteration $j$, the network receives the top $T=j$ frames from $F_{1}$ to $F_{T}$ as input (if $j>N$, we choose $T=N$). The network computes the multi-resolution output $\{F^{\prime}_{k}\}_{K}$ based on the current parameters $\theta$ for all $T$ frames. Subsequently, a multi-resolution loss function evaluates the difference between the actual video $X_{j}$ and the network’s output $\{F^{\prime}_{k}\}_{K}$, generating a loss value $L_{i,j}$, where $i$ indicates the $i^{th}$ inner loop result and $j$ indicates the $j^{th}$ outer loop.

The algorithm updates the network parameters $\theta$ with the previous loss. This update is performed using gradient descent, specifically by calculating the average gradient across all losses incurred in iteration $t$ (i.e., $L_{1,j},L_{2,j},\ldots,L_{m,j}$). The learning rate $\eta$ regulates the step size in the parameter update. As the training progresses from iteration to iteration, more frames are incorporated, enabling the network to learn from a growing dataset. The updated equation from equation (3) can be represented as:

UCF101 (Soomro, Zamir, and Shah 2012) dataset is an extension of UCF50 and consists of 13,320 video clips, which are classified into 101 categories. These 101 categories can be classified into 5 types (Body motion, Human-human interactions, Human-object interactions, Playing musical instruments, and Sports). The total length of these video clips is over 27 hours. All the videos are collected from YouTube and have a fixed frame rate of 25 FPS with a resolution of 320 × 240.

EchoCP (Wang et al. 2021) dataset is an echocardiography dataset in contrast to transthoracic echocardiography (cTTE) targeting PFO (Patent foramen ovale) diagnosis. EchoCP consists of 30 patients with both rest and Valsalva maneuver videos which cover various PFO grades. The video is captured in the apical-4-chamber view and contains at least ten cardiac cycles with a resolution of 640 × 480.

EchoNet-LVH (Duffy et al. 2022) dataset is a standard echocardiogram study consisting of a series of videos and images visualizing the heart from different angles, positions, and image acquisition techniques. The EchoNet-LVH dataset contains 12,000 parasternal-long-axis echocardiography videos from individuals who underwent imaging as part of routine clinical care at Stanford Medicine. Each video was cropped and masked to remove text and information outside of the scanning sector. The resulting videos are at native resolution.

HMDB-51  (Kuehne et al. 2011) dataset is a large collection of realistic videos from various sources, including movies and web videos. The dataset is composed of 6,766 video clips from 51 action categories (such as “jump”, “kiss” and “laugh”), with each category containing at least 101 clips. The original evaluation scheme uses three different training/testing splits. In each split, each action class has 70 clips for training and 30 clips for testing. The average accuracy over these three splits is used to measure the final performance.

MCL-JCV  (Wang et al. 2016) dataset consists of 24 source videos with resolution 1920×1080 and 51 H.264/AVC encoded clips for each source sequence. Single-pass constant QP encoding (CQP) was used with the Quantization Parameter (QP) ranging from 1 to 51. More than 120 volunteers participated in the subjective test. Each set of sequences was evaluated by around 50 subjects in a controlled environment.

HOLLYWOOD2 (Marcin 2009): This dataset contains 3669 video clips, totaling 20.1 hours, featuring 12 human action and 10 scene classes from 69 movies, providing a benchmark for real-world action recognition.

SVW (Safdarnejad et al. 2015): Containing 4200 videos from the Coach’s Eye app, SVW covers 30 sports categories and 44 actions. Its amateur content poses significant challenges for automated analysis.

OOPS (Epstein, Chen, and Vondrick 2020): The OOPS dataset includes 20,338 diverse YouTube videos totaling over 50 hours, capturing unintentional human actions in various environments and with differing intentions.

Impact of Dataset Scale on Meta-Initialization training and Inference Performance. The dataset’s scale significantly affects the representation capacity and generalization of meta-initialization. As the dataset expands, meta-initialization’s representation capacity and generalization gradually enhance. Still, the computational cost also increases, resulting in longer training times for obtaining a superior meta-initialization.

Experimental results in figure 12 show that a straightforward combination of meta-learning and the NeRV-base model (MetaNeRV-NoG) benefits from larger datasets. However, its performance, even with 6000 training videos, doesn’t surpass that of our proposed method (MetaNeRV-STG) with just 200 videos. Our method, incorporating spatio-temporal guidance, improves with more training videos up to about 2000, after which performance stabilizes.

Our study demonstrates that adjusting the initialization parameters can markedly improve video representation effectiveness. Specifically, meta-initialization methods, with our proposed MetaNeRV-STG leading the way, consistently outperform random initialization across various reasoning steps. Notably, while both MetaNeRV-STG and MetaNeRV-NoG exhibit strong performance in the initial iterations, the effectiveness of MetaNeRV-NoG gradually diminishes as the number of iterations increases.

Figures 13 and figures 14 provide detailed insights into this conclusion. These figures present the statistical distributions of meta-initialization, obtained through training sets comprising different numbers of videos, across various inference steps on two datasets of 6000 videos. The videos are segmented and counted based on 1.5 PSNR value intervals. The curves in these figures are color-coded to indicate varying training video quantities, with 0 representing random initialization and 50 denoting a training set of 50 videos. From left to right, the distributions correspond to different inference steps, with the upper layer showcasing the MetaNeRV-STG method and the lower layer depicting the MetaNeRV-NoG method. The visual representations clearly illustrate the superiority of our meta-initialization techniques, particularly MetaNeRV-STG, in enhancing video representation effectiveness throughout the inference process.

More OOD Quantitative results. Note that our method significantly outperforms baseline methods on independent (a.k.a. OOD) test datasets even when trained on a single real-world dataset, demonstrating the robustness and generalizability of MetaNeRV. The ablation studies on ultrasound datasets prove the effect of different guidance, as shown in Fig. 11. As illustrated in Fig.10(a), we utilized parameters trained on the HMDB-51 dataset as the optimal initialization parameters for direct inference on the three new datasets. The solid line represents our method, while the dashed line represents the E-NeRV method. The other three sub-figures in Fig.10 demonstrate the performance of our method on the OOD dataset. It can be observed that the optimal initialization parameters trained on the real-world dataset can be well transferred to other real-world datasets, indicating that our method is not affected by data distribution and can be quickly transferred to other datasets with just training on one real-world dataset.

More OOD Qualitative results. When adopting the model to specialized domains, we acknowledge that the optimal initial parameters should ideally be obtained from data with the same distribution. However, due to diverse samples in the training set, our model trained on real-world datasets exhibits good generalization capabilities compared to random initialization. See results in Tab. 3. Parameters trained on general real-world datasets still generalize well to the ultrasound dataset, performing significantly better than random initialization. This highlights the effectiveness of our model’s generalizability.

Additional PSNR result for the video representation task on the MCL$\_$JCV (Wang et al. 2016) dataset is summarized in Tab. 4. This comparison underscores the proficiency of our method in substantially decreasing convergence time.

More visualization results. Additional visual comparisons between the outputs from E-NeRV (Li et al. 2022) and MetaNeRV are given in Fig. 15, Fig. 16, Fig. 17, Fig. 18. Despite having a few steps of iteration, the output from MetaNeRV is still noticeably better, with more detail from the original video frames preserved.

From Fig. 15 and Fig. 16, we observe that our approach can capture the approximate colors and shapes of the ground truth video in the first iteration by altering the initialization weights. In the second iteration step, our method has closely approximated the real video, reconstructing the primary cartoon characters and the background. Meanwhile, the shapes of the cartoon characters and background in the image reconstructed by E-NeRV are more pronounced. In subsequent iteration steps, our method has achieved sufficient clarity to discern the finer details of the cartoon characters and the background.

Our approach performs exceptionally well on medical datasets as observed in Fig. 17 and Fig. 18. We hypothesize that the videos in medical ultrasound datasets are often collected using similar ultrasound equipment. Consequently, the initialization representation learned by our method already encapsulates the blurred sector shape and black background characteristic of medical ultrasound videos. Additionally, the monochromatic nature of ultrasound videos contributes significantly to the performance enhancement of our model. Our method can achieve representation results similar to the original video after just one iteration, and subsequent iterations primarily focus on refining the details within the video. In contrast, random initialization typically learns the black background within a few iterations but struggles to capture fine details, resulting in only a rough shape representation.