MNeRV: A Multilayer Neural Representation for Videos

Implicit neural representations are a novel signal representation method that approximates a mapping function by fitting a neural network. Implicit neural representations have powerful modeling capabilities for various signals, such as data compression[23; 24; 25], 3D reconstruction[26; 27; 28; 29; 30], and 3D view synthesis[26; 27; 28; 29; 30], etc. In the early days of implicit neural representations for videos, it was usually pixel-wise encoding and decoding, which specifically trained the model to learn the mapping relationship between the coordinates of a point and its RGB value[31; 32]. This method has high training costs, slow encoding and decoding speeds, and lower compression rates.

NeRV first proposed an implicit representation method for image-wise videos, which trains and fits videos into a neural network using convolution and pixel shuffle. In this way, the decoding process of the video is transformed into the inference process of the model, which greatly improves the decoding speed of the video. In addition, due to the characteristics of neural networks themselves, NeRV-like videos also perform well in downstream video tasks such as video interpolation and video restoration, attracting more and more researchers to study and improve it. Bai et al[33] balanced the coordinate-based implicit neural representation (INRs) and the image-based implicit neural representation (NeRV). They introduced the idea of partitioning into NeRV, representing videos as multiple image blocks, each with a coordinate, and adding AdaIN to the network to improve the fitting effect. Li et al[34] improved the encoder and decoder layers based on NeRV and proposed E-NeRV, further removing redundant structures in the neural network and introducing a spatiotemporal context-based encoder, achieving an experimental effect that converges 8 times faster than NeRV. Mai et al[35] introduced a motion-adjustable neural implicit video representation, which maps time to a driving signal to modulate the frame-generation process, and achieved good results. Lee et al[36] further improved NeRV by introducing optical flow into the frame information. In addition, they introduced a fully convolutional architecture, enabled by one-dimensional temporal grids, improving the continuity of spatial features.

However, all of the above improvements are based on the positional embedding of time as the input. Time-based encoding cannot capture specific content information in the image, resulting in a low compression rate of the model. To make the embedding content-related, Chen et al. proposed a content-adaptive encoder CNeRV[37]. Subsequently, following the idea of feature embedding decoder, they proposed another decoder architecture HNeRV[38], which uses the convnext block in the encoder to encode a smaller feature map of image as an embedding. In this way, the encoded embedding is content-related, resulting in a high compression rate and good fitting effect of the model. However, due to the fewer decoding layers at the decoder end, the accuracy will decrease when fitting some fast-moving objects, moving cameras, and other dynamic videos. He et al. fully utilized the fitting role of neural networks in various types of videos and proposed D-NeRV[39], which uses massive video data to represent a large and diverse set of videos as a single neural network, employing the task-oriented flow as intermediate output to reduce spatial redundancies. It performs better in long videos and is one of the development directions of future large video models. However, the performance is poor when fitting a single short video. Zhao et al[40] fully utilized the diff flow of frames on the basis of HNeRV and proposed a differential encoder to model the spatial features of specific content in a short time, achieving good results in video interpolation and video restoration. However, the model cannot maintain small embeddings, resulting in a low overall compression rate of the video represented by neural networks, which is a disaster for multi-frame videos. Kwan et al. proposed HiNeRV[41], which pursues the ultimate bit rate performance and is currently the most competitive INRs method in video compression. However, its decoding method based on bilinear interpolation performs poorly in video interpolation and video restoration.

In MNeRV, a single video is fitted with a neural network while maintaining small embeddings. Thanks to the reasonable parameter volume and streamlined network architecture at each layer, MNeRV can reconstruct videos with better quality using fewer parameters. Unlike DNeRV and HiNeRV, which focus on a specific downstream task of video, MNeRV is consistent with HNeRV in achieving good performance on many downstream tasks such as video interpolation, video restoration, and video compression.

Inspired by HNeRV, we also use ConvNeXt blocks to construct the encoding layer and assemble them into M-Encoder to extract features as the encoder end of MNeRV. In M-Encoder, to make the features extracted by the encoder more compatible with the decoder, we changed the step size 5,4,4,2,2 in HNeRV to 5,2,2,2,2,2,2, while keeping the small embedding 16x2x4 unchanged. To enhance the competition between feature channels, we introduced the GRN layer in M-Encoder, which we proved to be necessary (Table 5).

In the NeRV decoder end, there are five decoding layers, and the kernel size k of the convolution layer in each NeRV block of each decoding layer is set to 3, and the channel attenuation factor r is set to 2, that is, the number of channels is halved for each layer. This parameter allocation makes it difficult for the later decoding layers to obtain enough parameters to fit more detailed videos. The authors of HNeRV also found this problem. In the decoder end of HNeRV, the kernel size of the convolution layer in the NeRV block of the later decoding layer was increased, and the channel attenuation factor was reduced, achieving certain results. But we proved that this is not enough. The too few decoding layers in the decoder end make it difficult to allocate parameters reasonably, and the uneven parameter allocation directly affects the fitting effect of the video. In this paper, we propose a new decoder: M-Decoder and a more efficient decoding block: MNeRV block. M-Decoder has seven decoding layers, that is, the feature map transmitted from the encoder end is enlarged seven times to obtain the fitted image. The comparison of the model parameter allocation of NeRV, HNeRV, and M-Decoder is shown in Figure 3. Thanks to more decoding layers and more reasonable MNeRV block size, it can use more subtle operations on images at different stages, so that the parameters are evenly distributed in each layer. Moreover, unlike the downsampling operation before the first decoding layer in HNeRV, there is no downsampling operation in M-Decoder, and all decoding layers perform upsampling operations, further reducing redundancy.

Regarding the loss function, compared to $L_{2}$, which has a larger penalty for outliers and makes the model unstable, we use a combination of $L_{1}$ and $L_{MS}$ to avoid such situations. The loss function is as follows:

where is the video frame fitted by the neural network, is the corresponding ground truth. T represents the total number of frames in the video. In MNeRV, we set $\alpha$ to 0.7 or 0.6. More detailed information will be explained in the experiment section.

We validated our model on UVG[43], DAVIS[44], RED[45], and Bunny[46] datasets. The experiments were run on an RTX3090 device. In the UVG dataset, we followed the processing method of HNeRV and cropped 7 videos to 640$\times$1280 resolution before inputting them into the M-Encoder. We also used the Adam optimizer to train the model, set the channel attenuation factor r to 1.2, the learning rate at 0.001, and the batch size as 2. To fit 640$\times$1280 videos, we set the stride to 5,2,2,2,2,2,2 and the kernel size to 1,5,5,3,3,3,3. In implicit neural representation of videos, the fitting speed of the neural network is the encoding speed of the video. To pursue faster encoding, we set the training rounds to 100 epochs. Unless otherwise specified, we performed the same operation on the DAVIS, RED, and bunny datasets.

We first conducted experiments on the bunny dataset, comparing it at 1.5M, and 3M sizes. The comparison results are shown in Table 1, where the loss functions of NeRV and HNeRV are L2, and the loss functions of NeRV$\dagger$, HNeRV$\dagger$, and MNeRV$\dagger$ are set $\alpha$ to 0.7. For fair comparison, we trained each of the three models three times with each loss function and took the average. We found that the loss function set $\alpha$ to 0.7 can improve the loss function on the Bunny dataset compared to the L2 loss function. Moreover, the MNeRV model performs better than NeRV and HNeRV.

We conducted comparative experiments on four scales of the Loading dataset. For the loss function part, we set $\alpha$ to 0.6. Figure 5 shows the performance of NeRV, HNeRV, and MNeRV models on different scales, and their visual comparison is shown in Figure 4. The experimental results on the Loading dataset show that our method has a significant improvement over NeRV and HNeRV. In addition, we compared our method with HNeRV on 10 REDs sub-datasets and 7 UVG sub-datasets, using L2 as the loss function, and the results are shown in Table 2 and Table 3, respectively. Note that our method achieves better results with fewer parameters. The model size is indicated in the second column of both tables. It should be noted that the ShakeNDry dataset is different from the other UVG sub-datasets, as it only contains 300 images. On the ShakeNDry dataset, the size of HNeRV is 1.52M, and the size of MNeRV is 1.49M.

We conducted detailed ablation analysis comparative experiments on NeRV, HNeRV, and MNeRV. In implicit neural networks, the main parameters for adjusting parameter allocation are kernel size and stride. Thanks to the design of its multi-layer network structure(ML), MNeRV offers a more extensive range of parameter adjustment options. We conducted ablation on kernel size and stride on the jockey dataset, and the results are shown in Table 4. We found that in the 5-layer NeRV-like block structure, due to the natural limitation of the number of layers, the parameter allocation scheme is lacking, and it is easy to have bloated parameters (as shown in Figure 3). The 7-layer NeRV-like block structure can adjust more kernel sizes and strides, and there are more parameter allocation methods, and the parameter volume of each layer is more uniform, resulting in better effects. Finally, we selected 5,2,2,2,2,2,2 and 1,5,5,3,3,3,3 as the stride and kernel size of MNeRV, respectively.

The effects of three novel components of our proposed method are studied separately: the GRN layer introduced in M-Encoder, the multi-layer network structure design and the header layer(HL) design at the beginning of the network. The ablation results are shown in Table 2, which shows the lifting effect of each component in detail.

To evaluate the performance of MNeRV on videos with significant camera motion, we conducted a comparative experiment on the REDs dataset (Figure 7) and performed an ablation study on the loss function (Table 6).

We compared MNeRV with other implicit neural representation models on various downstream tasks, including video interpolation, video restoration, and video compression.

video compression. Figure 6(a,b) shows the comparison of the reconstruction quality of HNeRV and MNeRV. In the video compression experiment, we followed the pipeline of HNeRV and conducted a comparison experiment of NeRV, HNeRV, and MNeRV on the UVG dataset using different compression ratios. The results are shown in Figure 8, where our method is better than NeRV and HNeRV in both PSNR and MSSSIM metrics. In addition, we show the best and worst results of MNeRV in Figure 9. In videos with intense camera movement (such as the readysteadygo dataset), MNeRV has shown significant improvement over HNeRV, but in videos with less camera movement (such as the honeybee dataset), MNeRV is inferior to HNeRV. On the UVG dataset, MNeRV’s video compression performance is generally better than HNeRV’s (+0.47 PSNR).

video restoration.In Figure 6(c,d), we show the comparison of the restoration effect of HNeRV and MNeRV. MNeRV also has excellent performance.

video interpolation. As a type of neural representation, MNeRV has powerful frame fitting ability. In the video interpolation experiment, we used HNeRV and MNeRV as test frames every other frame, and used interpolated embedding as input. Through learnable content embedding, MNeRV showed better results, as shown in Figure 6(e,f).

We conducted a detailed ablation study on the variants of the three components(GRN, ML, and HL) on the UVG dataset. Tables X and A present the performance of each variant on the UVG sub-datasets.

We show more visualizations for video interpolation (Figure 10), and video inpainting (Figure 11).

In Figure 12, we present the video compression results obtained on the UVG dataset(all) and each of its sub-datasets.