Anti-aliasing Predictive Coding Network
for Future Video Frame Prediction

Future video frame prediction has been rapidly developed recently [50; 5; 4; 12; 52; 55]. This looking-ahead ability could be applied to various fields including autonomous driving [30; 49], robotic systems [8; 11], rainfall forecasting [36; 35] etc. In particular, this self-supervised learning method for video representation can also be migrated to downstream tasks [15; 43], to solve the difficulties of label acquisition in supervised learning tasks.

Despite its enormous value in terms of application, predicting an accurate and clear video frame remains a challenging task due to the uncertainty and curse of dimensionality involved. Existing works [42; 31; 46; 24; 5] still suffer from lack of appearance details, low prediction accuracy, and high computational overhead. We observe that most of the works usually employ a similar unidirectional end-to-end architecture [42; 31; 46; 5; 4; 12; 55]. For instance, VPTR [55] first encodes historical frames into high-level representations, then uses transformer network for prediction, and finally decodes and reconstructs the pixels. It only predicts high-level representations in semantic space, potentially leading to a loss of prediction details, which can manifest as predicted images being blurred or inconsistent with actual motion trajectories finally. The convolutional LSTMs (ConvLSTM) solve the problems well [35; 46], it directly calculates video frames and performs predictions at each level. However, the memory state is still only updated within each Convolutional LSTM level, relying less on the hierarchical visual features of the other network levels.

The predictive coding framework [32; 14; 1; 40] provides a novel and effective way to solve the above problems. It is updated through a combination of bottom-up and top-down information flows to enhance the interaction between different network levels, and uses prediction errors to achieve effective feedback connections. One of the most typical predictive coding model is the PredNet proposed by Lotter et al. [28], which strictly follows the computational manner of traditional predictive coding framework and achieves state-of-the-art results at that time. However, the PPNet [26] proposes different ideas according to recent cognitive science and neuroscience hypotheses, e.g., the information being propagated up the hierarchy isn’t just limited to prediction errors, but also includes signals such as sensory input [40]. The model’s unique temporal pyramid architecture enables longer-term predictions at higher levels with lower update frequencies, broadening the temporal receptive field of higher-level neurons and reducing computational overhead. Our model is inspired by this idea, but differs in several ways.

We first observed the inconsistencies in its temporal signal processing in PPNet. Specifically, the sensory input propagated to higher level is lagged, which exacerbates as the network level increases. Furthermore, the sensory input and prediction error are simply concatenated for calculation, which forces the predictive unit at higher-level to predict the lower level prediction error simultaneously. It is both contradictory and challenging. We will expose and analyze these issues in detail and redesign the computational specification in Section 2.

Additionally, how to construct clear and natural video frames is not well considered by most works including PPNet [26; 38; 12; 39; 37]. For instance, the Conv-TT-LSTM [38] reports pretty good pixel-wise accuracy (usually evaluated using Structural Similarity SSIM [48] and Peak Signal-to-Noise Ratio PSNR [18]), while its visual performance seems unsatisfactory. Besides, the SimVP [12] and TAU [39] achieve higher SSIM and PSNR scores, however, they do not report the evaluation on perceptual metric LPIPS [57] nor show adequate visualization results. We identify two sources that contribute to the problem: 1) using a deterministic loss such as mean squared error (MSE) forces the model to prioritize reducing the overall error, which usually resulting in generating blurred images (especially in complex and unpredictable scenes); 2) current downsampling or upsampling artifacts are not aggressive in suppressing aliasing. Several works [22; 2; 19; 25] propose to use adversarial training to sharpen the generated images. However, adversarial training may cause the generated frames to deviate from the actual trajectory, which needs to be carefully controlled.

We find that directly using deep features to compute the loss without adversarial training is also effective for image sharpening. We expose that the insufficient representation ability of deterministic loss or metric cause the first problem described above, which only measure differences at pixel-level. The LPIPS [57] also points out that the scores of MSE, SSIM and PSNR may not align with the human visual system. Therefore, it proposes to use neural networks with stronger representation capabilities to measure the perceptual similarity between images. In this work, we use the pre-trained model provided by LPIPS to compute the perceptual loss, and the results demonstrate the superiority of this approach. Compared with adversarial training, its training is more stable and energy efficient, while avoiding the predicted images from deviating from the actual trajectory.

To further address the aliasing cause by non-ideal downsampling and upsampling filters, we propose to employ low-pass filters with learnable cutoff frequencies. According to the Shannon-Nyquist signal processing framework [34], in order to restore the signal without distortion, the sampling frequency should be greater than twice the highest frequency in the signal spectrum. Typically, a low-pass filter is used to remove high-frequency signals above half the sampling frequency, which can be ignored. However, selecting an appropriate cutoff frequency can be challenging. The proportion of high-frequency signals that need to be suppressed may vary depending on the scenarios, input size or even the feature maps. To address this issue, we propose define the cutoff frequencies as learnable parameters, to enable the model adaptively choose which signals need to be attenuated. Additionally, the upsampling artifact has also been redesigned. It does not modify the continuous representation, its sole purpose is to increase the output sampling rate. Once aliasing is adequately suppressed, the model will be forced to implement more natural hierarchical refinement, thus generating more realistic and natural-looking images.

Finally, input inconsistency between training and testing in temporal sequential tasks remains a challenging topic. The predicted frames usually serve as new inputs to enable continuous prediction. In this work, we propose to calculate the “Encoding Loss" during training to alleviate the impact of prediction error accumulation during long-term predictions, which measures the Euclidean distance between the higher-level representation of the predicted frame and the ground truth. This forces network modules to extract features from “imperfect" predicted frames that are as similar as possible to the ground truth. Ideally, the features of the predicted frame encoded by the encoding module are exactly the same as those of the ground truth, then the input inconsistency is finally resolved. Our project is available at https://github.com/ANNMS-Projects/PNVP

Figure 14 shows the overall framework of our model. In the video frame prediction task, $f_{l}^{t}$ denotes the representation of frame $x_{t}$ at level $l$. The predictive unit, composed of “ModError", “ConvLSTM" and “ModPred" modules, combines $f_{l}^{t}$ with other signals such as prediction from higher level and prediction error from lower level, to calculate the local prediction $P_{l}^{t}$. The $P_{l}^{t}$ is then propagated downward to lower level for calculating prediction $P_{l-1}^{t}$ of lower-level representation. Moreover, it is also utilized to compare with the sensory input $f_{l}^{t+1}$ of next time step to calculate the prediction error $E_{l}^{t+1}$, which reflects the current prediction performance and is used to correct the calculation of the network at the next time step. The design of the overall architecture is inspired by the temporal pyramid framework of PPNet [26], but it differs in several ways.

Firstly, we find that the sensory input propagated to higher level is lagged in PPNet, which exacerbates as the network level increases. As illustrated in Figure 2 (a), the sensory input $f_{l}^{t+1}$ is calculated from the combination of $f_{l-1}^{t}$ and prediction error $E_{l-1}^{t+1}$, while $P_{l}^{t}$ is considered to be a prediction of $f_{l}^{t+1}$. In other words, the $P_{l}^{t}$ is considered to predict $f_{l-1}^{t}$ and $E_{l-1}^{t+1}$. However, the $P_{l}^{t}$ is eventually propagated downward and combined with $f_{l-1}^{t}$ to make the lower-level prediction, so what $P_{t}^{l}$ predicts is lag and meaningless. Particularly, this delay exacerbates as the network level increases. For instance, the sensory information contained in $f_{l+1}^{t+1}$ (the third level in Fig 2 (a)) is actually derived from $f_{l-1}^{t-1}$. Therefore, to address this problem, we propose propagating up the sensory input $f_{l-1}^{t+1}$ (Fig 2 (b) 1.) of next time step instead of $f_{l-1}^{t}$, which matches in the time dimension.

Secondly, in PPNet, the higher level sensory input $f_{l}^{t+1}$ represents both lower-level signals $f_{l-1}^{t}$ and $E_{l-1}^{t+1}$, which means that the $P_{l}^{t}$ needs to predict the relevant information of prediction error $E_{l-1}^{t+1}$ at the same time. Paradoxically, the prediction error $E_{l-1}^{t+1}$ is calculated from $P_{l-1}^{t}$ and $f_{l-1}^{t+1}$, while the generation of $P_{l-1}^{t}$ is inseparable from the higher-level prediction $P_{l}^{t}$. In short, $P_{l}^{t}$ predicts the $E_{l-1}^{t+1}$, while the $E_{l-1}^{t+1}$ must be generated by $P_{l}^{t}$ first, which is contradictory. To address the problem, we propose to calculate the prediction error and sensory input separately. As shown in Fig 2 (b), the higher-level sensory input $f_{l}^{t+1}$ only represents $f_{l-1}^{t+1}$ at lower-level, while the prediction error is excluded.

The training strategy plays an important role in generating high-quality predicted frames. Specifically, we calculate three kinds of losses including prediction loss $\mathcal{L}_{1}$, encoding loss $\mathcal{L}_{2}$ and LPIPS loss $\mathcal{L}_{lpips}$ during training. Firstly, the prediction loss $\mathcal{L}_{1}$ is indispensable which measures the squared Euclidean distance between local prediction $P_{l}^{t-1}$ and next sensory input $f_{l}^{t}$. Its calculation is expressed as Eq.7, where $L$ denotes the number of network level, $\lambda_{t}$ and $\lambda_{l}$ represent the weighting coefficient at time-step $t$ and level $l$ respectively. $T=T_{1}+T_{2}$ denotes the total length of input sequence, where $T_{1}$ is the length of ground truth sequence and $T_{2}$ is the length of predicted sequence.

The encoding loss $\mathcal{L}_{2}$ is calculated to alleviate the inconsistency between ground truth frames and predicted frames. Long-term prediction is a important task for future video frame prediction. Predicted frames instead of ground truth frames are used to enable continuous prediction, however, the predicted frames are “imperfect”, which will cause the predictions to becoming increasingly blurry as prediction errors accumulate. Therefore, we propose to calculate the encoding loss $\mathcal{L}_{2}$ to force the network modules to extract features from “imperfect" predicted frames that are as similar as possible to the ground truth frames. The specific calculation is shown in Eq.7, where $f_{l}^{t}$ and $\hat{f}_{l}^{t}$ indicate the representation of the ground truth frame and predicted frame at time-step $t$ and level $l$, respectively. They are obtained by calculating with the same encoding module.

The LPIPS loss $\mathcal{L}_{lpips}$ plays an important role in image sharpening, which is described as Eq.8. The $f_{lpips}$ denotes the LPIPS pretrained model (VGG as backbone), $P^{t-1}$ denotes the predicted frame generated at time-step $t-1$ and $f_{t}$ is the target frame. Deterministic losses such as mean square error or Euclidean distance usually calculate differences between images at pixel-wise level, which may cause the model to generating specious and ambiguous results (according to Figure 8, the model generates increasingly blurred images when LPIPS loss is not utilized, however, it still maintains high pixel-wise level accuracy (SSIM and PSNR)). Therefore, we suggest to simultaneously employ deep features to measure the differences between images, which is achieved by a neural network, to generate more believable results. Finally, the total loss $\mathcal{L}_{total}$ is defined as the sum of the above losses (Eq.8). The $\lambda$ is a coefficient to increase the proportion of LPIPS loss.

In this work, we propose to define learnable cutoff frequency for low-pass filter to suppress aliasing in neural networks. Unfortunately, the detailed reasons why the network chooses such a ratio coefficient have not been explored well. We suspect that this is mainly depended on the size of the input images. For example, the Caltech (size of $128\times 160$) dataset has similar scenarios to KITTI (size of $128\times 256$), but the ratio coefficients learned are quite different (Table 1, Level IV). This may limit the model to achieve good performance on tasks with different input sizes. Moreover, anti-aliasing is not suitable for some special scenarios such as datasets with only black and white pixels. Low-pass filtering can severely blur the boundary and produce unwanted pixels. Therefore, it is better to remove the low-pass filter while performing on this kind of datasets.

In the future, we will employ filters more carefully and explore additional applicable scenarios. For instance, a learnable band-pass filter may be utilized to extract signals in specific frequency bands of interest, mitigating the interference from extraneous information during calculation. Additionally, using LPIPS to calculate the loss for training plays an important role in this work, however, its representation ability could still be enhanced since the authors only employ simple backbones such as VGG and AlexNet. It might be interesting to train a more powerful neural network which would enable the generator network to generate accurate and clear future frames without using any deterministic losses. Finally, the present network architecture lacks control over the update frequency of neurons at different levels. We will further improve the architecture, with the goal of making it possible to adjust the update frequency of higher-level neurons according various conditions such as the scene to be predicted, the variations between frames and the prediction error.