GAN-Based Multi-View Video Coding with Spatio-Temporal EPI Reconstruction

Multi-view video coding adds inter-view prediction to the standard of High Efficiency Video Coding (HEVC). It also introduces the concept of depth map, in which each viewpoint has an additional depth video. Therefore, we divide multi-view video coding into two categories: general multi-view video coding and depth map-based multi-view video coding.

General multi-view video coding. MV-HEVC is the sate-of-the-art standard, which inspires many improvements on its modules. Hannuksela et al. (Hannuksela et al., 2015) made a stage summary of multi-view extensions for HEVC and described the standard practices for multi-view video coding, which sets a milestone for the future work. Unlike the traditional RD model, Li et al. (Li et al., 2020) proposed a multi-view bit allocation method based on the exact target bit relationship between base view and dependent view. To reduce coding complexity, Khan et al. (Khan et al., 2021) propose an Efficient Inter-Prediction Mode Decision (EIPMD) technique utilizes the Coding Unit (CU) splitting information of the base view to find its relation with the prediction Modes. Xu et al. (Xu et al., 2021) proposed a flexible complexity optimization framework that reduces encoder complexity according to external-defined constraints by minimizing the cost of alternative partitioning using a probability-driven approach known as APC. This method dynamically adjusts local candidate partitions to meet the target local complexity constraint. Li et al. (Li et al., 2021b) proposed an RD optimization for the dependent viewpoint based on inter-viewpoint dependencies, and greatly improved its performance in MVC. In these general methods, the bitrates may increase sharply with the number of viewpoints, which is because the original video needs to be encoded in each viewpoint rather than the side information.

Depth map-based multi-view video coding. Various coding methods have been proposed for depth map sequences from different aspects such as RD optimization, enhancement, bit allocation, and virtual view synthesis for depth maps. Müller et al. (Müller et al., 2013) improved motion compensation module to encode depth map sequences, and thus proposed an extended method for depth map HEVC based on inter-view prediction. By synthesizing the intermediate view with depth map and adjacent views, this method greatly saved bitrates and thus set a major milestone in the development of MVC. To address the problem of degraded quality at boundaries of synthesized viewpoints, Rahaman et al. (Rahaman and Paul, 2017) used Gaussian Mixture Models (GMM) to separate the foreground and fills holes in synthesized views. Besides, the amount of data transmitted can be further reduced by frame interpolation. Considering the application of depth map in intermediate view construction, a feasible method to improve MVC is to obtain accurate depth maps. Yang et al. (Yang et al., 2018) recommended a cross-view multi-lateral filtering scheme, which enhances the quality of depth map using color and depth priors from adjacent views at different time-slots. To reduce the complexity in coding mode selection, Zhang et al. (Zhang et al., 2020b) proposed an efficient MVD scheme based on depth histogram projection and allowable depth distortion. Lin et al. (Lin et al., 2021) proposed to accelerate 3D-HEVC deep intra-frame coding using the characteristics of the human visual system. For the above methods, their qualities are limited by the quality of the depth map.

Deep learning have been introduced into MVC with significantly improved performances. These works include deep learning-based MVC optimization and deep learning-based MVC post-processing.

Deep learning-based MVC optimization. The deep learning-based MVC optimization approach introduces deep learning into specific modules of MVC framework. Lei et al. (Lei et al., 2022) put forward a deep reference frame generation method for MVC. This method employed a Disparity-Aware reference frame Generation Network (DAG-Net) to transform the disparity relationship between different viewpoints and generate a more reliable reference frame. Lei et al. (Lei et al., 2021) exploited spatial, temporal, and inter-view correlations and proposed a deep multi-domain prediction for 3D video coding. By employing CNNs to fuse multi-domain references, they achieved significant bitrates savings compared to 3D-HEVC. Peng et al. (Peng et al., 2023) proposed a multi-domain correlation learning module to recover the high frequency details of distorted frames by exploring multi-domain correlation. In addition, based on the block partition information generated in video coding, this method proposes a partition-constrained reconstruction module to better attenuate compression artifacts by designing partition losses. Chen et al. (Chen et al., 2022) proposed the first end-to-end optimized stereo video coding framework to reduce redundancy by iteratively using motion estimation and disparity estimation for left and right views in feature space.

Deep learning-based MVC post-processing. Deep learning-based methods are applied to the post-processing stage of the framework, which not only enhances quality of multi-view videos but also effectively removes compression artifacts. Recently, multi-frame quality enhancement approaches (Guan et al., 2019; Zhao et al., 2021; Luo et al., 2022) have been proposed. They significantly reduce quality fluctuations between compressed video frames by locating peak-quality frames and enhancing low quality frames with adjacent high quality frames. He et al. (He et al., 2020) recommended a graph-neural-network-based compression artifacts reduction method, which reduces compression artifacts by fusing adjacent viewpoint messages and suppressing misleading information. Pan et al. (Pan et al., 2021) proposed a Two-Stream Attention Network (TSAN)-based synthetic viewpoint quality enhancement method, which significantly improves the quality of synthetic viewpoints by extracting features at different scales using Multi-Scale Residual Attention Blocks (MSRAB) for enhancement. Zhang et al. (Zhang et al., 2022) modeled the elimination of temporal distortion as a perceptual video denoising problem and proposed a CNN-based synthetic viewpoint quality enhancement to reduce temporal flicker distortion while improving the perceptual quality of 3D synthetic viewpoint videos.

Inspired by these methods, we propose a method of multi-view latent code extraction and GAN-based multi-view video with spatio-temporal EPI reconstruction method to render new views by learning the least redundant latent code. The details of our proposed method are described as follows.

To facilitate the use of cross-viewpoint correlation, we employ the Epipolar Plane Image (EPI) method. On epipolar plane, an object projected into different viewpoints will appear on the same straight line of EPI. Using this approach, we group the corresponding data from different viewpoints into spatially adjacent locations in EPI, which enables us to exploit the inter-view correlation for subsequent processing. Then, the EPI is connected in temporal domain as the input of convolutional network to extract latent code.

We model the latent code extraction of EPI:

where $X$ and $Z$ represent the original EPI and its high-level semantic features, respectively. $E$ represents a feature extraction network to extract the high-level semantic features of original EPI. To further reduce the bitrates of features, $Z$ will be compressed by using a compression network $C$ before being transmitted to the decoder side:

At the decoder side, we recover the EPI information with $\hat{Z}$ and a generator $G$.

Our task is to find an $E$ and its inverse operation $G$ to minimize the reconstruction error of EPI and the bitrates of latent code transmission:

In this work, we employ CNN and GAN to perform latent code extraction & compression and EPI reconstruction, respectively. The whole networks are co-trained in an end-to-end manner. As shown in Fig. 2, the whole framework consists of spatio-temporal EPI construction, latent code extraction & compression and GAN-based EPI reconstruction, which are elaborated as follows.

Traditional EPI with pixel rows of images cannot well reflect the spatio-temporal correlations of multi-view images. In this work, we construct a spatio-temporal EPI with the following two steps. Firstly, the multi-view images are decomposed and reassembled based on their spatial locations. Let $M$, $N$ and $K$ denote the width, height and number of views of a multi-view video, respectively. As shown in Fig. 3, images of each view are divided into $8\times N$ strips with all color channels, where the value 8 is chosen empirically(Bolles et al., 1987) and thus an image consists of $m=M/8$ strips. Then, all strips at the same spatial locations are grouped to formulate a spatial EPI with a dimension of $8K\times N\times 3$. In total, there are $m$ spatial EPIs at a same time. Secondly, a spatio-temporal EPI is obtained by stacking $L$ successive spatial EPIs at time axis, in order to embed the temporal correlations. A spatio-temporal EPI is then with the dimension of $8K\times N\times 3L$. In particular, we set $L=3$ in this work.

In the following, we refer a spatio-temporal EPI as ${\rm EPI}_{j}^{t}$, where $j=1,2,…,m$ and $t$ denotes the temporal index. The spatio-temporal EPI is then fed into convolutional layers to extract latent code.

Latent code extraction. As shown in Fig. 2, the latent code extraction step transforms the spatio-temproal EPI to a latent vector $Z$, which is critical to reconstruct the intermediate viewpoint. In order to obtain more accurate latent code with low bitrates, we use latent code extraction module to remove the redundant information at the encoder end. Through iterative training of the whole network, the module extracts latent code from the spatio-temporal EPI based on a trade-off between the quality of the reconstructed EPI and the bitrates. In this work, we achieve this with Fully Convolutional Network (FCN) due to its capacity to effectively learn data distribution with a compact feature size.

For the extraction module, its input and output are the spatio-temporal EPIs and latent code of intermediate viewpoint, respectively. The extraction network includes 1 convolutional layer, 4 residual blocks and another 2 convolutional layers. Each residual block includes 2 convolutional layers, 2 Bach Normalization (BN) layers, a Rectified Linear Unit (ReLU) function and an elementwise sum. The residual blocks are connected with skip connections and elementwise sum, in order to ensemble diverse feature information that benefits the visual quality of viewpoint reconstruction. As a result, the final output of extraction is with a dimension of $8$K$/3\times N\times 3L$ due to we only need the latent code of intermediate viewpoint. As shown in Fig.4, the latent code contains structural information of intermediate viewpoint, which can be used as a priori information of the GAN network to guide the generator $G$ to reconstruct the intermediate viewpoint.

Latent code compression. For the compression module $C$, we employ an auto-encoder with three downsampling layers and three upsampling layers that are implemented with convolutions and deconvolutions, respectively. The extracted latent code $Z$ will be compressed and quantized as a compact representation $\bar{Z}^{c}$, and then reconstructed as $\hat{Z}$.

Entropy coding. The quantized latent code $\bar{Z}^{c}$ will be transformed into bit-streams by performing entropy coding. In this work, we employ the hyperprior entropy model presented in (Lu et al., 2019) for an accurate bitrates estimation. During the training process, the calculated bitrate is used as a part of the loss function.

By using a compact representation of EPI features, we provide multi-view video at a lower bitrate. At the receiver side, we reconstruct the generated EPI $\hat{X}$ from $\hat{Z}$, $\hat{X}=G(\hat{Z})$, where $G$ denotes the function of reconstruction process. To ensure that the reconstructed EPI $\hat{X}$ is as close as possible to the original EPI $X$, we refer to the GAN framework to introduce the discrimination function $D$. Through the interaction between $D$ and $G$, the EPI generated by $G$ can increasingly approximate the original EPI $X$. As shown in Fig. 2, the EPI reconstruction is composed of two modules: reconstruction and discrimination, as well as an interaction mechanism.

The reconstruction module uses a neural network with convolution as the generator. We consider the following two distributions: Joint probability density function in latent code extraction $p(Z,X)=p(X)p(Z|X)$; Joint probability density function in reconstruction $p(X,\hat{Z})=p(\hat{Z})p(X|\hat{Z})$. In these distributions, $p(X)$ is the prior probability function of the original EPI, $p(Z)$ is the probability density function of the latent code and $p(\hat{Z})$ is the probability density function of the compressed latent code. In the latent code extraction & compression process, the extraction network $E$ maps the original EPI $X$ to the latent code $Z$: $Z=E(X)$ and the compression network $C$ compresses $Z$ to $\hat{Z}$, while in the reconstruction process, the generator network $G$ maps the samples of the compressed prior $p(\hat{Z})$ to the input space $X=G(\hat{Z})$. To accurately reconstruct the EPI, it is necessary to make the conditional probability $p(X|\hat{Z})$ coincide with the prior probability $p(X)$ as much as possible.

The discrimination module determines whether the input EPI is the original EPI by a classification network. To better discriminate whether the input image is the original EPI, the training goal is to make the value of $D(X,E(X))$ as large as possible and the value of $D(G(\hat{Z}),Z)$ as small as possible. Unlike the GAN network which only discriminates the input image, our discriminator network needs to discriminate both the image and the latent code.

The interaction mechanism uses the discrimination results to guide the generator to reconstruct images closer to the original EPI, and also to navigate the discriminator to better identify differences between the generator’s output and the original EPI. To this end, we design this mechanism with an adversarial game in which the discriminator and the generator are trained alternately. The discriminator is trained to distinguish between sample pairs from the encoder $(X,\hat{Z}=C(E(X)))$ and sample pairs from the generator $(\hat{X}=G(\hat{Z}),Z)$, both of which satisfy the joint probability distribution $p(X,\hat{Z})$ or $p(Z,X)$. The generators are trained to fool the discriminators obtained from the previous training. At this point, the discriminant value $D(X,\hat{Z})$ is as large as possible.

The latent code extraction & compression and GAN-based EPI reconstruction are co-trained within a network flow. A joint objective function is thereby designed and utilized to optimize both the bitrates of latent code and the visual quality of reconstructed videos. As discussed above, the spatio-temporal EPI $X$ is extracted as latent code $Z$ by convolutional network $E$, compressed to $\hat{Z}$ by compression module $C$ and finally reconstructed as $\hat{X}$ by generator $G$. A discriminator $D$ is deployed to identify the reconstruction performance. Inspired by GAN, the objective function of whole network can be expressed as:

We calculate the distortion and bitrates losses based on distance and entropy, respectively. The distortion loss is calculated between the original and reconstructed spatio-temproal EPIs,

where the distance $d(\cdot)$ is obtained as a combination of pixel-domain Mean Squared Error (MSE) and feature-domain VGG loss,

Here $w$ and $h$ represent the dimension of EPI. $\phi(\cdot)$ denotes the operation of the VGG network to extract the feature map.

As depicted in Section 3.3, we can use the entropy model to accurately estimate the bitrates of the compressed and quantized latent code $\bar{Z}^{c}$:

With the loss function defined in Section 3.5, we are able to train the whole network consisting of the extraction module $E$, the compression module $C$, the generator $G$ and the discriminator $D$. The whole training process is summarized in the Algorithm1.

Datasets. We train our method, which is named as MVLL for the sake of simplicity, with 5 typical multi-view sequences including Scence_Door_Flowers (1024 $\times$ 768), Scence_Leaving_Laptop (1024 $\times$ 768), Scence_Outdoor (1024 $\times$ 768), Champagne_Tower (1280 $\times$ 960) and Dog (1280 $\times$ 960). Each of them is with 5 views. They are further split into EPI images, according to the steps shown in Section 3.2. We selected 12,800 EPIs from each produced EPI sequence, and a total of 64,000 EPIs were used for training. To examine the performance of our method, we test on the Common Test Conditions (CTC) of MVC (Müller and Vetro, 2014). The details are listed in Table 1. We selected 30 frames of video from each sequence for the EPI construction and put them to the test.

Baseline codecs. There are four other methods implemented for performance comparison. The state-of-the-art multi-view video coding standard, MV-HEVC (Hannuksela et al., 2015). The multi-view video plus depth standard method, MVD (Tech et al., 2015), uses the depth maps as the auxiliary information. The deep-learning based video coding method, DCVC (Li et al., 2021a), used conditional coding to replace the traditional predictive coding paradigm. The deep-learning based multi-view video enhancement method, TSAN (Pan et al., 2021), used a dual-stream attention network to improve the synthesized view quality. To make a fair comparison, for MV-HEVC, we use the HTM-16.3 software with baseCfg_3view.cfg configuration setting and set the QP to {30, 35, 40, 45}, while for MVD, we use the HTM-16.3 software and VSRS3.5 view synthesis software with baseCfg_2view+depth.cfg configuration and setting the QP pairs of $(QP_{t},QP_{d})$ for texture and depth videos, i.e. (30, 39), (35, 42), (40, 45) and (45, 48). For the DCVC method, we use the HTM16.3 software with baseCfg_2view.cfg configuration setting to encode the left and right views, and use the DCVC method with their pre-trained models to compress the intermediate view. For the TSAN method, we use their pre-trained models to perform quality enhancement on the intermediate view synthesized by the MVD method.

Metrics. The popular video coding criteria are employed to evaluate and compare all the above four methods. These criteria include the PSNR-aware measurements (rate-PSNR curves, the BDPSNR and BDBR (Bjontegaard, 2001)) and the SSIM-aware measurements (rate-SSIM curves, the ADSSIM and ADBR (Zhao et al., 2015)).

Implementation details. In our implementation, we use the HTM-16.3 software with baseCfg_2view.cfg configuration setting to encode the left and right views, and compress the intermediate view using our proposed method. We train four models with different $\lambda$ values ($\lambda$ = 512, 1024, 2048, 4096). The other parameters for training are set as follows: epochs=50, iterations=64000, batchsize=1.

Fig. 5 shows the rate-PSNR curves of all compared methods: MV-HEVC, MVD, DCVC, TSAN and MVLL. In each curve, the four data samples correspond to the four Qp settings; while in each data sample, the rate-PSNR performances are obtained as the averaged results of all views. Fig. 6 shows the rate-SSIM curves of these methods, where the SSIM index is considered is more consistent with human vision system than PSNR.

From Figs. 5-6 we can get several conclusions. Firstly, all methods achieve good performances in terms of R-Q curves, by retaining a high compression ratios at the acceptable visual quality. Generally speaking, they are still superior to the original encoder without optimizations. Secondly, the MV-HEVC methods achieve superior performance than MVD in most sequences, which may be due to the extra bitrates for depth maps in MVD. In case of multi-view plus depth coding, the MVD method is still preferred. Thirdly, the DCVC achieves acceptable performance but still inferior to our MVLL. This method was designed for general video coding and thus do not exploit the inter-view correlations between multi-view videos. Fourthly, the TSAN methods enhances the quality of synthetic view of the MVD method, thus achieves superior performance than MVD in all sequences. Fifthly, our MVLL method outperforms the compared methods in most sequences, which is contributed to its GAN-based inter-view prediction.

Tables 2 and 3 presents the quantitative comparisons of R-Q performances, where the efficiency of our MVLL is evaluated with MV-HEVC, MVD, DCVC and TSAN as the benchmarks. The terms BDBR and ADBR indicate the average bitrates savings at the same PSNR and SSIM, respectively. The terms BDPSNR and ADSSIM indicate the average quality increments at the same bitrates, respectively. From the tables, we can see the quantitative comparison results are consistent with those in Figs. 5-6. Compared with the other methods, our MVLL reduces 20.44%, 44.14%, 36.28% and 23.86% bitrates at the same PSNR, or 18.82%, 32.59%, 16.65% and 32.90% bitrates at the same SSIM. It also significantly improves the video quality in terms of PSNR and SSIM at the same bitrates. These results demonstrate the efficiency and effectiveness of our proposed method.

As for the computational complexity, we use the balloons sequence to evaluate our encoder time on the machine with a single 2080TI GPU (11GB Memory). The coding time of our proposed method is 0.204s per frame, which is faster than the traditional hand-crafted MVC standard MV-HEVC or 3D-HEVC.

The main contributions of the proposed method consist of two parts: the spatio-temporal EPI structure, and the GAN based autoencoder network structures. Therefore, to validate their effectiveness, the ablation experiments are performed.

Contribution of spatio-temporal EPI structure: To analyze the effect of the structure of EPI on the results of MVLL, we compare the algorithm performance of three different input structures: traditional EPI, spatial EPI and spatio-temporal EPI. In the traditional EPI structure, we set the width of the column occupied by each viewpoint in the EPI to 1. In the spatial EPI, the column width occupied by each viewpoint in the EPI is 8. And in the spatio-temporal EPI, we stack the time-domain information of the 3 spatial EPIs on 9 different channels in the intermediate viewpoint. With other experimental conditions being consistent, Fig. 7 represents the rate distortion results of the algorithmic based on the three EPI structures, where the horizontal axis is the bitrates of the latent code in the intermediate viewpoint. Since applying a different EPI structure requires retraining the neural network model, we use a training data set of size 8000 to reduce the computational complexity. As can be seen in Fig. 7, with the increase of the bitrates of the latent code, the image quality of the reconstructed image results from the algorithm of the traditional EPI structure grows slowly. However, the image quality growths of the results from both the algorithm of the spatial EPI and the spatio-temporal EPI structures are more obvious. Furthermore, the image quality of the spatio-temporal EPI structure is superior to the performance of the spatial EPI. Therefore, we choose to use the structure of spatio-temporal EPI in this paper.

Comparison of GAN and autoencoder network structures: To analyze the contribution of the GAN network structure, we compared the performance of both GAN and autoencoder network structures. Since the difference between these two network structures is whether they have a discriminator network or not, we implement the two different algorithms by adding or subtracting the discriminator network from the original network structure. With all other experimental conditions being equal, Fig. 8 represents the rate distortion performance results of the multi-view video coding algorithm based on latent code learning for two network structures, GAN and autoencoder network. We can see that the rate distortion performance of the algorithm with discriminator network structure is higher. Therefore, in this paper we use a GAN structure with discriminator network for multi-view video coding.