A Flexible Lossy Depth Video Coding Scheme Based on Low-rank Tensor Modelling & HEVC Intra Prediction for Free Viewpoint Video

The market of 3D videos will continue to flourish in upcoming days. The strongest demand of 3D contents currently comes from industries in the creative economy and immersive media such as Gaming, Virtual Reality (VR) or Augmented Reality (AR) platforms, Displays, Film Entertainment, Imaging Systems and Retail. More pervasive applications of 3D videos, we will find in industries as diverse as healthcare, education, military, and the real estate over time. The advancement of affordable Time-of-flight (ToF) sensing or 3D depth cameras, further, promote application of 3D videos in academia and industry. There exist a variety of 3D video content formats in the user mass market to support 3D applications [1]. This includes conventional stereo video (CSV), mixed resolution stereo (MRS), hybrid mono-stereo, video plus depth (V+D), multiview video (MVV), multiview video plus depth (MVD), and layered depth video (LDV). These formats are specifically used for rendering in 3D and VR head mounted displays [2, 3]. Moving Picture Experts Group (MPEG) standardized MVD and LDV representations and depth-image-based rendering (DIBR) for flexible display adaptation of current 3D display technology. Latest call for 3D video standardization support depth enhanced stereo (DES) as a generic backward compatible 3D video format that would support extended functionality like baseline adaptation, post production, multi-view rendering, specific to display types and sizes [4, 5]. The objective is to decouple content creation from the display system requirement. The key functionality in this context is identified as manipulation of the depth composition of a scene via view synthesis [6, 7]. Owing to the constraints in transmission and broadcasting channels, only limited texture views and their corresponding depth maps are acquired, compressed, and transmitted for 3D video applications. At decoder side, with the aid of depth information, DIBR is employed to synthesize an arbitrary number of views to support head motion parallax viewing and baseline adaptation within practical limits of different multiview display types. The perceptual quality of synthesized novel views is mainly determined by the quality of RGB images and depth maps as well as DIBR algorithm [8, 9]. Predominately, rendered view quality is sensitive to the compression distortion in depth images.

A depth map is basically stored as a gray-level image. It describes the relative distance from recording camera to actual objects in the 3D space [10]. The characteristics of depth maps are fairly different from RGB texture images. Typically, a depth map consists of smooth regions and sharp edges around the object’s boundaries [11]. The compression distortion of sharp edges causes visual artifacts such as ringing effect at object’s boundaries in synthesized views using DIBR algorithm [8, 9]. Thus, preserving sharp edges is a critical task for depth video coding and high-quality view synthesis. The conventional 2D block matching based video coding algorithms are suboptimal for depth image sequences [11, 12, 13]. Such approaches generally divide large homogeneous regions in depth maps into small blocks. The sharp discontinuities around object boundaries can be placed within the same block. Block matching with arbitrary motion prediction is fairly demanding. This leads to significant coding artifacts around the depth edges in decoded depth images at high compression ratios [14, 15, 16, 17].

Joint Collaborative Team on Video Coding (JCT-VC) introduced a new generation of coding standards for 3D video such as High Efficiency Video Coding (HEVC) [18], 3D-HEVC [19], MV-HEVC [20], etc. New tools have been introduced such as depth modeling mode (DMM) [21], segment-based depth coding (SDC) [22], and view synthesis optimization (VSO) [21, 23, 24] to explicitly handle depth video coding with consideration of preserving sharp edges. DMM intra mode of depth maps in 3D-HEVC offers flexibility in representing the sharp edges by following a non-rectangle partition approach based on wedgelet [25] or contour [26] based segmentation. It can save about 5% of the transmitted bitrate maintaining acceptable view quality. The SDC approach chooses an alternative coding path and compresses the residual signal with constant pixel value, instead of following conventional transformed and quantization coefficients. The VSO mode selects different coding modes and the unit partition to trade off rate-distortion (RD) optimization and synthesized view distortion. In VSO evaluation, all combinations of block sizes need to be checked, considering DMM intra mode with and without SDC. This improves the coding efficiency, but at the expense of high computational complexity. This issue cannot be solved even adopting complex HEVC quadtree coding structure in 3D-HEVC [27].

Several other research efforts are made to remove temporal or intra/inter-view redundancies in depth video images considering the specific properties of depth images [28, 29]. The basis of such schemes is motion estimation considering 2D block matching algorithms. Most of the algorithms exploit correlation between the RGB color and depth frames. Grewatsch and Miiller [28] adopts a conventional block matching approach and determine motion vectors (MVs) using the RGB video. The estimated MVs from texture information are then considered for encoding both the texture and depth sequences. Contrary, Daribo et al. [29] considered the motion of a block at the same coordinates in both texture and depth videos and adopted a joint distortion criterion to estimate common MVs for both texture and depth frames. Such approaches remove temporal or inter-view redundancies if accurate estimation of motion can be determined. Thus, these algorithms work well for coding the depth images acquired from cameras that remain in a fixed position. The case becomes inevitably much more complicated when moving cameras are involved, such as handheld RGB-D capturing. Unlike static camera acquisition, the same objects depth values change in successive depth frames as the distance between a mobile camera and the objects in a scene change across the time. Therefore, motion compensation based depth coding schemes output very inaccurate results with mobile RGB-D cameras [16, 17, 30, 14].

The above mentioned depth video codecs/methods usually can only work for a specific bitrate. This restriction of one system per bps (bits per second) limits the generality and flexibility of existing codecs/methods for practical 3D video rendering applications. This paper proposed a novel coding scheme which flexibly performs depth video compression at multiple bitrates, accomplishing varying quality levels. The idea of lossy compression is inspired from low-rank modelling by tensor approximation that represents the high-dimensional depth data more compactly. A simple yet effective CANDECOMP/PARAFAC (CP) tensor decomposition based scheme is developed, where there is a tensor modelling block to approximate a low-rank representation of depth video data into a set of factor matrices. By differing the rank of factor matrices in tensor decomposition and quantization parameters of the HEVC intra coding block, one can flexibly adjust the bitrate within a single compact system. The proposed scheme applied to coding depth maps used for view synthesis in a multi-view video coding system. It demonstrates significant bitrate savings on average, maintaining state-of-the-art performance in terms of both PSNR and SSIM indices of rendered views. Withstanding their demonstrated success, the proposed scheme benefits in three major ways:

• •

  Different from systems which generally employ a series of cascaded modules to compress the depth data, our proposed scheme does not introduce cumulative errors because there are few interactions between low-rank modelling and HEVC encoding modules.

• •

  The mathematical multi-linear tensor decomposition model for processing depth data is compact and the procedure is not handcrafted, which can efficiently represent the various complex structures in depth maps.

• •

  The performance of proposed tensor approximation based scheme is effective for compression with a low bitrate and does not get much affected with visual artifacts in rendering views from decoded depth maps (e.g., blocky artifacts, blurs, and ringings).

Several schemes have been introduced that give natural extension to existing video codec for depth compression [31, 32, 33, 34, 35, 36]. Belyaev et al. [31] proposed method to compress 16 bit depth infrared images via 8 bit depth codecs. They utilize JPEG and H.264/AVC codecs with 8 bits per pixel input format. They compare rate-distortion performance with JPEG 2000, JPEG-XT and H.265/HEVC codecs. Their approach with two 8 bit H.264/AVC codecs achieve similar results as 16 bit HEVC codec. In [32, 33], Hamout and Elyousfi improved the 3D-HEVC depth map intra prediction model for practical 3D applications. Their approach classified depth video regions as a homogenous or non-homogenous region. Further, they allow conventional intra coding and depth modelling modes step to be skipped. This intuitive approach applies automatic merging possibilistic clustering for region classification based on tensor feature extraction and data analysis. This leads to fast depth map intra coding. Fu et al. [37] introduced a Depth Intra Skip (DIS) mode that allows early determination for intra mode decision in depth map coding. Pece et al. [34] developed codec for depth streaming by converting single channel depth images of Microsoft Kinect to standard 8 bit three channel images and using existing codecs such as VP8 or H.264. Besides, Pajak et al. [35] extends H.264 codec for depth compression by capturing a relation between depth perception and contrast in luminance and improve decompressed visual quality of depth for rendering content. Contrary to above mentioned approaches, Liu et al. [36] research suggests that coding scheme based on hybrid lossless-lossy methods provide a better tradeoff between quality and compression ratio for the real-time compression of multiple depth streams. They suggested using x264 for lossy run length encoding to keep the highest bits of 12 bit depth images.

An overview of proposed lossy depth video compression scheme is illustrated in Fig. 1. The proposed system contains mainly three modules: 1) an encoder module, which consists of two subblocks: first, a tensor decomposition block, which approximates varying low-rank representation of depth video data, and the second HEVC intra encoding subblock, which encodes approximated depth data with varying quantization parameters, 2) a corresponding decoder module, which performs reverse decoding operations, i.e., dequantization and tensor reconstruction, and 3) 3D rendering module, where decoded depth maps are used for novel view synthesis.

In the first module, the system performs factorization of scene geometry into a set of factor matrices following CP decomposition via alternating least squares algorithm. The representation of the depth data into a high-dimensional sparse tensor domain provides an economic solution to 3D video storage and transmission. Further, performing lossy quantization in the second module with HEVC intra modes on the tensor components of low-rank approximated scene geometry effectively encodes depth data at multiple bitrates. Thus, the scheme overcomes decisive bottlenecks of memory and network bandwidth when handling high-resolution depth frames. The proposed scheme considers a range of rank values for factorizing tensor. Further, encoding with HEVC adopts a variable quantizer to allocate different quantization parameters to the factor matrices. As an indispensable step in multi-view 3D video system, this is a critical step in preserving the major texture and edges in spatially variant depth frames and mitigate rendering artifacts or sampling issues in synthesized views from decoded data. The flexibility proposed scheme offers for changing the rank of factor matrices in tensor decomposition and its quantization levels, facilitates to easily adjust the bitrate and quality levels of reconstructed depth maps within a compact system. Thus, a single system could enable display adaptation with different 3D devices at reduce storage space, while maintaining backward compatibility and extended functionality such as N-view synthesis without sacrificing much the rendered view quality for baseline adaptation under multiple bitrates.

Different components of our proposed depth video compression pipeline are described in the following sections.

A series of experiments are conducted to assess the performance of the proposed depth compression scheme. We tested our model on “Ballet” and “Breakdancing” sequences provided by Microsoft Research. The multi-camera data is acquired by eight cameras located along a 1D arc covering about $30^{\circ}$ from one end to the other. The color and depth videos are provided with resolutions of $1024\times 768$ [70].

We encode depth video of each camera sequence using the proposed coding model and High Efficiency Video Coding (HEVC) reference software HM-16.0 [71]. The color and depth video of left and right cameras are encoded considering various quantization parameters (QPs). The QP values of $2$, $6$, $10$, $14$, $20$, $26$, $38$ are used to encode the depth videos.

The experimental model to test the performance of proposed scheme analysed the quality of synthesized intermediate view. We selected reference cameras $3^{rd}$ and $5^{th}$ among eight cameras of “Ballet” and “Breakdancing” multi-camera data. The intermediate views are synthesized at virtual camera $4^{th}$ using free-viewpoint rendering algorithm developed by Sharma et al. [9]. The objective quality assessment is performed by measuring Peak Signal-to-Noise ratio (PSNR) and structural similarity index measure (SSIM). The PSNR is measured by comparing the synthesized video rendered using original depth maps as the reference and the synthesized video using the decoded depth maps. It is critical to note that decoded video is used for virtual view synthesis to measure the distortion caused by depth map compression. The rate distortion (RD) curves are plotted considering the total bitrate required to encode depth video of both reference cameras and PSNR/SSIM scores of the synthesized virtual video.

Fig. 2 shows the “Bitrate vs PSNR” and “Bitrate vs SSIM” comparison graphs between the proposed method and HEVC codec. The x-axis plots total bitrate required to encode left (camera 3) and right camera (camera 5) depth sequence. The y-axis plots PSNR/SSIM measures of the synthesized intermediate camera view (camera 4). We achieve significant bitrate reduction using proposed coding scheme compared to directly encode the depth video of left and right camera using HEVC codec, while maintaining appropriate quality of synthesized virtual camera views in terms of PSNR and SSIM measures. The encoder results obtained by applying proposed scheme and HEVC in compressing “Ballet” and “Breakdancing” depth sequences are tabulated in Table I-IV. Total number of bytes required to encode left (camera 3) and right (camera 5) depth sequences are summarized in Table VI-VIII. The Bjontegaard metric calculation results demonstrate the coding efficiency of proposed scheme compared to HEVC codec [72]. The Bjontegaard delta bitrate (BDBR) reduction results considering different ranks and quantization values are given in Table V. The BDBR measures are computed by considering the total bitrate needed to encode left and right camera depth video and PSNR scores of the synthesized intermediate camera video. Note that minus sign indicates bitrate reduction. Compared to HEVC, our scheme achieves 62.8143%, 35.7449%, 43.1331%, 21.5080%, 30.7221% rate gains on average with “Ballet” data, considering tensor ranks $1$, $5$, $10$, $15$, $20$ respectively. On “Breakdancing” data, we obtain 70.7924%, 73.3179%, 49.1670%, 26.6356%, 16.2384% rate gains, considering tensor ranks $1$, $5$, $10$, $15$, $20$ respectively. This is a substantial improvement over HEVC for directly encoding depth videos with the same quantization parameters and configuration. It can also be observed in Fig 2 that for higher ranks, particularly large PSNR gains are obtained using the proposed model even for low bitrates, comparable to HEVC. The SSIM scores demonstrate that the perceived quality of synthesized views is also appropriate for different bitrates, obtained considering varying tensor decomposition ranks and quantization parameters in our scheme. This explains significant redundancies can be removed, which lowers the overhead in transmission for bitrate coding without affecting much the perceived change in structural information of rendered views quality. Another critical advantage of our scheme is to allow scalable depth video coding. Since the proposed scheme can render virtual views of varying quality at the decoder considering different ranks and QP values. This flexibility could be preferable in 3D video transmission and broadcasting scenarios as it provides additional levels of scalability.

In this paper, we presented a simple yet effective depth video codec as an extension of the HEVC standard for stereoscopic and autostereoscopic multi-view displays. The proposed mathematical scheme is developed on tensor modelling of higher-order information in visual depth data. Our system decomposes a tensor representation of scene geometry into a set of factor matrices following CP decomposition via alternating least squares for lossy depth compression. By varying the rank of factor matrices and its quantization levels, we could smoothly adapt the bitrate in coding the depth content. Thus, appropriately achieved the goal of using a single system to cover a range of bit rates and quality levels. The scheme offers scalable and flexible encoder for coding of depth content, its delivery and display on a variety of 3D device in terms of supported spatial and temporal resolution and decoder complexity.

Our depth coding scheme is applicable to be used together with other hybrid coding architectures such as, e.g., MPEG-4 Visual, H.264/MPEG-4 AVC, MV-HEVC, 3D-HEVC, H.263, and H.262/MPEG-2, etc. Low-rank decomposition by pairwise perturbation retain the multi-dimensional dependencies in activation tensors of depth data and results in significant bitrate savings. This encoder control of the depth data and view synthesis quality, guarantees display adaptation via intermediate view generation at the decoder. Using a subset of the coded depth components, 2D video, stereo video or full MVD can be reconstructed from the 3D decoded depth bitstreams and reference texture data. The objective results shown in Table V show significant bitrate reduction (BDBR) in comparison to HEVC, while retaining the almost similar synthesis quality. The PSNR and SSIM measurements demonstrate satisfactory performance and not much obtrusion due to geometrical distortions in the synthesis coming from compression along depth discontinuities. In the future, we analyse view quality considering different factors such as camera baseline variation, scene characteristics, artifacts caused by inherent visibility, disocclusion, and resampling problems in DIBR [9, 73, 74, 75]. This enables a more precise estimate of synthesize view quality which helps to gain an optimal RD performance, choosing different parameters and mode selection in low-rank tensor based coding algorithms for 3D display applications.

Other avenues for future research include exploration of different tensor models in revealing latent correlations residing in high dimensional spaces of 3D video data and remove redundancies using optimized linear/multilinear algebra. A generic higher-order tensor representation in the context of an efficient HEVC extension for MVD coding is critical. While matrices based coding methods are limited to a single mode of variation, however, to naturally accommodate different modes of variation, tensors could be used in MVD video compression. We would develop an adaptive control mechanism that would allow for efficient representation of the MVD content description along each mode by a factor matrix and analyze different grounds of the decomposition with modes interaction by a core coefficient tensor for model reduction applications. We believe preliminary coding results with CP decomposition reported in this paper could motivate further investigation of tensor-based methods as a crucial mathematical object for representation and standardization of improved coding tools for candidate extensions of HEVC and its variant for 3D display applications. This could particularly benefit for further space savings for low bit rates, higher-resolution 3D content and low-delay application encodings. We would also explore tensor regression approaches for achieving higher coding gains.