RCLC: ROI-based joint conventional and learning video compression

As shown in Figure 1, for the incoming frame, we decide whether it is background-updating (BU) or ROI-updating (RU) frame based on our Group-Of-Frame (GOF) definition (see Section 3.2). For BU, the whole frame is compressed by a codec that can satisfy the realtime-demand in collating to the hardware. Because BU’s purpose is to provide the background information, which is not much important, the codec is configured with a high quantization parameter to get very fast compression speed and low-bitrate.

For RU, we have to recalculate the compressed area since we reused the previous background (see Section 3.2 and 4.1), only the new calculated ROI area is compressed with a low quantization parameter, although ROI area is usually much smaller than the whole frame, we still need to carefully choose the parameter in considering about compression speed. On the decoder side, we receive the information of the whole frame for BU and the ROI area for RU along with the position information. While the ROI-area of BU is enhanced, based on the received position information, ROI in RU is blended into previous reconstructed BU or previous frame and an edge smoothing network will filter-out the blended edges (see Section 4.2).

Since background information is not so important in an online meeting, we need to limit the number of bits used for background transfers. Therefore, in our framework, Group-Of-Frame is defined as the period that background is updated, which means the frame at the start of GOF is chosen as BU (see Figure 2). This definition is conducted based on our consideration of normal online meeting video, where the camera angle is rarely changed, so the background remains similar in a close range of visual similarity.

In each GOF, BU provides the background information for all the following RU. Whereas, the compressed ROI of RU is calculated not only based on the RU-frame but also based on the ROI position in BU or the previous frame (see Section 4.1). This manner is to avoid missing information when the ROI moving over frames.

To avoid specific hardware demand, we use the YOLO_v4 method [6] to perform the ROI detection task. YOLO_v4 can do the detection by input a normal RGB camera frame. To meet the real-time with limited hardware resources, the ‘tiny’ version [13], which has a smaller weight-load, is used. Here, we define the ROI as one person. For BU, the detected area is formed to two points of bounding boxes area, this information is used at the decoder to enhanced the ROI-area in BU. Furthermore, in the same GOF, RUs also use this information to recalculate their ROIs.

For RU, after performing normal detection as BU, since the object or ROI can move over time, a direct blending of ROI will lead to the missing pixels information. Therefore, by calculating the difference with the BU ROI bounding box in the same GOF, we can include the missing pixels into the new ROI (BU_blending in Figure 3.a). Furthermore, for large GOF size, where the number of missing pixels between the BU and RU increases due to the large movement of ROI, we introduce another updating rule. Instead of using BU as the background provider, we calculate the compressed area of current RU based on the previous frame, therefore, the distance of movement and the number of missing pixels are reduced, especially on very large GOF size (RU_blending in Figure 3.b).

Since there is a gap in the quantization parameter between BU and RU frames, the smoothness of visual quality over the sequences may be affected in the case of the limited bandwidth, at which we have to increase that gap. Hence, we have to perform an enhancement step for BU to catch up with the RU visual quality. However, since the background is not useful information, we only enhance the ROI area of BU on the decoder-end based on the received bounding box information.

While BU facing the quality gap with RU, the ROI inside of RU also facing that gap with the blended background. The blending approach usually leads to noticeable edges at boundary pixels. Therefore, we have to smooth those edges by inputting the decoded-ROI of RU plus the neighboring pixels of the blended background to an edge smoother.

We try to avoid using two separated model-weight sets for enhancing and edge smoothing because of the hardware on normal user PC consideration. Therefore, we use the deep recursive residual network [12] as our enhancement model. This model is suitable for small GPU consumption with its recursive approach while easy to perform fine-tuning and avoid the overfitting problem. After training this network on the edge smoothing task, we fine-tune the network by squeezing the recursive residual block and input layer while training on the quality enhancement task (see Figure 4).

Based on our framework, the computation time of the system will be calculated as follows:

• •

  For BU: Encoder time = max(Detection, Compression); Decoder time = Decompression + ROI Enhancement.

• •

  For RU: Encoder time = max(Detection) + Re-ROI calculation + Compression; Decoder time = Decompression + ROI Blending + Edge smoothing.

By setting the H.265 codec with ultra-fast preset and all_intra configuration, our system can get framerate higher or equal to 60fps for all components on 1080p video.

Our experiments were conducted on an NVIDIA RTX 2080Ti GPU while an Intel Core i7-8700K CPU was used to perform non-GPU tasks. For training our enhancement/edge smoothing network, we collected 20 videos with CIF and 720p resolution from [3], all frames were extracted and used. We used compressed frames as input for the enhancement task. For the smoothing task, the ground-truth ROI was blended to the compressed background to form a combination input. All frames were ROI-centered cropped with the size of 512x512 for a fixed training size. We implemented our proposal using the PyTorch[1] framework. We used Adam[10] optimizer and started the edge smoothing task learning with a learning rate of 1e-04, then terminated the training after 20 epochs. Next, we perform the enhancement task training with layers freezing (see Section 4.2) and learning rate of 1e-05 then terminated after 10 epochs.

For testing, since there is no available online meeting related non-compressed dataset with 1080p resolution, we conducted two new test sets. The first set is for the online meeting scheme, where the camera angle is fixed (fixed camera set/tech-reviewer set) and the second one has the viewpoint that changes over time (moving camera set). We collected 24 videos with 4K resolution from YouTube[4] and Videvo[2], then down-scaled them to 1080p using a bicubic operator to remove the compression artifacts. Especially, for the fixed camera set, because there is almost no available 4K online meeting video, we collected the sequences from tech-reviewers, which also satisfies the scheme requirement. Figure 5 shows a brief view of our proposed test sets. In our experiments, we use PSNR and Bjontegaard-delta (BD)-rate [5] metrics to evaluate our results.

We compare our RCLC with H.265 codec[11] over two proposed test sets. Note that, we also use H.265 as our anchor in this comparison. For H.265, QPs equal to 32, 37, 42, 47 which ensure the 60fps compression. For our RCLC, QPs for ROI are 22, 27, 32, 37, while QPs for background are 32, 37 (for only ROI QPs = 22, 27), 42, 47. Except for the Blending approach selection comparison, we use the BU_blending approach (see Section 4.1) for all comparisons because it can cover the smallest GOF size, which is suitable for moving camera cases.

Bitrate reduction compare to H.265. As shown in Table 1, we first compare our RCLC using our basic GOF = 2 with H.265 codec on the ROI are of two proposed test sets. We can see that our RCLC got better results for ROI quality than H.265. In particular, our RCLC can gain up to 35.26% BD-rate reduction over H.265 on the moving camera set and 32.55% on the other. In the best case, our RCLC can reduce up to 70.89% RD-rate and 2.6% for the worst case of Doctor_22704 sequence, where the ROI area is bigger or equal to two-third of frame size. It is worth mentioning that our RCLC can achieve this result while keeping the compressing speed intact from H.265 by leveraging the unused GPU hardware.

GOF selection. Figure 6 shows several RD curves of our RCLC over different GOF selections on the fixed camera set. We can see that when the camera angle is fixed, by increasing the GOF, RCLC can get better performance. And in an ideal situation, where the camera is fixed and the person ratio does not change much during the meeting, we can set the GOF equal to full sequence (one_BU), which means we only need to send the background only one time.

Visual results. We compare several compressed frames with H.265 in Figure 7 . When looking into the cropped ROI part, we can see a lot of noise and block artifacts from H.265 compression. While with smaller transferring bits, our RCLC can get higher-quality visual in texture and smoothness for the ROI with all GOF. In our RCLC framework, BU may have worst quality than RU, however, by increasing the GOF’s size, the RU ratio also increases which leads to the higher average PSNR value for the ROI area along with the sequence’s frames. Hence, in the ideal case, where only the first frame is BU, all remained frames will have much better visual quality compared to H.265.

Blending approach selection. We further exam two blending approaches in Section 4.1. The results of BD-rate reduction over GOFs = 4, 8, and one_BU on fixed camera set are tabulated in Table 2. Here, b_ and r_ denote BU-blending and RU-blending approaches, respectively. We can see that the RU-blending approach gets better performance than the BU-blending approach for most cases. By increasing the GOF size, the performance distance increases from 0.56% at GOF4 to 0.91% at GOF8 and 2.61% at the ideal one_BU case. The performance gain comes mainly from the smaller compressed area because of the minimum motion between two consecutive frames. This result demonstrates that the RU-blending approach is much suitable for stable sequences which have fixed camera angle over time.