Learning Hierarchical Color Guidance for Depth Map Super-Resolution

Non color-guided DSR directly reconstructs a HR depth map from a LR depth map without any external guidance information. Earlier works proposed some local filtering-based methods, which mostly use high-pass filters to recover the boundary information of the depth map. For example, Yang et al.[29] proposed a post-processing method to improve the spatial resolution and accuracy of depth images by iterative bilateral filtering. The filtering-based method has lower operational complexity, but its ability to recover depth details is unsatisfactory, and the over-smooth and blurred boundaries are prone to appear in the reconstructed depth map. In recent years, with the development of deep learning [30, 31, 32, 33], the research focus has gradually shifted to deep learning solutions for SR, and many high-performance algorithms have emerged, such as DBPN [30], DEWRN [31], SRN [32], SADN [34], DTSR [35], [36, 37], etc. Taking into account the particularity of the depth map, deep learning-based DSR methods usually needs to design a specific network structure to improve the reconstruction performance. Riegler et al.[38] incorporated the total generalized variational constraint at the back-end of DCNN to form an end-to-end ATGV-Net. Song et al.[39] proposed to reconstruct the depth map in a way that a series of view synthesis sub-tasks can be learned in parallel. Sun et al.[40] proposed a depth-controlled slicing network that learns a set of slicing branches in a divide-and-conquer manner and is parameterized by a distance-aware weighting scheme to adaptively integrate the different depths in the set.

As mentioned earlier, it is effortless for some depth cameras (such as Kinect) to acquire HR color images while acquiring depth maps. Therefore, the color-guided DSR model has received widespread attention in recent years and has become the mainstream model. The color-guided DSR is based on the similar structural information between the depth map and the aligned color image, i.e., the depth boundaries have a strong symbiotic relationship with the luminance boundaries. Filter-based approaches consider coeval structural relationships in addition to depth-neighborhood relationships when designing filters. For example, Kopf et al.[41] proposed a joint bilateral upsampling filter model by combining a Gaussian function based on the depth image neighborhood position relationship. He et al.[42] developed a local linear model of the filtered image and the bootstrap image model, and then proposed a bootstrap filter. Wang et al.[43] proposed a dual normal-depth regularization term to constrain the edge consistency between the normal map and the depth map. Recently, learning-based approaches have successfully applied DCNN to the field of color-guided DSR. Wen et al.[28] used a coarse-to-fine DCNN network to learn different filters with different kernel sizes, thus enabling data-driven training to replace the manually designed filters. Huang et al.[44] proposed a deep dense residual network with a pyramidal structure that leverages multi-scale features to predict high-frequency residuals through dense connectivity and residual learning. Guo et al.[25] designed a residual U-Net structure for the deep reconstruction task and introduced hierarchical feature-driven residual learning. Zuo et al.[21] proposed a data-driven super-resolution network based on global and local residual learning. Sun et al.[24] proposed a progressive multi-branch aggregation network that utilizes multi-scale information and high-frequency features to fully reconstruct the depth map in a progressive way. They also demonstrate that the low-level color information is only suitable for early feature fusion and does not help much for DSR at $\times$2 and $\times$4 cases.

The overview of the proposed network is shown in Fig. 2, which is a dual-stream hierarchical reconstruction architecture. Given a LR depth map $D_{LR}\in\mathbb{R}^{h\times w\times 1}$ and the corresponding HR color image $I_{HR}\in\mathbb{R}^{H\times W\times 3}$ as inputs, the goal of our task is to reconstruct and generate a SR version of depth map $D_{SR}\in\mathbb{R}^{H\times W\times 1}$ with the same resolution of the color image. To be concise, we first extract the multi-level features of RGB and depth features via five progressive convolution blocks (green blocks in Fig. 2), where each block includes two $3\times 3$ convolution layers and a $1\times 1$ convolution layer. The obtained RGB and depth features are denoted as $\mathrm{F}_{c}^{i}$ and $\mathrm{F}_{d}^{i}$ ($i=\{1,2,3,4,5\}$), respectively.

Then, we achieve color-guided depth feature learning and detail restoration under the cooperation of the AFP, LDE, and HAG modules. It is worth noting that there are three inputs (if any) are sent to the AFP module: (1) The depth backbone features $\mathrm{F}_{d}^{i}$ in the corresponding level; (2) The low-level detail features $\mathrm{F}_{LDE}^{i}$ generated by the LDE module, which is used for detailed restoration in the low-level reconstruction stage; (3) The dense transfer features ($\mathrm{F}_{tr}^{i+1},\mathrm{F}_{tr}^{i+2},\cdots,\mathrm{F}_{tr}^{5}$) from all the completed reconstruction levels. At different reconstruction levels, the input features of the AFP module are different, which are specifically formulated as:

where $Concat$ denotes the concatenation operation along the channel dimension, $\mathrm{F}_{d}^{i}$ represent the depth backbone features of the $i$-th level, $\mathrm{F}_{LDE}^{i}$ are the low-level detail features generated by the $i$-th LDE module, and $\mathrm{F}_{tr}^{k}$ denote the transfer features from the $k$-th completed reconstruction level, which can be calculated by:

where $\mathrm{F}_{HAG}^{k}$ represent the output features of the $k$-th HAG module, $\downarrow$ denotes the downsampling operation, and $k=\{i+1,i+2,\cdots,5\}$. It should be noted that the inputs of the LDE module include the depth features $\mathrm{F}_{d}^{i}$ and color features $\mathrm{F}_{c}^{i}$ of the corresponding layer.

After that, the high-level abstract features $\mathrm{F}_{c}^{5}$ and the depth backbone features $\mathrm{F}_{d}^{i}$ are fed into the HAG module to modify the output features $\mathrm{F}_{AFP}^{i}$ of AFP module and generate the reconstruction features $\mathrm{F}_{HAG}^{i}$. Finally, the pixel-shuffle and convolution operations are performed on the features $\mathrm{F}_{d}^{1}$ and $\mathrm{F}_{HAG}^{1}$ to obtain the final upsampled depth map $D_{SR}$.

Note that, our model is trained by minimizing $L_{1}$ loss, which can be formulated as:

where $D_{SR}$ and $D_{HR}$ denote the predicted depth SR result and ground truth, respectively, and $\left\|\cdot\right\|_{1}$ is the $L_{1}$ norm function. In the following subsections, we will provide the technical details of the AFP, LDE, and HAG modules one by one.

In order to achieve the depth map super-resolution, we need to map the low-resolution features to the desired high-resolution reconstruction features. Specifically, there are two issues that need to be paid attention to: (1) In order to recover more severely degenerated local details (such as depth boundaries and fine objects), simply increasing the depth of the network is insufficient and unwise. Therefore, we introduce a Multi-scale Content Enhancement (MCE) block to enhance the depth features before projection, using different receptive fields to recover detailed features at different scales as much as possible. (2) The information between the LR and HR domains is not absolutely one-to-one correspondence in the projection process, and the interference of excessive interfering information is likely to introduce additional errors, thereby impairing the reconstruction accuracy. To this end, we propose an Adaptive Attention Projection (AAP) block to project valid information in the attention domain, guaranteeing the effectiveness and compactness of the projected features. Note that, four cascaded AAP blocks are used in the AFP module to achieve better performance. In summary, as shown in Fig. 3, the MCE block and AAP blocks together form the AFP module to achieve depth feature reconstruction.

As is well-known, high-resolution color images are readily available and contain much useful information, such as boundaries, textures, and semantic information, etc. Therefore, introducing color guidance into the DSR model has become the mainstream idea in this field. However, there is no complete consensus on which color information to use and how to use it. Considering the different roles of color features at different levels, we provide a differentiated solution of color guidance strategy in this paper. Concretely, we design a Low-level Detail Embedding (LDE) module at the low-level reconstruction stage to leverage low-level color features for enhancing the high-frequency details of depth features, such as boundaries. In addition, we design a High-level Abstract Guidance (HAG) module, where the high-level abstract color features are used to perform content correction on the original reconstruction features, preventing content shifts during the depth reconstruction. We will introduce the LDE module in this subsection, and provide the details of the HAG module in the next subsection.

For depth map super-resolution, accurate and sharp boundary reconstruction has always been the focus of researchers’ unremitting efforts. It just so happens that what the low-level layer of the color branch learns is detailed information such as texture, boundary, etc. Therefore, we introduce the color features at lower levels (i.e., the first two layers) of the HR color branch through the LDE module and take the output as one of the inputs of the AFP module. However, the depth boundaries are not absolutely consistent with the RGB boundaries. In fact, the boundaries in the depth map are mainly the object boundaries, while the color image includes rich texture boundaries inside the object in addition to the object boundaries. Obviously, the texture boundaries are interferences for DSR. However, it is difficult to determine the delineation of texture and boundaries very clearly in network reconstruction, so completely discarding texture details is not the best option. Therefore, instead of removing all texture information with absolute gradient boundaries, our proposed LDE module suppresses the interfering information of RGB by learning residual masks, which is shown in Fig. 4.

Concretely, we first make a domain mapping of the RGB-D features using point-wise convolution and depth-wise convolution. The subsequent key step of subtraction actually serves to distinguish between similar as well as interfering regions in the RGB-D features. Since explicitly filtering out the appropriate complementary regions is difficult, we obtain the regions with large variances in the RGB-D modality, i.e., the redundant regions, by subtractive operations. The reverse residual mask, unlike the binarized mask, will increase the weight of complementary information and reduce the weight of information in redundant regions, thus differentiating the guiding role of low-level color features:

where $\mathrm{RM}^{i}$ denotes the residual mask, $W_{c}$ and $W_{d}$ represent the mapping matrix for the color features and depth features, $Sigmoid$ is the normalization operation, and $i$ indexes the lower level here, which is equal to 1 or 2.

In this way, the residual mask highlights the most relevant part of the color and depth information, so we multiply it with the initial color features to obtain the effective color features that can be used for depth reconstruction guidance. Moreover, we believe that the feature representation of the color information filtered by the residual mask in the lower levels is in the same domain as the depth information, so the final fusion adopts a direct summation scheme, which is also consistent with objective rules. Therefore, the final output of the LDE module can be formulated as:

where $\otimes$ denotes the element-wise multiplication.

As analyzed earlier, the existing methods mainly focus on extracting color features to supplement the details for depth reconstruction, just like the functions implemented by our LDE module. However, let us rethink the role of color-guided features: Is this detailed guidance strategy sufficient? In fact, high-level color features are very important for many tasks, which contains abstract global information and preserves semantic outlines. In the DSR task, the existing methods ignore one issue, i.e., the global abstract information preserving ability of the reconstructed features. As the reconstruction process progresses, there is a possibility that semantic consistency may be shifted or blurred, which is very unfavorable for the subsequent depth-oriented application tasks. This is mainly caused by the lack of global guidance in the reconstruction. Fittingly, our color branch can provide high-resolution, offset-free color guidance information. This also benefits from the fact that the semantic information extraction for the color branch does not rely on the process of depth reconstruction, but extracts shallow texture features and high-level abstract features through multi-layer convolution deepening. Since no resolution change is involved, the high-resolution color information is not shifted during the extraction process. Inspired by these, we design a HAG module to maintain the content attribute during the depth reconstruction, which is equipped after each AFP module. Specifically, the high-level color information at the top layer of the color branch is utilized to generate a mask that encodes the global content guidance information, and is further used to modify the initial reconstruction features $\mathrm{F}_{AFP}^{i}$ (i.e., the output of the AFP module).

As shown in Fig. 5, we first enhance the top-layer color features $\mathrm{F}_{c}^{5}$ at the spatial level [47], thereby generating the reweighted color features $\mathrm{F}_{E}^{i}$ that highlight the important locations:

where $FC$ denotes the fully connected layers, $Max$ and $Avg$ are the max-pooling operation along channel dimension and global average pooling, respectively, and $\mathrm{Act}_{SL}^{i}$ is the spatial-level attention.

Considering the auxiliary role of semantic features, we still take the depth reconstruction features as the dominant one in the guiding process. Thus, we concatenate the enhanced color features with the original reconstruction features, and then generate a semantic mask:

where $PReLU$ is the parametric rectified linear unit, and $Conv_{n\times n}$ denotes the convolution layer with the kernel size of $n\times n$.

With the semantic mask, the original depth reconstruction features can be refined by:

where $\mathrm{F}_{HAG}^{i}$ are the output of the corresponding HAG module. It should be noted that $\mathrm{F}_{HAG}^{i}$ are the final reconstruction feature at each stage, where the features of the last layer $\mathrm{F}_{HAG}^{1}$ will be directly used to generate the upsampled depth map, and the reconstruction features of other layers will be sent to the AFP module through the dense transmission to realize the progressive learning of the entire network.

To demonstrate the effectiveness of the proposed method, we conduct comprehensive experiments on the Middlebury dataset, NYU v2 dataset [58], real-world RGB-D-D dataset [59], and Lu dataset [60]. All these datasets are produced with the alignment of color and depth images, and all the ground-truth values of the upsampled depth map are owned by the dataset. From the perspective of dataset construction, for depth cameras such as Kinect, most of the scenes are RGB-D aligned with default parameters. With tools such as Matlab, the accuracy can be further improved by co-calibration. In addition, the device usually includes an SDK that enables users to interpolate the depth output to match the resolution of RGB images. However, the quality of the interpolated depth map can not be guaranteed. Thus, the NYU v2 dataset undergoes additional processing such as depth completion and calibration for improved quality and accuracy.

• •

  We collect 36 RGB-D images from Middlebury dataset (6, 21, 9 images from 2001 [61], 2006 [62], and 2014 [63] datasets, respectively) for training, and 6 standard depth maps from Middlebury 2005 dataset [64] for testing. The resolution of the image is mostly around $1000\times 1000$.

• •

  As for the NYU v2 dataset, it is a real-scene dataset captured by the Kinect camera. We use the first 1000 pairs with the resolution of $640\times 480$ for training and the remaining 449 pairs for testing. The model trained on the NYU v2 dataset is also directly used to test on the Lu dataset and RGB-D-D dataset for generalization evaluation.

• •

  The RGB-D-D dataset is a real-world indoor dataset captured by a Huawei P30 pro cellphone equipped with the color camera and time of flight (TOF) camera, also with a resolution of $640\times 480$. Following the setting in [59], 2215 pairs are used for training and 405 pairs for evaluation.

• •

  The Lu dataset only consists of 6 RGB-D pairs acquired by the ASUS Xtion Pro camera, all of which are used for testing.

For quantitative evaluation, the metrics of Mean Absolute Difference (MAD) and Root Mean Square Error (RMSE) are introduced [22, 70, 71]. Following PMBANet [24], we augment our training samples by cropping the HR pairs into around 15000 HR patches with the squared size of 64, 128, and 256 according to the upsampling factors of $\times 4$, $\times 8$, and $\times 16$. Meanwhile, we perform horizontal and vertical flipping of the training samples with a probability of 0.5. The LR depth patches are downsampled into a fixed size of $16\times 16$ from HR depth patches via Bicubic interpolation. With respect to the structural details of our HCPNet, in the feature extraction phase, the network maps the RGB-D features to each of the 64-channel dimensions and maintains the number of channels constant throughout the feature extraction phase. For the LDE, HAG, and MCE modules, the input channel dimension changes depending on the number of input features, and the output is uniformly 64 channels. Our HCGNet is implemented by PyTorch with an NVIDIA 3090 GPU. As a hyperparameter, the batch size is set to 8 at $\times 8$ and $\times 4$ factor cases and set to 4 for the $\times 16$ factor case. During training, we use Adam optimizer with momentum of $0.9$, and the network optimization parameters of $\beta_{1}$, $\beta_{2}$, and $\epsilon$ are set to $0.9$, $0.99$, and $1e^{-8}$, respectively. The initial learning rate is set to $1e^{-4}$, and it is decreased by multiplying by 0.1 for the first 100 epochs and the next 50 epochs.

1) Middlebury Dataset: We compare our method with some state-of-the-art DSR methods under different upsampling factors ($\times 4$, $\times 8$, and $\times 16$), including four traditional depth SR methods (i.e., TGV [48], EG [49], JGF [50], and CDLLC [51]) and nine deep-learning based methods (i.e., GSRPT [52], MDDL [53], DEIN [54], DJF [55], CCFN [28], CTKT [56], BridgeNet[22], PMBANet [24] and MIGNet [57]).

The quantitative comparisons of the MAD score are reported in Table I. We can observe that the traditional DSR models often fail to achieve satisfactory performance, especially when dealing with more complex scenes (e.g., Art) or larger upsampling factors (e.g., $\times 16$). In contrast, the deep-learning based DSR models show more competitive performance, in which the proposed HCGNet outperforms other SOTA methods among different scenarios and achieves the best overall average MAD performance under different upsampling factors. Moreover, our method achieves obvious performance improvement under a challenging large sampling factor (such as $\times 16$). Compared with the second best method, the MAD value of the Mobius scene is decreased from 0.54 to 0.45, with a percentage gain of 16.7%, and the MAD percentage gain of the Books scene reaches 13.7%. Fig. 6 demonstrates the visual comparisons of different methods under the factor of $\times 8$. As visible, our method can recover more complete and accurate depth details. In the first image, compared with the results of other methods, the boundaries around sticks are sharper and smoother with less jagged noise, and the structure of the distant object is more complete and continuous. For the Dolls image, our method can restore the clear and sharp boundaries of the reconstructed objects. For example, the left ear of the doll in the third row is restored with complete contour and the surrounding values are artifact-free, and the sleeves of the doll (e.g., the raised part in the middle) shown in the last row are also recovered more satisfactorily than other methods. In general, the comparison methods can more or less introduce additional shape distortion and noise, resulting in blurring, discontinuities, and changed depth values in the reconstructed images. In contrast, our method is able to recover objects more completely while taking into account the restoration of depth details without introducing destructive contamination.

2) NYU v2 Dataset: We evaluate our method on the NYU v2 dataset and compare it with other SOTA methods, including Bicubic, TGV [48], DJFR [65], SDF [66], SVLRM [67], DKN [19], FDKN [19], FDSR [59], PMBANet [24], JIIF [68] and DCT [69].

The quantitative results are provided in Table II. We can see that our method outperforms all the SOTA methods under different upsampling factors. In the most challenging $\times 16$ upsampling situation, compared with the second best method, the RMSE of our method reaches $4.85$, with a percentage gain of $7.4\%$. Fig. 7 shows some visual comparisons on the NYU v2 dataset under the $\times 8$ factor case. It can be seen that our method has obvious advantages in boundary reconstruction and depth preservation. For example, in the overlapping regions of the bed sheet and bed frame marked in purple in the first image, the depth map reconstructed by our method has sharper boundaries and more accurate depth values. In the second image, our method shows a stronger ability to portray details, such as the object above the trash can in the left image and the oblique border area of the chair in the right image.

3) RGB-D-D Dataset: On this dataset, we evaluate our method (i.e., HCGNet and HCGNet⁺) and other SOTA methods, including SDF [66], DJFR [65], DKN [19], FDKN [19], FDSR [59], FDSR⁺, DCT [69], and JIIF [68]. The superscript ‘$+$’ indicates the corresponding model was retrained on the RGB-D-D dataset, and no superscript mark indicates the model was only trained on the NYU v2 dataset [58] without retraining or fine-tuning.

The quantitative results are reported in Table III. Compared to other models without retraining, our method (i.e., HCGNet) achieves the best performance under large factor cases and even outperforms the retrained FDSR⁺ method under the most challenging case of $\times 16$, which also demonstrates the generalizability of our model. Concretely, compared with the second best method without retraining, the average RMSE value drops from 3.05 to 2.70 under $\times 16$ case with a percentage gain of 11.5%, and the percentage gain under $\times 8$ factor reaches 5.5%. At the same time, we can see a further improvement in the performance of the model after retraining. Our retrained version (i.e., HCGNet⁺) achieves the best performance under all factor cases and the percentage gain reaches 33.6% under $\times 16$ case compared with the retained FDSR⁺ method. For the visual comparison, Fig. 8 demonstrates the details on the RGB-D-D dataset under $\times 8$ case. In the first image, our method restores the leaves with clear and sharp boundaries, and even the sawtooth shape of the leaves in the lower right corner is partially restored. In the more challenging second image, our method recovers not only the accurate values both around and inside the cube in the lower left corner but also the complete boundaries of the books.

4) Lu Dataset: We also evaluate our method on the Lu Dataset and compare it with other SOTA methods, including Bicubic, FDKN [19], DKN [19], FDSR [59], DCT [69] and JIIF [68]. The Lu dataset was captured in low light and a relatively complex environment, which challenges the generalization ability of the method. The scale of the dataset is relatively small, so all methods are not retrained on this dataset. As shown in Table IV, our method achieves competitive performance compared with other methods, especially at the large factor cases (i.e., $\times$8, $\times$16). For example, compared with the second best method, the average RMSE value is decreased from 4.16 to 3.75 under $\times 16$ factor case, with a percentage gain of 9.9%. Fig. 9 shows a visual example of different methods on the Lu dataset. As visible, the proposed method punches above its weight in terms of detail reconstruction, such as the sharper borders around the statue and more accurate depth values (e.g., the ears) inside the statue.

1) Validation of modules. Ablation studies are conducted to verify the effectiveness of each key component designed in the proposed HCGNet on both the Middlebury 2005 dataset and the NYU v2 dataset. We remove the color branch and simplify the AFP module as the baseline model. Specifically, we replace the MCE block with a few simple convolution layers and replace the AAP block with the residual block accordingly. Then, we sequentially add the HAG, LDE, and AFP modules to the baseline to verify the effectiveness of each module. For the validation of MCE and AAP blocks in AFP, we replace the MCE with the similar RDB [72] block and the AAP module with the similar DBPN [30] block. The quantitative results under the $\times 8$ case are reported in Table V, and some visual examples are shown in Fig. 10.

First, we verify the role of color guidance in DSR. After introducing the HAG module into the baseline to provide high-level abstract guidance, we can see that the MAD value on the Middlebury dataset is improved from 0.42 to 0.36 with the percentage gain of 14.3%, and RMSE values on the NYU v2 dataset decreased from 3.80 to 2.88 with the percentage gain of 24.2%. As visible, compared with the second and third columns in the first image of Fig. 10, the indistinguishable hole regions in the baseline model (marked in red box) have been obviously restored. Then, we embed the color detail information into the depth reconstruction branch through the LDE module. As reported in model 3 of Table V, the performance is further improved, where the percentage gains of the MAD value on the Middlebury 2005 dataset and RMSE value on the NYU v2 dataset reach 5.6% and 4.9% compared with model 2, respectively. At the same time, as shown in Fig. 10(d), the holes in the visualization result are also reconstructed more clearly. In summary, all these results demonstrate that our proposed HAG and LDE modules provide effective guidance information for depth map super-resolution reconstruction from two perspectives, and improve the reconstruction accuracy.

Furthermore, we verify the effectiveness of the proposed AFP module. On the basis of our DSR framework for color guidance, we introduce the AFP module to achieve multi-scale content enhancement and adaptive attention projection. Compared with model 3 in Table V, the MAD score is further improved from 0.34 to 0.29 with a percentage gain of 14.7%, and the RMSE score is decreased from 2.74 to 2.53 with a percentage gain of 7.7%. In terms of the visualization results shown in Fig. 10(f), the final full model (model 5) can not only maintain the details of the holes in the red-marked area but also effectively update the incorrectly reconstructed holes in the lower right corner of model 3 in the first row. To further illustrate the superiority of the AFP module design over similar methods, we add the ablation experiments of model 4. Comparing model 4 and model 5, we can see that after replacing MCE and AAP with RDB and DBPN respectively, the model performance drops obviously. On the Middlebury 2005 dataset, the MAD score increases from 0.29 to 0.32, with a performance drop of 10.3%. While on the NYU v2 dataset, the RMSE score increases from 2.53 to 2.61, with a performance drop of 3.2%. The experimental results also prove the effectiveness of our proposed AFP module in-depth reconstruction.

Finally, we conduct an ablation study to compare the performance of different numbers of modules within the AFP and AAP components, the results of which are illustrated in Table VI. Specifically, we conduct experiments on the AFP modules by sequentially removing them from level 5 and level 4 of the network, while still retaining the HAG’s global guidance. The results indicate that the removal of one AFP module (reducing the number from five to four) leads to an increase in RMSE from 2.53 to 2.63, corresponding to a performance decrease of 4%. Since the proposed HCGNet is a hierarchical progressive reconstruction, the quality of the reconstruction at each layer has an essential impact on subsequent layers. For the ablation of AAP blocks, compared with the four AAP blocks, the RMSE value of three blocks increases from 2.53 to 2.64, with a performance drop of 4.3%. Meanwhile, observation reveals that too many AAP blocks can also lead to performance degradation. This is because, at the current stage, the quality of the depth features is limited, and the correction capability of the AAP module is similarly constrained. Excessively deep stacking may, in fact, hinder the restoration of the depth features. Based on the above experimental results, we finally set the number of AAP to 4.

2) Portability of modules. To further validate the effectiveness of the proposed LDE and HAG modules, we conduct validation experiments by adding LDE and HAG modules to the PMBANet [24], which is a network considering only low-level color information (Mode (a) in Fig. 1). Specifically, we use the depth features of the corresponding layer, which is modified by the LDE module, as the input of the current MBA block, while the global HAG module is used to refine the depth features after each reconstruction branch. As shown in Table VII, it can be seen that after adding these two modules to the PMBANet, the performance is improved on both two datasets. For example, on the Middlebury 2005 dataset, with both LDE and HAG modules, compared with the original PMBANet, the average MAD value is improved from 0.33 to 0.30. On the NYU v2 dataset, introducing both the LDE and HAG modules reduces the RMSE of PMBANet from 2.75 to 2.58 with a percentage gain of 6.2%. Some visual examples of adding our modules to PMBANet are shown in Fig. 11. We can see that, compared with the original result of PMBANet, the PMBANet model with our LDE and HAG modules improves the accuracy and details of the reconstruction, such as the hole regions. These experiments all again demonstrate the effectiveness and portability of our proposed hierarchical color guidance mechanism.

For the training time, it takes about 24 hours to obtain the final $\times 8$ model for 200 epochs with a batch size of 8. And our $\times 4$ model only requires about 10 hours for training with a batch size of 8. This suggests that the training time is correlated with both our specific task and batch size settings, thus existing methods focus more on the running time of the test. The running time comparisons on the NYU v2 dataset ($640\times 480$) are shown in Table VIII. It can be seen that our method processes $\times 8$ DSR task in less than 20ms per image on the NYU v2 dataset ($640\times 480$) while achieving optimal performance compared to other SOTA methods. In other words, our model achieves a good balance in terms of performance and efficiency.

In terms of performance, the advantages are more pronounced when dealing with more complex scenes or larger upsampling factors. But as reported in Table III and Table IV, our method has no obvious advantages when dealing with low-scale ($\times 4$) DSR on the Lu dataset and real-world RGB-D-D dataset. Some failure cases under the $\times 4$ factor on the RGB-D-D dataset are also provided in Fig. 12. It has been observed that our algorithm may suffer from reconstruction errors when confronted with regions of sudden changes in brightness inside objects in the low-depth range of the scene. In these regions, the brightness boundary of the color image varies greatly, but the depth variation is small, so the inconsistency between the color brightness and depth boundaries may cause reconstruction errors in small regions. Therefore, the robustness of the model to such challenging scenarios needs to be further optimized in the future.