One at a Time: Progressive Multi-step Volumetric Probability Learning for
Reliable 3D Scene Perception

Given a ground truth depth map $d^{gt}$, we first construct the target volume $\boldsymbol{y}_{0}$ following the unimodel projection (Ding et al. 2022; Peng et al. 2022) along depth dimension $D$ as diffusion input:

$$\boldsymbol{y}_{0}=Project^{Uni}\left\{d^{gt},Dim=D\right\},$$

(1)

We gradually add noise on $\boldsymbol{y}_{0}$ to generate the noisy volume $\boldsymbol{y}_{T}$ over $\boldsymbol{T}$ steps following a discrete-time Markov chain. Given distribution of $\boldsymbol{y}_{0}$, the forward process can be characterized as:

$$q\left(\boldsymbol{y}_{t}\mid\boldsymbol{y}_{0}\right)=\mathcal{N}\left(\boldsymbol{y}_{t}\mid\sqrt{\bar{\alpha}_{t}}\boldsymbol{y}_{0},\left(1-\bar{\alpha}_{t}\right)\boldsymbol{I}\right),$$

(2)

where $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$ and $\alpha_{t}$ is the pre-defined coefficient. $\mathcal{N}$ and $\boldsymbol{I}$ denote the normal distribution and the identical matrix, respectively.

The conditional reverse sampling process is dedicated to iteratively denoise $\boldsymbol{y}_{T}$ with conditions to recover $\boldsymbol{y}_{0}$. Each step of the reverse process can be defined as conditional distribution transition (Saharia et al. 2022), which is formulated as:

$$p_{\theta}\left(\boldsymbol{y}_{0:T}\mid\boldsymbol{x}\right)=p\left(\boldsymbol{y}_{T}\right)\prod_{t=1}^{T}p_{\theta}\left(\boldsymbol{y}_{t-1}\mid\boldsymbol{y}_{t},\boldsymbol{x}\right),$$

(3)

where $p_{\theta}$ represents the reverse function, $\boldsymbol{x}$ denotes the two conditions of the diffusion model.

Online Filtering. Since VPD is dedicated to learning for approximating the target volume $\boldsymbol{y}_{0}$ with unimodal distribution, we propose to filter the predicted ${\boldsymbol{y}}_{t}$ online at each iteration to suppress the perturbation caused by the generated multi-model representation before sending it to the next reverse sampling step. Our implementation can be formally written as:

$${\boldsymbol{y}}_{t}^{\prime}=Project^{Uni}\left\{WTA^{D}\left({\boldsymbol{y}}_{t}\right),Dim=D\right\},$$

(4)

where $Project^{Uni}$ denotes unimodel projection same as GT volume construction in Section 3.1. $WTA^{D}$ represents Winner-Takes-All operation (Cheng et al. 2020), which maintains the unique peak along the depth dimension.

We employ Coarse Volume Probabilization (CVP) to construct the coarse probabilistic volume, which is concatenated with the input noisy volume as the basic prior volume condition of the diffusion model. Given a coarse cost volume $\textbf{V}_{cost}$ from baseline networks, we employ $softmax$ along the depth dimension for each pixel $(h,w)$ in space to implement the volume probabilization, which is formally written as:

$${\textbf{V}_{prob}^{h,w,m}=Softmax(\textbf{V}_{cost}^{h,w,m})=\frac{\exp(d_{m}^{h,w})}{\sum_{n=1}^{D_{max}}\exp(d_{n}^{h,w})},}$$

(5)

where $d_{m}^{h,w}$ represents cost value of $m^{th}$ depth hypothesis plane ($1<m<D_{max}$). $D_{max}$ denotes the number of depth hypothesis planes. For multi-view stereo (MVS), we adopt regularized cost volumes (Gu et al. 2020; Long et al. 2021; Ding et al. 2022; Peng et al. 2022) as the volume $\textbf{V}_{cost}$, while the geometric cost volumes (Li et al. 2023a) are employed for semantic scene completion (SSC).

Although the CVP provides basic geometry prior, it is still hard to achieve compelling results, especially in challenging regions like occlusions, reflections, textureless regions, etc. Thus, we propose a Confidence-Aware Contextual Collaboration (CACC) module to further apply continuous refinement with the contextual feature condition on the estimated volumes.

The overall structure of CACC is shown in Figure 2. Given a depth volume $\textbf{V}_{depth}^{i}\in\mathbb{R}^{C\times D\times H\times W}$ in $i^{th}$ downsample block of the 3D UNet (i.e., diffusion model) and $i^{th}$ scale contextual features $\textbf{F}_{context}^{i}\in\mathbb{R}^{C^{\prime}\times H\times W}$ from feature extraction networks, our goal is to retrieval reliable multi-scale contextual features from $\textbf{F}_{context}^{i}$, and refine $\textbf{V}_{depth}^{i}$ according to the confidence information along the spatial dimension. It is worth noting that we directly obtain multi-scale contextual features from the off-the-shelf baseline networks for computational efficiency.

Specifically, we first form a confidence map $\textbf{C}^{i}\in\mathbb{R}^{C\times H\times W}$ by checking the highest probability value among all depth hypothesis planes across the depth dimension. Next, we reverse the values in $\textbf{C}^{i}$ to obtain query $Q^{i}$ for cross attention that measures the matching uncertainty in $\textbf{V}_{depth}^{i}$:

$$Q^{i}=Sigmoid(-{\mathbf{C}}^{i})=Sigmoid\left\{-\left(WTA^{D}\left(\mathbf{V}^{i}\right)\right)\right\},$$

(6)

where $WTA^{D}$ denotes Winner-Takes-All operation along depth dimension. To generate key $K^{i}$ and value $V^{i}$, we apply deformable convolution on the corresponding contextual features $\textbf{F}_{context}^{i}$ for efficient geometric transformation modeling and receptive fields adaption. For each location point p on the contextual features $\textbf{F}_{context}^{i}$, the process is formulated as:

$$K^{i}(\textbf{p}),V^{i}(\textbf{p})=Chunk\left\{\sum_{c=0}^{C^{\prime}-1}W\cdot\textbf{F}_{context}^{i}\left((\textbf{p})+\Delta_{(\textbf{p})},\mathbf{c}\right)\right\},$$

(7)

where $W$ and $\Delta_{(\textbf{p})}$ denotes the deformable weight and learnable offset, respectively. $Chunk$ represents splitting the input into halves along the feature channel. To reduce computation cost, we adopt linear attention (Kitaev, Kaiser, and Levskaya 2020; Shen et al. 2021) as:

$$\textbf{F}_{conf}^{i}=LinearAtten(Q^{i},K^{i},V^{i})=\phi_{q}(Q^{i})\left(\phi_{k}(K^{i})^{T}V^{i}\right),$$

(8)

where $\textbf{F}_{conf}^{i}$ represents confidence-aware context. $\phi_{q}$ and $\phi_{k}$ are $softmax$ operations along each row and column of the input matrix, respectively. In this way, the relevant information of contextual features is retrieved according to the matching uncertainty of the depth volume $\textbf{V}_{depth}^{i}$.

Subsequently, we implement element-wise multiplication between the depth volume $\textbf{V}_{depth}^{i}$ and confidence map $\textbf{C}^{i}$ to obtain a filtered volume. To match in dimension, $\textbf{F}_{conf}^{i}$ is projected into 3D contextual volume $\textbf{V}_{context}^{i}$ before adding to the filtered volume following lift operation (Philion and Fidler 2020). Finally, the refined volume is constructed by element-wise summation between the filtered volume and the contextual volume:

$$\textbf{V}_{refine}^{i}=\textbf{V}_{depth}^{i}\odot\textbf{C}^{i}+\textbf{V}_{context}^{i},$$

(9)

where $\odot$ denotes element-wise multiplication. Note that CACC is applied on each downsample block of the 3D UNet with different dimension sizes. Through the refinement operation of CACC on the depth volume, volumetric distribution in high-confidence regions is retained, while that in low-confidence regions is optimized with multi-scale contexts.

In Figure 3, we visualize the confidence map $\textbf{C}^{i}$, uncertainty map $Sigmoid(-{\mathbf{C}}^{i})$, estimated depth map without CACC and estimated depth map with CACC. It can be seen that the model without CACC struggles to achieve compelling results in challenging regions (e.g. object boundaries, low-texture regions). The confidence map and the uncertainty map illustrate the regions with poor estimation, which are effectively refined by retrieving information from the contextual features with CACC.

DTU (Aanæs et al. 2016) is a large-scale indoor dataset, which consists of 124 different scenes with 7 different illumination conditions. We split the dataset into training, validation, and test set following the setting of MVSNet (Yao et al. 2018). BlendedMVS (Yao et al. 2020) dataset is a synthetic dataset that consists of 106 training scans and 7 validation scans. ScanNet (Dai et al. 2017) is an RGB-D video dataset that consists of more than 1600 scans, annotated with depth maps and camera poses.

Our model is implemented on the Pytorch platform with 4 NVIDIA A100 GPUs. We train our model for 16 epochs on the DTU dataset and 7 epochs on the ScanNet dataset, respectively. For the BlendedMVS dataset, we implement tests using the model trained on the DTU dataset to evaluate the generalization ability. The initial learning rate is set to $2.5\times 10^{-5}$, which decays following the same strategy of baseline networks. During the training process, the batch size is set to 12 and we adopt Adam as the optimizer. The diffusion forward step $\boldsymbol{T}$ is set to 1000 and we adopt 4 iterations in the reverse process.

For quantitative evaluation, we conduct standard metrics (Eigen, Puhrsch, and Fergus 2014; Long et al. 2021; Cai, Ji, and Xu 2023), including absolute relative error (Abs Rel), absolute error (Abs), square relative error (Sq Rel), root mean square error in linear scale (RMSE), threshold distance error (Th) and inlier ratios ($\delta<1.25^{i}$, where $i\in\left\{1,2\right\}$). As reported in Table 1, our method shows significant improvements compared to TransMVSNet, reducing Abs by $31.46\%$.

We evaluate the zero-shot generalization ability of our method from DTU to BlendedMVS validation set without any fine-tuning. As shown in Table 2, our method has a notable performance gain compared to baseline networks, which demonstrates that our approach generalizes well across different datasets without post-processing. Table 3 shows quantitative results on the ScanNet test set. ESTD (Long et al. 2021) with VPD outperforms other methods in terms of accuracy, which indicates that our method also has strong modeling capability for temporal cost volumes.

Moreover, We visualize qualitative results on the DTU test set and BlendedMVS validation set in Figure 4. Our approach significantly enhances outcomes from baseline models, generating more complete depth maps with heightened accuracy, particularly in challenging areas like object boundaries and repetitive patterns.

SemanticKITTI (Behley et al. 2019) is a popular semantic scene completion dataset, which contains 22 outdoor driving scenes. SemanticKITTI holds LiDAR annotations that are voxelized as 256$\times$256$\times$32 grid of 0.2m voxels. The target voxel grids are labeled with 21 classes (1 free, 1 unknown, and 19 semantics). Following (Li et al. 2023a), we only adopt RGB images of the dataset as inputs.

We extend StereoScene (Li et al. 2023a) with our proposed VPD for SSC evaluation. More specifically, the geometric cost volume in StereoScene is leveraged as the coarse volume. The whole model is trained on SemanticKITTI for 30 epochs with a learning rate of $2.5\times 10^{-5}$. AdamW is adopted as a training optimizer following (Li et al. 2023a).

For quantitative evaluation, we adopt mIoU (mean Intersection over Union) to account for the SSC task. We compare our method with other state-of-the-art SSC networks: (1) camera-based methods including MonoScene (Cao and de Charette 2022), VoxFormer (Li et al. 2023b) and StereoScene (Li et al. 2023a), (2) LiDAR-based method of SSCNet (Song et al. 2017). As shown in Table 4, our method surpasses StereoScene by $6.18\%$ in terms of mIoU, which demonstrates the application of VPD effectively improves the accuracy of depth estimation and thereby enhances the performance of semantic scene completion. It’s worth noting that our method even surpasses the LiDAR-based method of SSCNet in terms of mIoU. Figure 5 visualizes the qualitative results, our method produces more accurate and complete results in large-scale driving scenarios compared with StereoScene.

For the experiment in the first row, we adopt basic CasMVSNet as the baseline setting. The reverse iterations are set to 4 unless otherwise stated. As shown in Table 5, the VPD framework brings significant improvement with the basic prior volume condition of CVP. The CACC and OF obviously enhance the depth estimation performance with reliable volumetric learning, reducing Abs by $5.20\%$ and $4.27\%$, respectively.

We report the running time and memory consumption of several schemes that are built upon different baselines equipped with our proposed VPD on the NVIDIA A100 GPU, which are detailed in Table 6. It’s worth noting that our method could effectively achieve compelling performance gains with acceptable time consumption. As shown in the table, the proposed VPD delivers satisfactory performance improvements with relatively slight increments in time consumption, while the memory consumption of our proposed method remains constant regardless of the number of reverse steps. In addition, the performance gain of more than 4 reverse steps with more running time is not obvious, thus we adopt 4 steps as the default setting to balance efficiency and effectiveness.