3D Scene Flow Estimation on Pseudo-LiDAR: Bridging the Gap on Estimating Point Motion

In recent year, many works [15, 16, 17] build pseudo-LiDAR based 3D object detection pipeline. Pseudo-LiDAR has shown significant advantages in the field of object detection. Wang et al. [15] introduce pseudo-LiDAR into an existing LiDAR object detection model and demonstrate that the main reason for the performance gap between stereo and LiDAR is the representation of the data rather than the quality of the depth estimate. Pseudo-LiDAR++ [16] optimizes the structure and loss function of the depth estimation network based on Wang et al. [15] to enable the pseudo-LiDAR framework to accurately estimate distant objects. Qian et al. [17] build on Pseudo-LiDAR++ [16] to address the problem that the depth estimation network and the object detection network must be trained separately. The previous pseudo-LiDAR framework focuses more on scene perception with the single frame, while this paper focuses on the motion relationship between two frames.

Some works study the estimation of dense scene flow from images of consecutive frames. Mono-SF [4] proposes ProbDepthNet to estimate the pixel depth distribution of the single image. Geometric information from multiple views and depth distribution information from a single view are used to jointly estimate the scene flow. Hur et al. [5] introduce a multi-frame temporal constraint to the scene flow estimation network. Chen et al. [1] develop a coarse-grained software framework for scene-flow methods and realize real-time cross-platform embedded scene flow algorithms. In addition, Rishav et al. [18] fuse LiDAR and images to estimate dense scene flow. But they still perform feature fusion in image space. These methods rely on 2D representations and cannot learn geometric motion from explicit 3D coordinates. The pseudo-LiDAR is the bridge between the 2D signal and the 3D signal, which provides the basis for directly learning the 3D scene flow from the 2D data.

The original geometric information is preserved in the 3D point cloud, which is the preferred representation for many scene understanding applications in self-driving and robotics. Some researchers [7] estimate 3D scene flow from LiDAR point clouds by using the classical method. Dewan et al. [7] introduce local geometric constancy during the motion and introduce a triangular grid to determine the relationship of the points. Benefiting from the point cloud deep learning, some recent works [8, 9, 10, 11] propose to learn 3D scene flow from raw point clouds. FlowNet3D [8] firstly proposes the flow embedding layer which finds point correspondence and implicitly represents the 3D scene flow. FLOT [10] studies lightweight structures for optimizing scene flow estimation using optimal transport modules. PointPWC-Net [9] proposes novel learnable cost volume layers to learn 3D scene flow in a coarse-to-fine approach, and introduces three self-supervised losses to learn the 3D scene flow without accessing the ground truth. Mittal et al. [11] propose two new self-supervised loss.

The main purpose of this paper is to recover 3D scene flow from stereo images. The stereo images are represented by $I^{l}$ and $I^{r}$ respectively. As Fig. 2, given a pair of stereo images, which contains reference frames $\{I_{t}^{l},I_{t}^{r}\}$ and target frames $\{I_{t+1}^{l},I_{t+1}^{r}\}$. Each image is represented by a matrix of dimension $H\times W\times 3$. Depth map $D_{t}$ at time $t$ is predicted by feeding the stereo image $\{I_{t}^{l},I_{t}^{r}\}$ into the depth estimation network $D_{net}$. Each pixel value of $D$ represents the distance $d$ between a certain point in the scene and the left camera. Pseudo-LiDAR point cloud comes from back-projecting the generated depth map to a 3D point cloud, as follow:

where $f_{x}$ and $f_{y}$ are the horizontal and vertical focal lengths of the camera, and $c_{x}$ and $c_{y}$ are the coordinate center of the image, respectively. The 3D point coordinate $(x_{w},y_{w},z_{w})$ in the pseudo-LiDAR point cloud $\mathcal{PL}$ is calculated by pixel coordinates $(u,v)$, $d$ and camera intrinsics. $\mathcal{PL}_{t}=\{c_{1,i}\in{\mathbb{R}^{3}}\}_{i=1}^{N_{1}}$ with $N_{1}$ points and $\mathcal{PL}_{t+1}=\{c_{2,j}\in{\mathbb{R}^{3}}\}_{j=1}^{N_{2}}$ with $N_{2}$ points are generated from the depth maps $\mathcal{D}_{t}$ and $\mathcal{D}_{t+1}$, where $c_{1,j}$ and $c_{2,j}$ are the 3D coordinates of the points. $\mathcal{PL}_{t}$ and $\mathcal{PL}_{t+1}$ are randomly sampled to $N$ points, respectively. The sampled pseudo-LiDAR point clouds are passed into the scene flow estimator $\mathcal{F}_{sf}$ to extract the scene flow vector $\mathcal{SF}_{t}=\{sf_{i}|i=1,2,\cdots,N\}$ for each 3D point in frame $t$.

where $\theta_{1}$ and $\theta_{2}$ are the parameters of the network. $\mathcal{P}$ represents the back-projection by Eq. (1).

It is difficult to obtain the ground truth scene flow of pseudo-LiDAR point clouds. Mining a priori knowledge from the scene itself to self-supervised learning of 3D scene flow is essential. Ideally, $\mathcal{PL}_{t+1}$ and estimated point cloud $\mathcal{PL}_{t+1}^{\omega}$ have the same structure. With this priori knowledge, point cloud $\mathcal{PL}_{t}$ is warped to point cloud $\mathcal{PL}_{t+1}^{\omega}$ through the predicted scene flow $\mathcal{SF}_{t}$,

Based on the consistency of $\mathcal{PL}_{t+1}$ and $\mathcal{PL}_{t+1}^{\omega}$, the loss functions Eq. (7) and Eq. (10) are utilized to implement self-supervised learning. We provide the pseudocode in Algorithm 1 for our method, where $L$, $CV_{pc}$, and $Pred_{SF}$ are described in detail in Section III-D.

The disparity $z$ is the horizontal offset of the corresponding pixel in the stereo image, which represents the difference caused by viewing the same object from a stereo camera. $I_{l}(u,v)$ and $I_{r}(u,v+z)$ represent the observation of the same 3D point in space. Two cameras are connected by a line called the baseline. The distance $d$ between the object and the observation point can be calculated by knowing the disparity $z$, the baseline length $b$, and the horizontal focal length $f$.

Disparity estimation networks such as PSMNet [19] extract deep feature maps $\{F_{t}^{l},F_{t}^{r}\}$ and $\{F_{t+1}^{l},F_{t+1}^{r}\}$ from $\{I_{t}^{l},I_{t}^{r}\}$ and $\{I_{t+1}^{l},I_{t+1}^{r}\}$, respectively. As shown in Fig. 2, the features of $F_{t}^{l}(u,v)$ and $F_{t}^{r}(u,v+z)$ are concatenated to construct 4D tensor $C_{disp}(u,v,z,:)$, namely the disparity cost volume. Then 3D tensor $H_{disp}(u,v,z)$ is calculated by feeding $C_{disp}(u,v,z,:)$ into the 3D convolutional neural network (CNN). The predicted pixel disparity $D^{*}(u,v)$ is calculated by softmax weighting $\sum\limits_{z}softmax(H_{disp}(u,v,z))\times z$ [19]. Based on the fact that disparity and depth are inversely proportional to each other, the convolution operation in the disparity cost volume has disadvantages. The same convolution kernel is applied to a few pixels with small disparity (i.e., large depth) resulting in an easy skipping and ignoring many 3D points. It is more reasonable to run the convolution kernel on the depth grid that produces the same effect on neighbor depths, rather than overemphasizing objects with large disparity (i.e., small depth) on the disparity cost volume. Based on this insight, the disparity cost volume $C_{disp}(u,v,z,:)$ is reconstructed as depth cost volume $C_{dep}(u,v,d,:)$ [16]. Finally, the depth of the pixel is calculated through a similar weighting operation as mentioned above.

The sparse LiDAR points are projected onto the 2D image as the ground truth depth map $D_{gt}$. The depth loss is constructed by minimizing the depth error:

$D^{*}$ represents the predicted depth map. $\tau$ represents smooth L1 loss.

Scene flow estimator cannot directly benefit from the generated original pseudo-LiDAR point clouds due to its containing many points with estimation error. Reducing the impact of these noise points on the 3D scene flow estimation is the problem to be solved.

The image convolution process is the continuous multiplication and summation of the convolution kernel in the image space. This operation is flawed for matching points in real-world space. Points that are far apart in 3D space may be close together on a depth map, such as the edge of a building or the edge of a car. This problem cannot be paid attention to by convolution on the image, which leads to incorrect feature representation or feature matching. Convolution on 3D point clouds better avoids that flaw.

The generated pseudo-LiDAR point clouds $\mathcal{PL}_{t}$ and $\mathcal{PL}_{t+1}$ are encoded and downsampled by PointConv [9] to obtain the point cloud features $F_{PL_{t}}=\{F^{t}_{i}|i=1,2,\cdots,n_{1}\}$ and $F_{PL_{t+1}}=\{F^{t+1}_{i}|i=1,2,\cdots,n_{2}\}$. Then the matching cost between point $pl_{t,i}$ and point $pl_{t+1,j}$ is calculated by concatenating the features $F^{t}_{i}$ of $pl_{t,i}$, the features $F^{t+1}_{i}$ of $pl_{t+1,j}$, and the direction vector $pl_{t,i}-pl_{t+1,j}$ [9], where $pl_{t,j}\in\mathcal{PL}_{t}$ and $pl_{t+1,j}\in\mathcal{PL}_{t+1}$. The nonlinear relationship between $pl_{t,i}$ and $pl_{t+1,j}$ is learned by using multilayer perceptron. According to the obtained matching costs, the point cloud cost volume $CV_{pc}$ used to estimate the movement between points is aggregated. The scene flow estimator constructs a coarse-to-fine network for scene flow estimation. The coarse scene flow is estimated by feeding point cloud features and $CV_{pc}$ into the scene flow predictor $Pred_{SF}$. The input of $Pred_{SF}$ is the point cloud features of the first frame of the current level, $CV_{pc}$, scene flow $\mathcal{SF}_{l+1}^{up}$ from the last level of upsampling, and point cloud features from the last level of upsampling. The output is the scene flow and point cloud features of the current level. The local features of the four variables of $Pred_{SF}$ input are merged by using the PointConv [9] layer and the new $N_{l}\times C$ dimensional features are output. The new $N_{l}\times C$ dimensional features are used as input to the multilayer perceptron to predict the current level of scene flow. The final output is the 3D scene flow $\mathcal{SF}_{t}=\{sf_{i}|i=1,2,\cdots,n_{1}\}$ of each point in $\mathcal{PL}_{t}$.

The self-supervised loss is used at each level to optimize the prediction of the scene flow. The proposed network has four unsupervised losses Chamfer loss $\mathcal{L}_{C}$, smoothness constraint $\mathcal{L}_{SC}$, Laplacian regularization $\mathcal{L}_{LR}$, and disparity consistency $\mathcal{L}_{DC}$. $\mathcal{PL}_{t}$ is warped using the predicted scene flow $\mathcal{SF}_{t}$ to obtain the estimated point cloud $\mathcal{PL}_{t+1}^{\omega}$ at time $t+1$. The $\mathcal{L}_{C}$ loss function is designed to calculate the chamfer distance between $\mathcal{PL}_{t+1}^{\omega}$ and $\mathcal{PL}_{t+1}$. The formula is described as:

where $\left\|\cdot\right\|_{2}^{2}$ represents the operation of Euclidean distance. The design of smoothness constraint $\mathcal{L}_{SC}$ is inspired by the a priori knowledge of smooth scene flow in real-world local space,

where $R(sf_{i})$ means the set of all scene flow in the local space around $pl_{i}$. $|R(sf_{i})|$ represents the number of points in $R(sf_{i})$. Similar to $\mathcal{L}_{C}$, the goal of Laplacian regularization $\mathcal{L}_{LR}$ is to make the Laplace coordinate vectors of the same position in $\mathcal{PL}_{t+1}^{\omega}$ and $\mathcal{PL}_{t+1}$ consistent. The Laplace coordinate vector $o(\mathit{pl}_{2}^{\omega})$ of the point in $\mathcal{PL}_{t+1}^{\omega}$ is calculated as follows:

where $R(\mathit{pl}_{2}^{\omega})$ represents the set of points in the local space around $\mathit{pl}_{2}^{\omega}$ and $|R(\mathit{pl}_{2}^{\omega})|$ is the number of points in $R(\mathit{pl}_{2}^{\omega})$. $\overline{o}$ is the interpolated Laplace coordinate vector from $\mathcal{PL}_{t+1}$ at the same position as $\mathit{pl}_{2}^{\omega}$ by using the inverse distance weight. $\mathcal{L}_{LR}$ is described as:

Inspired by the coupling relationship between depth and pose in unsupervised depth pose estimation tasks [21], we propose a disparity consistency loss $\mathcal{L}_{DC}$. Specifically, each point on the first frame image is warped into the second frame by an estimated 3D scene flow, and the disparity or depth values from the warped points and the points in the real second frame should be the same. The disparity consistency loss is specifically described as:

where $D_{t+1}$ represents the depth map at frame $t+1$. $Pj(\cdot)$ means the projection of the point cloud onto the image using the camera internal parameters. $Bl[\cdot]$ means the index of bilinear interpolation. $\mu(\cdot)$ means averaging over the tensor.

The overall loss of the scene flow estimator is as follow:

The loss of the $l$-th level is a weighted sum of four losses. The total loss $\mathcal{L}$ is a weighted sum of the losses at each level. $\Lambda_{l}$ represents the weight of the loss in the $l$-th level.

Table I and table II gives the quantitative results of our method evaluated at sfKITTI [13]. The accuracy of our method is substantially ahead of supervised learning methods FlowNet3 [22], FlowNet3D [8], and FLOT [10]. Compared with the self-supervised learning method [9, 23, 11, 25, 24, 28] based on point clouds, learning 3D scene flow on pseudo-LiDAR from real-world images demonstrates an impressive effect. Compared to PointPWC-Net [9], our method improves over 45% on EPE3D, Acc3DS and EPE2D, and improves over 30% on Acc3DR, Outliers and Acc2D, which is a powerful demonstration that Chamfer loss and Laplacian regularization loss can be more effective on pseudo-LiDAR. Our method without fine-tuning still outperforms the results of Mittal et al. (ft) [11] fine-tuning on sfKITTI.

From the bottom of Table I, the 3D scene flow estimator is trained from scratch on a synthesized dataset [12], LiDAR point clouds [20, 14], and real-world stereo/monocular images [20], respectively. It is obvious that self-supervised learning works best on pseudo-LiDAR point clouds estimated from images. This confirms that our method avoids the limitation of estimating the scene flow on the synthesized data set to a certain extent. It also confirms that our method avoids the weakness of learning point motion from LiDAR data. Furthermore, in the last row of Table I, the model trained on FT3D [12] serves as a prior guide for our method. As we have been emphasizing, pseudo-LiDAR point clouds will stimulate the potential of self-supervised loss to a greater extent.

The accuracy of learning 3D scene flow on the 128-beam LiDAR signals [14] is improved compared to the 64-beam LiDAR signals. According to the sentence according to the quantitative results of Table I and Table II, the proposed framework for learning 3D scene flow from pseudo-LiDAR signals still presents greater advantages. To qualitative demonstrate the effectiveness of our method, some visualizations are shown in Fig. 4. Compared to PointPWC-Net [9], the estimated points by our method are mostly correct points on the Acc3DR metric. From the details in Fig. 4, the point clouds estimated from our method overlap well with the ground truth point clouds, confirming the reliability of our method. Finally, the methods in this paper also show excellent perceptual performance in the real world as shown in Figure 5.

We test the runtime on a single NVIDIA TITAN RTX GPU. PointPWC-Net takes about 0.1150 seconds on average to perform a training step while our method takes 0.6017 seconds. We consider saving the pseudo-LiDAR point cloud from the depth estimation and enabling the scene flow estimator to learn the scene flow from the saved point cloud. After saving the pseudo-LiDAR signals, the training time is greatly reduced and achieves the same time consumption (about 0.1150 seconds) as PointPWC-Net.

The edge points of the whole scene and the outliers within the scene are the two factors that we focus on. In Table III, the experiments demonstrate that the choice of a suitable scene range and the elimination of outlier points both facilitate the learning of 3D scene flow. In the ablation study, the proposed disparity consistency loss $\mathcal{L}_{DC}$ is verified to be very effective for learning 3D scene flow. A point in the point cloud whose distance from its nearest point exceeds a distance threshold $d_{max}$ is considered as an outlier, where $d_{max}$ is calculated by Eq. (6). The probability that a point in the point cloud is considered as an outlier is determined by the number of selected points $m$ and the standard deviation multiplier threshold $\alpha$. The experiments in Table IV show the best results for elimination of outlier points when $m$ is 8 and $\alpha$ is 2.