Towards Viewpoint Robustness in Bird’s Eye View Segmentation

Recent work has drawn attention to the susceptibility of NNs to misclassify when presented with distributions not seen during training. Madan et al. [14] show that both convolutional- and transformer-based classifiers are fooled by small viewpoint changes, and they introduce a search strategy for finding adversarial viewpoints, which leads to misclassifications over 71% of the time. Similarly, [13] shows that small viewpoint changes degrade classification performance, especially when paired with out of distribution (OOD) categories, and demonstrates that increasing the diversity of training data is an effective strategy to mitigate this issue. Do et al. [4] use homography to move images closer to the distribution of training data at inference time. Coors et al. [3] study the impact of viewpoint changes for 2D semantic segmentation for AV, but do not explore 3D tasks. In contrast, we focus on providing a thorough analysis and a solution to the problem of viewpoint robustness for 3D AV perception tasks, focusing on BEV segmentation.

Novel view synthesis (NVS) provides a way to render images from unseen viewpoints of a scene, and thus could be used to improve the robustness of perception models to viewpoint changes. Many methods have been proposed for NVS in recent years [18, 16, 19], many of which are based on Neural Radiance Fields (NeRF) [17]. However, NeRF still faces two challenges that limit its applicability for our use case: (1) getting NeRF to generalize to dynamic scenes, which are common in AV, is an open research problem, and while there is promising work in this direction [28, 22], the setup is often too constrained and simplified to fit the AV problem setting, and (2) NeRF is challenging to scale due to lack of generalizability, so multiple NeRFs must be trained to perform NVS across scenes. While there is work aimed at generalizing NeRF [12], it remains an open problem and current methods are often constrained. Other methods for NVS rely on monocular depth estimation and can generalize across scenes when the depth estimation network is trained on diverse data. We leverage Worldsheet in our work [11], which is described in more detail in Sec. 4.1.

Bird’s eye view (BEV) segmentation — the task of segmenting a scene in the top-down view (BEV) from 2D images — is a useful task for benchmarking AV perception [21, 24, 1]. BEV segmentation requires a 2D to 3D unprojection to predict the position of objects surrounding the ego-vehicle from the BEV perspective. BEV segmentation models usually consist of an image encoder, which extracts the features from images from the camera rig, and a decoder, which uses the image features to predict the objects of interest in the BEV coordinate frame. Existing methods condition on the extrinsics and intrinsics of each camera in different ways. Lift-Splat-Shoot (LSS) [21] and Orthographic Feature Transform (OFT) [25] unproject features into a point cloud according to each camera’s intrinsic and extrinsic parameters. LSS performs sum pooling along each pillar in the map-view, while OFT performs average pooling. Other methods, such as Cross View Transformers (CVT) [29], treat camera intrinsics and extrinsics as a feature, rather than explicitly unprojecting. We use LSS and CVT to conduct benchmarks, since these two methods encompass both convolutional and transformer-based architectures and explicit and implicit geometric representations.

We build off of Worldsheet [11], a recent method for single image NVS of static scenes, extending it to work on complex AV scenes that have dynamic objects and occlusions. While NeRF-type approaches generate impressive NVS results, generalizing to dynamic scenes and across many scenes is still an active area of research. Worldsheet, on the other hand, is able to generalize across scenes, which is why we choose to use it in our work. The goal of Worldsheet is to build a 3D scene mesh, $M$, by warping a $W\times H$ lattice grid onto the scene based on predicted depths and vertex offsets. Given an input image, $I$, a ResNet-50 [9] is trained to predict depth, $z$, and grid offset of each vertex, $V_{(x,y)}$ at each $(x,y)$ in $I$. $z$ and $V_{(x,y)}$ are used to build $M=(\{V_{(x,y)}\},\{F\})$, where $F$ are the mesh faces. A differentiable texture sampler is then used to splat the RGB pixel intensities from the original image onto the mesh’s UV texture map. The pipeline is trained end-to-end on a multi-view consistency loss. Given two views of the scene, an input and a target, the mesh is predicted from the input view and then projected to the target view based on the target camera pose, $\theta_{t}$. The target view is rendered and compared to the GT with L1 and perceptual losses. A pix2pixHD generator inpaints parts of the scene in the generated target view that were not visible in the input. In contrast, we omit the pix2pixHD generator and use lidar depth supervision (LS), SSIM loss [27], automasking (AM) & minimum loss (ML) over neighboring frames [7] to build an NVS model that generalizes to complex, dynamic, AV scenes.

where $V$ are vertices, $F$ are mesh faces, and $T$ is the texture map. We render the meshes built from $I_{n-1}$ and $I_{n+1}$ in $I_{n}$’s viewpoint, forming novel view renderings $\hat{I_{n}}\in(\mathbf{\hat{I}}_{n}^{n+1},\mathbf{\hat{I}}_{n}^{n-1})$ and their corresponding depth maps $\mathbf{\hat{D}}_{n}^{n+1},\mathbf{\hat{D}}_{n}^{n-1}$. We then compute the per-pixel image loss $L_{im}$, where $\mathcal{P}$ is the valid pixel number and $I_{n,i}$ is the $i$-th pixel of $I_{n}$. Different from NeRF, worldsheet applies a single-layer mesh to synthesize novel views. In the discontinuous depth regions(e.g., boundaries), distortion might happen. To make the training more robust, we apply $L_{1}$ and SSIM loss between the GT image $I_{n}$ and the re-rendered image $\hat{I_{n}}$, where we follow the same setting in [7].

We then apply two depth losses, an L1 loss between the predicted depth and GT lidar depth (direct depth loss) and the L1 loss between the predicted depth and ground truth depth after the prediction is projected into the viewpoint of the cameras at frame $n+1$ and $n-1$ (rendered depth loss). As above, we also use minimal loss for depth supervision:

Fig. 5 shows how our method (SSIM, ML, AM, LS) improves depth estimation and NVS compared to Worldsheet. These improvements are quantitatively validated in Table 1.

The focus of our paper is not on NVS quality, but on the impact using NVS generated data can have on the problem of viewpoint robustness in AV. Given a labeled BEV segmentation training dataset, $D_{source}$, of size $N$, we use our NVS method to transform $n$ images from $D_{source}$ to the viewpoint of the target rig, obtaining $D_{target}^{pred}$ of size $n$. This transformation is done by (1) estimating the depth each image, (2) creating meshes, (3) changing the viewpoint of the cameras, and 4) rendering each image in the viewpoint of the target rig. Finally, we construct a new BEV dataset, $D_{final}$ of size $N$, containing the $n$ transformed images from $D_{target}^{pred}$ and $N-n$ images from $D_{source}$. The number of transformed images, $n$, is a hyperparameter and in our experiments we transform 25%, 50%, or 100% of $D_{source}$ to the viewpoint of the target. The reason we do not always transform all $N$ images is the NVS model may introduce other domain gaps; an ablation on this is done in Sec. 6. We train both the NVS model and BEV segmentation model on a real-world dataset, described in Sec. 5.1. An overview of the training pipeline is shown in Fig. 4.

We demonstrate our method by transforming the dataset from the source rig, $D_{source}$, to the viewpoint of six target rigs, training a BEV segmentation model for each, and evaluating the model on simulated data from the target rig. We conduct experiments with a single camera rig. The target rigs include pitch -10^∘, -5^∘, and 5^∘, depth 1.5 m, and height 0.2 m and 0.8 m. Examples of source rig data transformed to each of the target rigs with the NVS model are shown in Fig. 6. We note that, quantitatively, the NVS quality is best for changes in pitch and lowest for large changes in height. Despite lower quality for some transformed viewpoints, we show that the transformed data still leads to significant improvements in BEV segmentation accuracy for each target rig. For each target rig, we train a Cross View Transformers (CVT) model three times, with 25%, 50%, and 100% of $D_{source}$ transformed to the target rig viewpoint. We also train CVT on source rig data for comparison.

We find that our approach of training BEV segmentation models with 25%, 50%, or 100% data transformed into the view of the target rig significantly improves BEV segmentation accuracy compared to training with only data from the source rig, leading to the same level of accuracy as when there is no viewpoint change. Results are shown in Table 2. We report the best IoU from the models trained with 25%, 50%, or 100% transformed data, but note that all top performing models use only 25% or 50% transformed data, and the rest of the training data remains from the source rig. We observe that both baselines also significantly improve the IoU compared to the model trained only on source data, but not as much as our NVS approach. We also compute the IoU of the model trained only on the source rig and tested on synthetic data from the same viewpoint to serve as a reference upper bound for expected performance when there is no viewpoint gap, shown in the first row of Table 2. This upper bound is more reliable than training and testing on simulated data, which results in an average of 35.4% IoU across views due to the lack of domain gap and limited diversity, resulting in visually similar train and test data.

Lastly, we conducted an experiment in which we trained a model on $\frac{1}{2}$ source rig and $\frac{1}{6}$ +5^∘ pitch, $\frac{1}{6}$ +1.5 m depth, and $\frac{1}{6}$ +0.2 m height data, resulting in 0.19 mean test IoU across views in Tab. 2 (0.206 for train views and 0.178 for other views). This result suggests training on multiple views can improve IoU over training only on the target view. Altogether, our results support our hypothesis that using NVS to transform labeled train data from the viewpoint of a source rig to that of a target rig and then training a BEV segmentation model with that data can enable the creation of BEV segmentation models for target rigs without the associated cost of collecting and annotating data from each target rig.