Robust Self-Supervised Extrinsic Self-Calibration

Self-supervised depth and ego-motion learning is defined as the joint optimization of a depth network, which maps a target image $I_{t}$ to a depth map $\hat{D}_{t}$, as well as a pose network, that predicts the relative transformation $\hat{\mathbf{X}}^{t\to c}=\begin{pmatrix}\mathbf{\hat{R}^{t\to c}}&\mathbf{\hat{t}^{t\to c}}\\ \mathbf{0}&\mathbf{1}\end{pmatrix}\in\text{SE(3)}$ from a context $c$ to target $t$ frames. The synthesized target image $\hat{I}_{t}$ is obtained via reprojection [1] using $\hat{D}_{t}$, $\hat{\mathbf{X}}^{t\to c}$, camera intrinsics $\textbf{K}_{t}$, and the context image $I_{c}$. The photometric reprojection error is then calculated as:

$$\mathcal{L}_{p}(I_{t},\hat{I_{t}})=\alpha~{}\frac{1-\text{SSIM}(I_{t},\hat{I_{t}})}{2}+(1-\alpha)~{}\|I_{t}-\hat{I_{t}}\|$$

(1)

Here, $\mathcal{L}_{p}(I_{t},\hat{I_{t}})$ is the photometric reprojection loss [1, 16], and the SSIM term measures the structure similarity of two images [32]. This reprojection error provides the photometric consistency constraint we use as the self-supervised training objective. In addition, to promote smoothness in the predicted depth map, a regularization term is added to the above reprojection error [9, 16]:

$$\mathcal{L}_{s}(\hat{D_{t}})=|\delta_{x}\hat{D_{t}}|e^{-|\delta_{x}I_{t}|}+|\delta_{y}\hat{D_{t}}|e^{-|\delta_{y}I_{t}|}$$

(2)

To address our multi-camera setup, we follow [3] and define spatio-temporal constraints via photometric consistency between reprojected camera images across the spatial (i.e. between different cameras) as well as temporal axes (i.e. between different timesteps).

In the discussion below, we index cameras by subscripts and temporal indices (video frames) by superscripts. Also, pose refers to the motion of the vehicle, while extrinsics refers to the poses of cameras on the rig with respect to a vehicle coordinate frame. Note that we chose Euler angles as a representation for extrinsics because of their simplicity, but other parameterizations are possible.

First, we will review the pose consistency loss proposed in [3] which assumes calibrated extrinsic, and then extend it to our self-calibration setting. Given a camera $i$, the pose network predicts its transformation $\hat{\mathbf{X}}^{t\to\tau}_{i}$ from the current frame $t$ to a temporally adjacent frame $\tau$ (usually $\tau\in\{t-1,t+1\}$). Assuming that all cameras are rigidly attached to the vehicle, their motion should be consistent when observed from the vehicle coordinate system, i.e. the motion of the vehicle estimated from $t\to\tau$ in camera $i$ should be the same as the motion observed in camera $j$ (given that the cameras are rigidly attached). Given the known extrinsics $\mathbf{X}_{i}$ and $\mathbf{X}_{j}$, respectively, we can transform the vehicle pose estimated using the pose network predictions from camera $i$ ($\hat{\mathbf{X}}^{t\to\tau}_{i}$) into camera $j$ using:

$$\tilde{\mathbf{X}}_{i}^{t\to\tau}=\mathbf{X}_{j}^{-1}\mathbf{X}_{i}\hat{\mathbf{X}}_{i}^{t\to\tau}\mathbf{X}_{i}^{-1}\mathbf{X}_{j}$$

(3)

The above transformation can be used to transform pose predictions from all cameras into the coordinates of camera $1$, which is the label we give to the forward-facing camera. Given that there is inconsistency in the pose predictions, FSM [3] defines a pose consistency loss as the weighted sum of pair-wise translation and rotation errors:

$$\begin{split}\mathcal{L}_{pcc}(\hat{\mathbf{X}}^{t\to\tau}_{1},\tilde{\mathbf{X}}_{j}^{t\to\tau})=\sum_{\tau}\sum_{j=2}^{N}\{\alpha_{t}\mathcal{L}_{t}(\hat{\mathbf{t}}^{t\to\tau}_{1},\tilde{\mathbf{t}}^{t\to\tau}_{j})\\ +\alpha_{r}\mathcal{L}_{R}(\hat{\mathbf{R}}^{t\to\tau}_{1},\tilde{\mathbf{R}}^{t\to\tau}_{j})\}\end{split}$$

(4)

where $\hat{\mathbf{X}}^{t\to\tau}_{j}=\begin{pmatrix}\mathbf{\hat{R}}_{j}^{t\to\tau}&\mathbf{\hat{t}}_{j}^{t\to\tau}\\ \mathbf{0}&\mathbf{1}\end{pmatrix}\in\text{SE(3)}$.

In the above equation, $N$ is the number of cameras, and $\alpha_{t}$ and $\alpha_{R}$ are respectively the translation and rotation weighting parameters. In [3], the squared difference is applied for both $\mathcal{L}_{t}$ and $\mathcal{L}_{R}$ after converting the rotation matrix to Euler angles. For the fully-calibrated setting in [3], this consistency constraint is used to improve the pose as well as depth network predictions.

Pose Consistency for Extrinsics Learning: Here extrinsics refer to the poses of the cameras on the rig with respect to the vehicle coordinate frame, which in our experiments is chosen as the origin of the front camera. Therefore, $\mathbf{X}_{1}$ is $\mathbf{I}$. Note that the camera extrinsics $\mathbf{X}_{i}$ are present in Equation 3 and thus in Equation 4; this can be used as a differentiable constraint on the extrinsic parameters, and we use it to guide extrinsics learning. However, naively learning extrinsics from scratch, guided by Equation 4, causes extrinsics learning to fail because of a property of the pose network: the predicted rotation $\mathbf{\hat{R}}^{t\to\tau}$ is close to the identity matrix when the images are densely sampled and the camera moves in a straight line, which is common in driving scenarios. Next, we describe our initialization technique to alleviate this and enable joint depth, pose, and extrinsics estimation.

Consider this early training scenario where the predicted pose $\hat{\mathbf{R}}^{t\to\tau}_{j}\approx\mathbf{I}$ and $\hat{\mathbf{X}}_{j}=\begin{pmatrix}\mathbf{\hat{R}}_{j}&\mathbf{\hat{t}}_{j}\\ \mathbf{0}&\mathbf{1}\end{pmatrix}\in\text{SE(3)}$, plugging Equation 3 into Equation 4 will look like:

$$\begin{gathered}\mathcal{L}_{t}(\hat{\mathbf{t}}^{t\to\tau}_{1},\tilde{\mathbf{t}}^{t\to\tau}_{j})\approx\mathcal{L}_{t}(\hat{\mathbf{t}}^{t\to\tau}_{1},\mathbf{\hat{R}}_{j}^{-1}\hat{\mathbf{t}}^{t\to\tau}_{j})\\ \mathcal{L}_{R}(\hat{\mathbf{R}}^{t\to\tau}_{1},\tilde{\mathbf{R}}^{t\to\tau}_{j})\approx\mathcal{L}_{R}(\mathbf{I},\mathbf{\hat{R}}_{j}^{-1}\mathbf{\hat{R}}_{j})\end{gathered}$$

(5)

Here the rotation matrices are represented as Euler angles such that $\mathbf{\hat{R}}(\theta,\phi,\psi)=\mathbf{\hat{R}}_{x}(\theta)\mathbf{\hat{R}}_{y}(\phi)\mathbf{\hat{R}}_{z}(\psi)$. If the ego-motion in the dataset is only composed of forward or backward motion (giving $\hat{\mathbf{t}}^{t\to\tau}_{1}\approx(0,0,const.)^{T}$ ), and the camera $j$ is looking backwards relatively to camera $i$ (which gives $\hat{\mathbf{t}}^{t\to\tau}_{j}\approx(0,0,-const.)^{T}$ ), the above approximation shows that there is no useful gradient for $\psi$. Therefore, we set $(\theta,\phi,\psi)=(\pi,0,\pi)$ only for the initialization of the back-looking camera, and set $(\theta,\phi,\psi)=\mathbf{0}$ for the others.

Note that the approximation in Equation 5 also reveals another limitation of our training procedure: it is difficult to get the gradient of the translation vector $\mathbf{\hat{t}}_{j}$. Thus, we learn the camera translations in a sequential manner, first estimating the rotation and then learning the translation (followed by end-to-end finetuning), please refer to Figure 1 and Section III-C for more details.

Jointly learning depth, pose, and extrinsics from scratch is a highly underconstrained problem, and empirically we have observed that a naive approach fails to converge in the self-supervised setting. Thus, we propose a curriculum schedule composed of different learning modules (depth, ego-motion, and extrinsic parameters) and objectives to minimize (photometric loss and pose consistency loss), designed to enable self-supervised learning in this novel setting. Table I summarizes the different stages of our proposed curriculum learning schedule and Figure 1 shows the extrinsics update flow. Below we describe each stage in detail and quantify their impact in the experiments section.

DDAD [9]. DDAD (Dense Depth for Autonomous Driving) is a dataset with images recorded by $6$ cameras mounted around the vehicle, forming a $360^{o}$ field of view. It has a total of $12650$ training samples (composed of $150$ sequences each with $50$ or $100$ images) and $3950$ validation samples (composed of $50$ sequences each with $50$ or $100$ images). All cameras are synchronized at $10$ Hz, which facilitates self-supervised learning in a multi-camera setting. In all our experiments, images are downsampled to a resolution of $640\times 384$, and evaluated at distances of up to 200 meters. Furthermore, because the training sequences were collected using different vehicles in different cities, we learn instead per-sequence extrinsic parameters, as a way to investigate the performance of our self-calibration method in different scenarios. As reported in  [3], the DDAD dataset has several areas with self-occlusion from the vehicle body, which can harm monocular self-supervised learning. Thus, we use the same self-occlusion masking protocol for training. The ground-truth depth maps are used only for evaluation. In contrast to temporally adjacent image pairs that share $30-95\%$ field of view, the overlap area of spatial- and spatiotemporal-image pairs only shares $5-30\%$.

KITTI [13]. The KITTI dataset is the standard benchmark for depth and ego-motion estimation. We train our model on the Eigen train split [1] containing $22,600$ images per camera and validate on the test split containing $697$ images. Images are reshaped to $640\times 192$, and evaluated at distances up to 80m with the garg crop [1]. Extrinsics are learned according to timestamp date, so five configurations are trained from 2011_09_26 to 2011_10_03. A target image and corresponding image always overlap about $50-70\%$.

Our models were implemented using PyTorch [33] and trained across eight NVIDIA A100 GPUs of $80$ GB per device. Previous and subsequent frames are used as temporal context. Color jittering and left-right image flips are applied as data augmentation for the depth network.

For the Monodepth Pretraining stage, we use the Adam optimizer [34] with learning rate $lr=8\times 10^{-5}$ for both the depth and pose networks, a batch size $b=4$, and train for a total of $15$ epochs. Note that this step is done using pretrained weights, obtained by training standard self-supervised monocular depth and ego-motion networks considering each camera independently, without weak velocity supervision. Empirically, we found that the direct learning of scaled depth and ego-motion creates instability in the training pipeline.

After pretraining the scale-aware model, we begin the extrinsic learning stage with the following parameters: $b=3$ per GPU, the SGD optimizer is used with $lr=0.1$ for the Rotation Estimation stage, the Adam optimizer is used for the Extrinsic Estimation stage with the $lr=2\times 10^{-4}$ for the pose network and $lr=1\times 10^{-3}$ for the extrinsics parameters, and the Adam optimizer is also used for the last joint optimization with $1\times 10^{-5}$ for the depth and pose networks and $1\times 10^{-4}$ for the extrinsic parameters. The number of epochs is set to 10 for the Rotation Estimation and Extrinsic Estimation stages, and 5 for the End-to-end Training stage. For the pose consistency, the translation term is formulated following [3], and the geodesic loss [35] is used as the rotation term. We set $\alpha_{t}=\alpha_{R}=1.0$ in all experiments.

For the network architectures, following [2], we used a ResNet18-based depth and pose networks. Additionally, we introduce “per-camera” extrinsic parameters, which are optimized during training. We evaluate SESC on datasets which consist of sequences collected by vehicles that have the same sensory setup, but slight variations in their extrinsic parameters across sequences due to calibration differences as well as temporal variations. To account for this, we estimate one set of per-camera parameters over the whole dataset, as well as a per-camera residual which is optimized for each sequence.

Extrinsics Calibration Performance. Our results are summarized in Table II, and we note that SESC achieves on average $0.16$m translation error and below $1\degree$ rotation error for extrinsic calibration. We compare SESC with COLMAP [12], a widely used structure-from-motion library.

Since COLMAP predicts intrinsics as well as extrinsics, we present a version of our method, denoted by SESC(I), where we also learn the intrinsics for all the cameras in a self-supervised manner, following the approach described in Fang et al. [8]: the ground-truth intrinsic parameters are replaced with a learnable parameter vector which is optimized jointly with the other networks. The intrinsics are initialized using the image resolution (i.e. $H/2,W/2$), following [8], and we learn one parameter vector for each camera which is optimized across the entire dataset. The Adam optimizer with $lr=1\times 10^{-2}$ is applied to these parameters in all stages except Rotation Estimation. We note that this version of our model also achieves competitive results even in this very challenging setting of unknown intrinsics and extrinsics.

We run COLMAP separately on each DDAD sequence, and we metrically scale COLMAP predictions by adjusting their magnitude based on ground-truth relative translation, to ensure a fair comparison with our method. In all experiments, we run bundle adjustment with the rig_bundle_adjuster option and set each camera model as pinhole, to account for the fact that all cameras are rigidly attached to the vehicle and can be modeled using a pinhole geometry.

Table II shows a quantitative comparison between the two methods. We observed that COLMAP did not converge in all sequences. Furthermore, even when it converged, the translation error produced by COLMAP was very large (up to a few meters) in several sequences. Therefore, for a more nuanced comparison, we also report COLMAP results on the sequences where convergence was reached (for a total of $97$ sequences, marked as COLMAP^*), and those in which the translation error $t$ is smaller than $1$m (for a total of $89$ sequences, marked as COLMAP^**). We note that our method outperforms COLMAP^** on translation error and is competitive on orientation error. Note that both SESC (I) and SESC were able to converge in all sequences from the same initialization, as an indication that it is more robust to different environment conditions. Moreover, the total training time of our method is $\sim 12$ hours for the whole dataset, while COLMAP required $150\sim$ hours ($\sim 1$ hour per sequence if it converged, otherwise more time is required, up to 4 hours).

A qualitative comparison between our calibrated camera extrinsics and COLMAP is shown in Figure 2, alongside our predicted point clouds. These sequences are from significantly different settings: (a) a street scene at low speeds with mostly parked cars, and (b) a highway scene at high speeds with many dynamic objects. The latter is very challenging for COLMAP, as motion blur and the presence of dynamic objects makes it difficult to perform feature matching. Even though our method also degrades in this challenging setting, compared to the simpler former setting, we are still able to reasonably estimate camera extrinsics and depth maps.

Depth Estimation Evaluation. Next, we evaluate SESC on the task of depth estimation, to answer the question of whether our self-calibrated extrinsics can be used as proxy to ground-truth extrinsics and improve depth estimation performance by leveraging cross-camera constraints. These results are reported in Table III, including variations of our depth network trained in different conditions. In Monodepth+v we directly use the backbone from the Monodepth Pretraining stage. SESC $\dagger$ is trained using the same curriculum and number of epochs as SESC but without optimizing extrinsics; we include this baseline to highlight the fact that learning intrinsics via our method leads to improved depth performance, and that the increase is not due to additional training of the depth and ego-motion networks. SESC and SESC $\dagger$ are trained $7$ times using randomly generated seeds and the same hyperparameters, and we also report the mean of these runs. Finally, we also include FSM [3] results obtained using the same ResNet18-based backbone.

These result shows that SESC, which optimizes extrinsics as well as depth and ego-motion networks, improves depth estimation performance compared to the baseline of SESC $\dagger$. Furthermore, these results also show that SESC is competitive with methods that use ground-truth extrinsics, such as FSM [3]. In fact, our model outperforms FSM in several cameras (especially B.Right) although we use the same depth backbones. We hypothesize this is due to a slight miscalibration in the provided ground-truth extrinsics. In summary, our results are evidence that the learned extrinsics generated by SESC can contribute to improvements in depth estimation without requiring additional information or explicit cross-camera calibration [3].

Effect of Curriculum Learning. In Table IV, we investigate the effects of our proposed curriculum strategy on the DDAD dataset to show that it enables the learning of extrinsic parameters without task-specific prior knowledge. In this process, the first stage (Rotation Estimation) promotes the proper initialization of camera orientation from initial values. Afterwards, the second stage (Extrinsic Estimation) significantly reduces translation error while also improving rotation. Finally, the joint optimization further improves accuracy, including depth estimation performance.

By ablating both the Rotation Estimation (see Extrinsic Est. $-$ 2) and Extrinsic Estimation (see E2E $-$ 3) stages we show that SESC effectively decreases translation and rotation error at each training stage.

Table V shows the estimated error of our predicted extrinsic parameters in each learning stage, for the KITTI stereo setup. Even in this “simplest” calibration case, our proposed method improves performance through the proposed stages of our curriculum, achieving a final error of 0.018[m]. The depth estimation performance of the model is shown in Table VI. SESC achieves better results on all metrics compared to the baseline of SESC $\dagger$ (which does not use predicted extrinsics). In addition, SESC is competitive with the current state-of-the-art, even though it relaxes the stereo-camera assumption and does not have access to information about the stereo baseline.