EAR-Net: Pursuing End-to-End Absolute Rotations from Multi-View Images

Relative rotation estimation is to calculate the camera rotation between an input pair of images. A traditional strategy for relative rotation estimation contains the following two key stages: feature matching [29, 35, 42, 44] and two-view rotation estimation via the 5-point algorithm [33]. However, it is hard for such a strategy to obtain a reliable estimation on relative rotation in the cases of weak textures or degenerated configurations [19].

Unlike the above strategies, several recent works have adopted deep neural networks (DNN) to estimate relative rotations in an end-to-end manner [31, 18, 10, 5]. For example, Siamese architectures were employed for relative pose estimation in [31, 18]. Zhou et al., [50] gave an analysis on the continuity of rotation representations, and suggested representing 3D rotations in a 6D space, which benefits the training of neural networks. Different from the above regression-based methods [31, 18], Cai et al., [5] cast the relative rotation estimation problem as a classification problem, where each class represents an angle in the range [-180^∘, 180^∘]. However, it is worth noting that the above DNN-based methods are only available for estimating the relative rotation from an input pair of images, but could not estimate the absolute rotations from multi-view images, which are significantly different from our method.

Rotation averaging is a widely studied task in computer vision, which aims to recover the absolute rotations of cameras from a given set of relative rotations. It plays an important role in global SfM methods [24, 7, 14, 12], which is efficient for speeding up camera pose estimation, and could also reduce camera drift [12]. The early method by Govindu, [22] used a Lie-algebra representation to average relative rotations in a least-squares manner. To improve robustness, several works have either focused on using robust loss functions [24, 7] or removing outliers before the optimization [23, 48, 26]. For example, Hartley et al., [24] proposed an $\ell_{1}$ averaging method based on the Weiszfeld algorithm, considering that the $\ell_{1}$ norm was more robust to outliers than the $\ell_{2}$ norm. Chatterjee and Govindu, [7] proposed a generalized framework where different robust loss functions could be seamlessly embedded, and the optimization could be implemented in an efficient iteratively re-weighted least-squares (IRLS) manner in the Lie-algebra representation. Govindu, [23] used a RANSAC-based approach to detect and remove erroneous relative rotations. Zach et al., [48] proposed to filter outlier edges by using loop constraint, i.e., chaining all noise-free transformations along a loop should result in an identity transformation. Lee and Civera, [26] proposed an initialization scheme based on a hierarchy of triplet support, and removed edges that do not conform to an initial solution. Moreover, some works found that a good initialization approach is also helpful for improving the rotation averaging accuracy [7, 14, 12, 21, 26]. Sidhartha and Govindu, [40] extensively analyzed the performance of IRLS-based rotation averaging [7, 12, 26], and further pointed out that the performance of these methods are intimately related to both the robust loss function and initialization approach of the absolute rotations.

Additionally, some learning-based rotation averaging methods have been proposed by utilizing deep networks recently [34, 45, 27]. Here, it has to be explained that these learning-based rotation averaging methods take relative rotations as inputs. However, different from the above methods, the proposed EAR-Net recovers the absolute rotations directly from the original images.

As shown in Figure 2, the proposed EAR-Net takes a set of multi-view images as inputs, and it aims to output the corresponding absolute rotations. EAR-Net consists of the epipolar confidence graph construction module and the confidence-aware rotation averaging module. In the proposed EAR-Net, the referred operations (including relative rotation estimation, rotation averaging, etc.) for estimating absolute camera rotations are combined into one stage, and relative camera rotations are only considered as intermediate features, but not as final prediction results. It is noted that no matter an end-to-end manner or a multi-stage manner is employed to learn relative rotations, the learned relative rotations from different pairs of images generally have different noise levels. Hence, given an input set of images, the epipolar confidence graph construction module is designed to learn the relative rotations and their confidences that indicate whether the corresponding relative rotations are accurate enough, and then a weighted epipolar confidence graph is constructed according to the obtained relative rotations and confidences. Based on the constructed epipolar confidence graph, the confidence-aware rotation averaging module is designed to estimate the absolute rotations. We will introduce the two modules, the loss function and training strategy in detail in the following subsections.

Given an arbitrary set of $N$-view images $\{\mathcal{I}_{k}\}_{k=1}^{N}$ with the height $H$ and width $W$, i.e., $\mathcal{I}_{k}\in\mathbb{R}^{3\times H\times W}$, the epipolar confidence graph construction module is designed to simultaneously learn the relative camera rotations and their corresponding confidences, respectively. It consists of a feature encoder, a pairwise feature aggregation unit, and a dual-branch decoder:

Then, the obtained 4D correlation volumes are reshaped as 3D feature maps with size $\mathbb{R}^{HW/256\times H/16\times W/16}$, and are passed into a residual block and an average pooling layer to obtain a feature vector (namely the pairwise feature) with the size of 512.

Specifically, as shown in Figure 4, the rotation branch firstly adopts two fully connected layers and a softmax function to obtain the discrete distribution $p_{\alpha}$, $p_{\beta}$, $p_{\eta}$ for roll, pitch, and yaw respectively. Then, the values of roll, pitch, and yaw are computed according to the expectation operation in Eqn. 2. Finally, given the three Euler angles, the corresponding rotation matrix could be straightforwardly obtained.

Confidence branch: This branch consists of three fully connected layers and a sigmoid activation function. It maps each pairwise feature vector into a scalar (confidence) in the range of [0, 1], which is expected to reflect the degree of confidence on whether the estimated relative rotation in the above rotation branch is accurate enough. This is to say, when the confidence branch is trained, it tends to predict large confidence scores for reliable relative rotations, and small confidence scores for unreliable relative rotations.

Once both relative rotations $\bm{\mathrm{R}}_{ij}$ and their corresponding confidences $c_{ij}$ are obtained through the above two branches, the epipolar confidence graph $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$ which is a weighted graph is constructed as follows: Vertex $i\in\mathcal{V}$ represents the $i$-th camera with an unknown absolute rotation $\bm{\mathrm{R}}_{i}$; Edge $(i,j)\in\mathcal{E}$ represents the relative rotation $\bm{\mathrm{R}}_{ij}$ between the $i$-th and $j$-th images, which is weighted by the confidence $c_{ij}$.

Given the constructed epipolar confidence graph $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$ from the above module, the confidence-aware rotation averaging module aims to recover the absolute rotations in an iterative and differentiable manner, which allows the gradient to back-propagate.

Many hand-crafted loss functions (such as the Cauchy loss, Geman-McClure loss, etc.) are adopted in other methods [7, 26], which are manually designed according to noise distribution in data [34]. In contrast, our approach explores direct learning of the loss function from data. It should be pointed out that these hand-crafted robust loss functions could also be used seamlessly in our framework. However, we will show in Section 4.6 that our learning-based CAL gives a significantly better result.

To minimize the confidence-aware loss, we propose the Confidence-Aware Initialization (CAI) approach and the Confidence-Aware Optimization (CAO) algorithm, which will be introduced in the following.

Let $\bm{\mathrm{I}}\in\mathbb{R}^{3\times 3}$ be the identity matrix, $\{\bm{\tilde{\mathrm{R}}}_{i}\}_{i\in\mathcal{V}}$ be the set of current estimate of absolute rotations, $\{\bm{\mathrm{R}}_{ij}\}_{(i,j)\in\mathcal{E}}$ be the set of predicted relative rotations from the ECGC module, $\{\Delta\mathfrak{r}_{i}\}_{i\in\mathcal{V}}$ be the set of update amount we need to estimate, $\{\Delta\bm{\mathrm{b}}_{ij}\}_{(i,j)\in\mathcal{E}}$ be the set of residuals computed as $\Delta\bm{\mathrm{b}}_{ij}=\log(\bm{\tilde{\mathrm{R}}}_{j}^{\mathrm{T}}\bm{\mathrm{R}}_{ij}\bm{\tilde{\mathrm{R}}}_{i})$. Then, for edge $(i,j)$, we have:

Stacking all $\bm{\mathrm{B}}_{ij}$, $\Delta\bm{\mathrm{b}}_{ij}$, $\Delta\mathfrak{r}_{i}$ along the column, we obtain $\bm{\mathrm{B}}\in\mathbb{R}^{\frac{3N(N-1)}{2}\times 3N}$, $\Delta\bm{\mathrm{b}}\in\mathbb{R}^{\frac{3N(N-1)}{2}\times 1}$, $\Delta\mathfrak{r}\in\mathbb{R}^{3N\times 1}$ respectively. Then minimizing the CAL in Eqn. 3 is equivalent to solving the following problem:

where $\bm{\mathrm{W}}$ is a diagonal matrix constructed as follows:

The solution to Eqn. 5 is $\Delta\mathfrak{r}=(\bm{\mathrm{B}}^{\mathrm{T}}\bm{\mathrm{W}}\bm{\mathrm{B}})^{-1}\bm{\mathrm{B}}^{\mathrm{T}}\bm{\mathrm{W}}\Delta\bm{\mathrm{b}}$. Then the absolute rotations are updated as $\bm{\tilde{\mathrm{R}}}_{i}\leftarrow\bm{\tilde{\mathrm{R}}}_{i}\exp{(\Delta\mathfrak{r}_{i})},\forall i\in\mathcal{V}$, where ‘$\exp(\cdot)$’ is the mapping from Lie algebra to its Lie group.

Here, we introduce the training strategy and loss function used in the proposed EAR-Net as follows:

The solution could be obtained by the existing method [30]. Then the final loss function is computed as:

The above procedures are fully differentiable, thus the gradient could be computed automatically using existing deep-learning libraries. It should be pointed out that there is no direct constraint for the predicted confidences themselves. Instead, the only supervision signal for the confidences comes from the final predicted absolute rotations, i.e., the confidences are adjusted automatically in order to give a better absolute rotation estimation. Intuitively, a large confidence score should be assigned to the reliable relative rotation, and a small confidence score should be assigned to the unreliable relative rotation. Please see the analysis in Section 4.6.

As noted from Figure 2, the input of the network is a set of $N$ images at each training step, and both the memory and computational costs of EAR-Net have to be dependent on the number $N$ accordingly in principle. With the increase of $N$, the memory and computational cost would increase significantly.

To address the above issue, we generally set the image number $N$ to be a small integer (In our low-cost server, $N$ is always set to 7) during the training stage, and design the following manner to deal with large-scale scenes where hundreds and thousands of images are involved at the testing stage.

Specifically, for a testing large-scale scene, the feature maps are firstly extracted batch-by-batch from all the testing images via the feature encoder in Figure 2, and then they are temporarily stored. Next, an edge-by-edge processing strategy is used to obtain the relative rotations and confidences:

For each edge of an arbitrary image pair in the scene graph, we load the corresponding two feature maps into the GPU memory at each time. The feature maps are then processed by the PFA unit to obtain the pairwise feature vector, which is further decoded into the relative rotation and its confidence subsequently. Since only two feature maps are processed at each time, the memory cost would not increase with the total number of images. Once all relative rotations and the confidences within the scene graph are obtained, we employ the CARA module to obtain the final absolute rotations.

Basic setup: We train and test EAR-Net on ScanNet [15]. We follow the official data split for training and testing. For the training/testing split, we sample 150/15 sets of images with size 7 in each scene. This results in 226950/1485 sets of images for training/testing ¹¹1Since there are not enough image sets in the scan ‘scene00718_00’, in the test split, only 99 scans are indeed used for testing in total. The detailed sampling strategy is provided in the Appendix A..

Large-scale setup: To further evaluate the performance of EAR-Net on large-scale scenes, we conduct experiments on the full scans of ScanNet. Specifically, we evaluate the model on all the scans which contain more than 3000 images. This results in 22 large-scale scenes in total.

Cross-dataset setup: We further evaluate the generalization abilities on the DTU [1] and 7-Scene datasets [39]. We train the model on ScanNet and evaluate on the other two. For the DTU dataset, we sample 100 image sets of size 7 from each scan, resulting in 2200 image sets for testing. For the relatively small 7-Scene dataset, we evaluate on the full set. We sample 20 image sets with size 7 from each scan, resulting in 920 image sets in total.

EAR-Net is implemented in PyTorch. Firstly, the feature encoder, the PFA unit and the rotation branch of the dual-branch decoder are pretrained for 30 epochs with a learning rate of $5\times 10^{-4}$ and a batch size of 20 using the Adam optimizer. Then, we initialize the confidence branch of the decoder so that it outputs a constant value of 0.5 for all pairwise rotations. This could be achieved by dividing the logits (before the sigmoid activation in the last layer of the confidence branch) by a large constant. Next, the whole model is trained with a learning rate of $1\times 10^{-4}$ and a batch size of 8 using the Adam optimizer. We perform $T=3$ iterations of confidence-aware optimization. In total, the model is trained for 50 epochs.

In this setup, we train and test the EAR-Net and the comparative methods on ScanNet [15]. We compare EAR-Net with some typical multi-stage methods as listed in Table 1. Specifically, for the relative rotation, we adopt the following methods: 1) two detector-matcher-based methods with the 5-point algorithm [33]: SIFT [29]-MNN (mutual nearest neighbor matcher), SuperPoint [16]-MNN; 2) two end-to-end matcher-based methods LoFTR [42], ASpanFormer [8] with the 5-point algorithm; 3) two end-to-end relative rotation estimation methods Reg6D [50] and ExtremeRotation [5]. Then, once the relative rotations are obtained, the three state-of-the-art solvers (IRLS-$\ell_{\frac{1}{2}}$ [7], HARA [26], and the learning-based RAGO [27]) for rotation averaging are implemented respectively to calculate the absolute rotations.

Table 1 lists the numbers of the successfully recovered rotations from the total 1485 testing image sets from ScanNet [15] by all the methods in its right-most column, and reports the corresponding results obtained from these successful images. As seen from this table, the comparative methods [29, 16, 36, 8, 42, 49] sometimes fail to register all image sets, possibly due to (1) an insufficient number of inlier matches and (2) the outlier relative rotation filtering step in HARA [26]. EAR-Net does not only recover all the testing images successfully but also outperforms all the comparative methods significantly under all the five metrics. For example, compared with the method ‘ASpanFormer+HARA’ that performs better than the other comparative methods, EAR-Net achieves a relative reduction of 41.68%(=1-4.03/6.91)/50.95%(=1-2.06/4.26) in mean/median error. These results demonstrate the priority of the proposed end-to-end method EAR-Net over these multi-stage methods.

In this subsection, we conduct experiments on the 22 large-scale scenes from ScanNet [15]. It is noted that the four 5-point algorithm based methods (SIFT, SuperPoint, LoFTR, ASpanFormer) are very slow to evaluate on such large-scale scenes. And among them, ASpanFormer performs much better than the other three 5-point algorithm based methods. Thus, in these methods, we only evaluate the performance of ASpanFormer [8]. In addition, we also evaluate the performance of the regression-based Reg6D [50] and ExtremeRotation [5]. Then, we obtain the absolute rotations with the HARA rotation averaging solver, which demonstrates better performance than the other solvers [7, 27] in most cases. The mean errors, the median errors, and the time costs of these comparative methods in each scene are reported in Table 2. As seen from this table, the 5-point algorithm based method ASpanFormer [8] performs poorer than the regression-based methods Reg6D and ExtremeRotation. This is mainly because the baselines between pairs of cameras are much wider in the large-scale scenes than those in the basic setup, so that it is difficult for AspanFormer to obtain high-accuracy matching points, however, the Reg6D and ExtremeRotation are end-to-end learning-based methods that do not explicitly extract feature points, as indicated in [10, 5]. Moreover, the proposed EAR-Net outperforms the other three comparative methods [5, 7, 26, 27] significantly on all of the evaluated scenes. It is also noted that the EAR-Net has a significantly lower inference time cost in all the evaluated scenes, especially compared to the 5-point algorithm based AspanFormer [8] which generally takes 7-20 hours. The above experimental results demonstrate the effectiveness of our method on large-scale scenes in terms of both accuracy and inference speed.

Besides the above quantitative results, we also visualize the estimated rotations on several scenes with 1000-3000 images from ScanNet, including ‘scene720_00’, ‘scene729_00’, ‘scene764_00’. We compare Reg6D+HARA, ExtremeRotation+HARA as well as the proposed EAR-Net. The results are shown in Figure 6. As seen from this figure, our method aligns significantly better with the ground truth, and at the same time has a much faster speed.

In this subsection, we evaluate EAR-Net and the comparative methods under the cross-dataset setup. Specifically, all the referred methods that are trained on ScanNet (except SuperPoint we use the released model that is trained on CoCo [28] by the author) are further evaluated on DTU [1] and 7-Scene [39], and the corresponding results are reported in Table 3. As seen from this table, EAR-Net also performs best among all the referred methods, consistent with the results in Table 1. These results demonstrate the effectiveness of the proposed EAR-Net under the cross-dataset setup.

This subsection provides ablation studies on ScanNet [15] to evaluate the effect of the following key components:

Firstly, as seen from Table 4, ‘w/o confidence’ causes the mean and median errors to increase by 131.0% and 86.9% respectively, which demonstrates that the learned confidence is important for improving the performance of absolute rotation estimation.

Secondly, we conduct the following experiments to demonstrate the robustness of EAR-Net due to the learned confidence: For each sampled image set with size 7 on ScanNet, we corrupt them by appending another $\{0,2,4,6,8,10\}$ randomly selected images from other scenes with no overlap to each image set. As the number of randomly selected images increases, more and more estimated relative rotations will become outliers, and thus it poses a greater challenge for absolute rotation estimation. The results are reported in Figure 7(a). As seen from this figure, with the increasing amounts of outliers, the performance of the ‘w/o confidence’ variant becomes much poorer, while the full EAR-Net is not sensitive to outliers mainly because low confidences are automatically assigned to the outliers.

Thirdly, we analyze the relationship between the errors of the predicted relative rotations and their confidences. We firstly sample around 230k image pairs from the ScanNet dataset, and compute the relative rotations and their confidences accordingly. Then, we divide the confidences in [0, 1] into 20 groups with equal intervals, and for each group, the mean and median errors of the predicted relative rotations are computed using the ground truth relative rotations. The mean and median errors of the relative rotations in these groups are plotted in Figure 8. As seen from this figure, the relative rotation error tends to drop when the confidence score increases. The error is significantly larger when the confidence score is close to zero, which is possibly because lots of outliers occur near this area.

Moreover, in Figure 9, we visualize a set of images and the errors of the predicted pairwise relative rotations as well as the corresponding confidences. As seen from this figure, EAR-Net tends to predict close-to-zero confidences for image pairs with large rotation errors, e.g., the 1st-6th image pair (the confidence is around 0 and the error is 20.3^∘), and large confidences for image pairs with small errors, e.g., the 3rd-5th image (the confidence is 0.81 and the error is 1.14^∘). Moreover, image pairs with small/large overlap areas generally have small/large confidences, possibly because the model could give a more reliable estimation when image pairs have a larger overlap area. For example, the 3rd-5th image pair has a large overlap area, and the corresponding confidence is around 0.81. The 5th-6th image pair has little overlap, and the corresponding confidence is close to zero. The above results confirm that reliable/unreliable relative rotations tend to be assigned with large/small confidences, as indicated in Section 3.2. In addition, we also visualize the case when image sets are corrupted. Specifically, five images are normally sampled and another five images are randomly selected (noise images) from other scenes with no overlap. Figure 10 shows the corresponding results. As seen from this figure, the confidence scores between the noise images and other images are close to zero, which could alleviate the negative influence of outlier edges via the proposed confidence-aware initialization approach and confidence-aware optimization algorithm.

In addition, we also investigate the effect of the CAI approach for resisting outliers by corrupting the image sets by appending another $\{0,2,4,6,8,10\}$ randomly selected images from other scenes with no overlap to each image set. This setup is the same as evaluating the effect of the confidence for resisting outliers. The results are shown in Figure 7(b). As seen from this figure, as the outlier level increases, EAR-Net maintains a stable performance, while the performance of the ‘w/o CAI’ variant is degraded severely. This indicates that the proposed CAI approach is essential for absolute rotation estimation.

The corresponding results are reported in Table 5. As seen from this table, incorporating the Cauchy and Geman-McClure loss functions leads to improved model performance compared with using the naive $\ell_{2}$ loss, consistent with the observations in [7]. In addition, it is noted that the EAR-Net with the proposed CAL achieves a significantly higher accuracy than all the evaluated model variants with different robust loss functions, demonstrating directly learning the weights from data is more beneficial.

It is noted that when a learning-based technique for predicting absolute rotations from relative rotations (e.g., RAGO [27]) is combined and jointly trained with a learning-based relative rotation estimation technique (e.g., Reg6D [50], ExtremeRotation [5]) from input images, we could straightforwardly obtain an end-to-end method for predicting absolute rotations from input images. Accordingly in this subsection, we train Reg6D$\rightarrow$RAGO (combining Reg6D and RAGO) and ExtremeRotation$\rightarrow$RAGO (combining ExtremeRotation and RAGO) respectively in an end-to-end training manner where only the ground-truth absolute rotations in the training set are used as supervision signals as done in the proposed method, and the corresponding results on the ScanNet dataset [15] are reported in Table 6. For a clear comparison, Table 6 also reports the results of the proposed method as well as the results (that are cited from Table 1) by training Reg6D and RAGO (also ExtremeRotation and RAGO) in a two-stage manner, i.e., the relative rotation estimation method Reg6D (or ExtremeRotation) is trained firstly by utilizing ground truth relative rotations as supervision signals, and then RAGO is trained by utilizing ground truth absolute rotations as supervision signals. Two points could be observed from this table: (i) Both Reg6D$\rightarrow$RAGO and ExtremeRotation$\rightarrow$RAGO perform better than their two-stage counterparts, demonstrating that end-to-end training is also effective for boosting the performance of existing models. (ii) The two end-to-end methods Reg6D$\rightarrow$RAGO and ExtremeRotation$\rightarrow$RAGO perform significantly worse than the proposed method EAR-Net, demonstrating that such a simple combination of existing techniques could not guarantee a competitive performance.