ConDo: Continual Domain Expansion for Absolute Pose Regression

Given an image $\mathbf{I}\in\mathbb{R}^{H\times W\times C}$, APR (Kendall, Grimes, and Cipolla 2015) learns a function $\mathbf{t},\mathbf{r}=f(\mathbf{I}|\boldsymbol{\theta})$ parametrized by the neural network weights $\boldsymbol{\theta}$ that maps $\mathbf{I}$ to the camera position $\mathbf{t}\in\mathbb{R}^{3}$ and orientation $\mathbf{r}\in\mathbb{R}^{4}$ in a pre-defined scene $\Omega$. $\Omega$ is defined by a set of training images $\mathcal{S}^{\Omega}=\{\mathbf{I}^{\Omega}_{i}\}_{i=1}^{N}$ with known poses $\mathcal{P}^{\Omega}=\{\mathbf{t}^{\Omega}_{i},\mathbf{r}^{\Omega}_{i}\}_{i=1}^{N}$. The function $f$ is a neural network commonly comprised of a feature extractor $g$ and a regressor $h$, i.e., $f=h\circ g$ where $g$ extracts the image level feature and $h$ projects the extracted feature to $\mathbf{t}$ and $\mathbf{r}$. Conventional APR frameworks train models on $\mathcal{S}^{\Omega}$ and $\mathcal{P}^{\Omega}$ with the regression loss (Kendall and Cipolla 2017):

$$\mathcal{L}(\mathbf{I},\mathbf{t}^{*},\mathbf{r}^{*})=\|\mathbf{t}-\mathbf{t}^{*}\|e^{-s_{t}}+s_{t}+\|\mathbf{r}-\frac{\mathbf{r}^{*}}{\|\mathbf{r}^{*}\|}\|e^{-s_{r}}+s_{r},$$

(1)

where $\mathbf{t}$ and $\mathbf{r}$ are the predicted pose on $\mathbf{I}$, $(\mathbf{t}^{*}$, $\mathbf{r}^{*})\in\mathcal{P}^{\Omega}$ are the ground truth, and $s_{t}$ and $s_{r}$ are learnable parameters to balance the position and orientation losses. After training, the APR model is deployed to the environment for inference.

Due to the scene-dependent nature, the deployed APR model cannot generalize well to images that have highly different poses or scene conditions compared to the training data $\mathcal{S}^{\Omega}$. The key idea of Continual Domain Expansion (ConDo) is to continually update the APR model using the unlabeled data seen naturally after model deployment, so that the model can generalize to more novel poses and scene conditions over time by simply running in the environment.

As shown in Fig. 2, ConDo starts from the model $\boldsymbol{\theta}$ trained on $(\mathcal{S}^{\Omega},\mathcal{P}^{\Omega})$. After model deployment, the (potentially multiple) clients, which use the newest model $\boldsymbol{\theta}$ to perform localization, upload (asyncronously) the observed images to the server. The server collects all newly received images at time step $k$. These images are added to the pool of unlabeled data $\Delta=\{\mathbf{I}_{j}^{\Delta}\}_{j=1}^{M}$ and the current model $\boldsymbol{\theta}_{k}$ is updated asynchronously from the clients using $\mathcal{S}^{\Omega}\bigcup\Delta$, with limited computation that is much less than model re-training. $\boldsymbol{\theta}_{k}$ is re-deployed to the client after the update is finished.

In the main experiment of Sec. 5, we impose the constraint so that the compute for the pre-exectued model training plus all update rounds of ConDo is the same as training one APR model from scratch on $\mathcal{S}^{\Omega}\bigcup\Delta$. This ensures that each ConDo update round uses much less compute and time compared to model re-training, so that it can be applied to life-long scenarios. We also experiment with various fixed computation budgets to validate the effectiveness of ConDo in applications with different resource limits.

To expand the generalization domain of APR without forgetting, we uniformly sample images from $\mathcal{S}^{\Omega}\bigcup\Delta$ to form a training batch during the model update. Though unsupervised domain adaptation (UDA) methods (Chen et al. 2021; Nejjar, Wang, and Fink 2023) have been proposed for standard image classification and regression, empirically (Sec. 5.2) they are not effective for APR. To generate effective supervision on unlabeled data $\Delta$, we opt for a distillation-based approach. Inspired by the fact that scene-independent methods (Arandjelovic et al. 2016; Sarlin et al. 2019; Von Stumberg and Cremers 2022), though slower and more memory consuming during inference, are much more robust than APR to novel poses and scene conditions, we distill the knowledge from these methods to APR using $\Delta$, so that the inference model can still maintain the memory and computation efficiency. Specifically, given a scene-independent method $f_{\text{teacher}}(\cdot)$, and a batch of data $\mathcal{B}^{\Omega}\bigcup\mathcal{B}^{\Delta}$ sampled from $\mathcal{S}^{\Omega}\bigcup\Delta$, the training objective is

$${\underset{\boldsymbol{\theta}}{\text{minimize}}\frac{\underset{\mathbf{I}^{\Omega}\in\mathcal{B}^{\Omega}}{\sum}\mathcal{L}(\mathbf{I}^{\Omega},\mathbf{t}^{*}_{\mathbf{I}^{\Omega}},\mathbf{r}^{*}_{\mathbf{I}^{\Omega}})+\underset{\mathbf{I}^{\Delta}\in\mathcal{B}^{\Delta}}{\sum}\mathcal{L}_{\text{distill}}(\mathbf{I}^{\Delta},f_{\text{teacher}})}{|\mathcal{B}^{\Omega}|+|\mathcal{B}^{\Delta}|},}$$

(2)

where $\mathbf{t}^{*}_{\mathbf{I}^{\Omega}},\mathbf{r}^{*}_{\mathbf{I}^{\Omega}}$ are the ground truth pose of $\mathbf{I}^{\Omega}$. We choose HLoc (Sarlin et al. 2019) as the default $f_{\text{teacher}}$, with scene map built on $(\mathcal{S}^{\Omega},\mathcal{P}^{\Omega})$ (See Table.5 for the robustness of ConDo with other teachers). We set $\mathcal{L}_{\text{distill}}=\mathcal{L}(\mathbf{I}^{\Delta},f_{\text{teacher}}(\mathbf{I^{\Delta}}))$, i.e., substituting the output of $f_{\text{teacher}}$ into Eq. (1). As shown later in Sec. 5, this simple yet effective loss is sufficient to approach the performance of training with ground truth poses on $\Delta$, and is robust to the choice of $f_{\text{teacher}}$. Meanwhile, distilling knowledge on data from new domains not only benefits the performance on the same domain, but can also improve the general robustness of APR, especially under scene condition changes.

By default, we assume the server has sufficient storage to maintain all historical data. For applications with limited server storage, we apply reservoir sampling (Rebuffi et al. 2017) to update the replay buffer. Given a sequence of $K$ images and a storage $\mathcal{M}$ sufficient to maintain $N$ images, we push the first $N$ images to the storage as usual. For the $i$-th image where $i>N$, we generate a random integer $\alpha$ between $[1,i]$. If $\alpha\leq N$, we replace the $\alpha$-th image stored in $\mathcal{M}$ with the $i$-th image in the sequence. Otherwise, we drop the $i$-th image. As shown later in Sec. 5.2, ConDo with reservoir sampling is robust to the replay buffer size.

For architectures capable of handling multiple scenes (Shavit, Ferens, and Keller 2021, 2023), ConDo can also be applied when training data of new scenes and the inference data of old scenes arrive sequentially and interchangeably. When training data of a new scene is available during ConDo, we simply add an extra regression head to cope with new scene coordinates and perform normal ConDo training following Eq. (2). The only difference is that now the replay data of ConDo are sampled from all observed scenes. This strategy ensures that APR can handle both data from the same scene and the sequentially revealed multiple scenes, with a minor model parameter increase.

Table 1 and 2 show the main results on benchmarks constructed in Sec. 4 with respectively the scene condition (Office Loop and Neighbourhood) and pose (7Scenes and Cambridge) changes. For each architecture (PN and PT), we show the results of 3 training frameworks:

1. 1.

   Train Only: Normal APR training on the initial data $\mathcal{S}^{\Omega}$. Representing the practical base APR performance.

2. 2.

   ConDo: The proposed ConDo strategy.

3. 3.

   Re-train with GT: Train an APR model from scratch until convergence (infinite computation budget at any time) on both the training and inference data ($\mathcal{S}^{\Omega}\bigcup\Delta$), with the GT label on $\Delta$ provided. This setup estimates the best performance that ConDo can achieve.

See Sec. 5.2 for further comparisons between ConDo and standard UDA methods.

ConDo significantly improved the performance across baseline architectures and datasets. This shows the capability of ConDo to adapt to both scene condition change (Office Loop and Neighbourhood) and data from novel poses (7Scenes and Cambridge), and sequentially learn to localize in multiple scenes (7Scenes and Cambridge). Before ConDo, the mean error of the baselines was much larger than the median error on the inference scans of large-scale datasets. E.g., PT had $42.15m$ mean error vs $6.12m$ median error on Office Loop. This indicates the existence of catastrophically failing predictions (see Fig. 5 for visualizations). After ConDo, not only mean and median errors were reduced significantly (by $23$x and $4$x respectively), but also the difference between them became small. This change shows the significantly improved generalization of ConDo.

The performance difference between ConDo and Re-train with GT was small, even though 1) ConDo only used unlabeled inference data, and 2) used limited compute for model updates on sequentially revealed data. For example, Re-train with GT on Office Loop with PT used $\sim 120h$ to reach the reported performance and performed much worse with less compute (see Sec. 5.2), while each ConDo update round only took $\sim 20h$, i.e., achieving similar accuracy with only $\frac{1}{6}$ of the compute. Note that this difference will further increase over time with more data collected. Interestingly, ConDo performed marginally better Re-train with GT in Office Loop, it was because HLoc we used is very accurate in this dataset, especially in terms of translation (see Table. 5 for details).

Another interesting observation is that whether new data helps the general robustness of an APR model depends on the type of change in the data. For data with scene condition changes (Table 1), learning from more data significantly improved the accuracy not only on the new scans, but also on previously seen scans. This result shows that training on more diverse data with different scene conditions helps to improve the general robustness of APR. On data with pose changes (Table 2), seeing new data may not always help the performance of old ones, even with ground truth labels (Re-train with GT). Appendix. A.5 further analyzes the detailed reason for this phenomenon.

Fig. 5 visualizes the result of PT on Office Loop. Consistent with the conclusion from quantitative results, due to the weather/lighting condition change, the performance of APR dropped significantly on the inference scans, with many severe localization errors, especially in Inference Scan 4. After ConDo, these severe errors completely disappeared, and the localization accuracy not only improved on the inference scans, but also on the training ones (1.87m to 1.22m on the held-out data Train Scan 3). Fig. 6 shows the model accuracy on the held-out data of different scans after each round of ConDo update, which shows that seeing more data during ConDo can improve the general robustness of APR, resulting in a steady accuracy improvement. Note that the accuracy improvement on the training data was not caused by more training iterations in ConDo, since we have ensured that the model has converged during the Train Only phase, i.e., more training iterations without additional data would not help.

This section analyzes the effectiveness of individual ConDo components, and we report results on Office Loop in the format of position ($m$)/orientation (^∘) error.