CPO: Change Robust Panorama to
Point Cloud Localization

Instead of focusing on point features, CPO relies on the regional color distribution of images to match the global context between the 2D and 3D measurements. To cope with color distribution shifts from illumination change or camera white balance, we first preprocess the raw color measurements in 2D and 3D via color histogram matching [15, 8, 1]. Specifically, we generate a single color histogram for the query image and point cloud, and establish a matching between the two distributions via optimal transport. While more sophisticated learning-based methods [12, 40, 24] may be used to handle drastic illumination changes such as night-to-day shifts, we find that simple matching can still handle a modest range of color variations prevalent in practical settings. After preprocessing, we compare the intersections of the RGB color histograms between the patches from the query image $I_{Q}$ and the synthetic projections of the point cloud $P$.

The efficient generation of color histograms is a major building block for CPO. While there could be an enormous number of poses that the synthetic projections can be generated from, we re-use the pre-computed histograms from another view to accelerate the process. Suppose we have created color histograms for patches of images taken from the original view $I_{o}$, as shown in Figure 3. Then the color histogram for the image in a new view $I_{n}$ can be quickly approximated without explicitly rendering the image and counting bins of colors for pixels within the patches. Let $\mathcal{S}_{o}=\{S^{o}_{i}\}$ denote the image patches of $I_{o}$ and $\mathcal{C}_{o}=\{c_{i}^{o}\}$ the 2D image coordinates of the patch centroids. $\mathcal{S}_{n}$ and $\mathcal{C}_{n}$ are similarly defined for the novel view $I_{n}$. For each novel view patch, we project the patch centroid using the relative transformation and obtain the color histogram of the nearest patch of the original image, as described in Figure 3. To elaborate, we first map the patch centroid location $c_{i}^{n}$ of $S_{i}^{n}\in\mathcal{S}_{n}$ to the original image coordinate frame,

where $R_{\text{rel}},t_{\text{rel}}$ is the relative pose and $\Pi^{-1}(\cdot):\mathbb{R}^{2}\rightarrow\mathbb{R}^{3}$ is the inverse projection function that maps a 2D coordinate to its 3D world coordinate. The color histogram for $S_{i}^{n}$ is assigned as the color histogram of the patch centroid in $I_{o}$ that is closest to $p_{i}$, namely $c_{*}=\operatorname*{arg\,min}_{c\in\mathcal{C}_{o}}\|c-p_{i}\|_{2}$.

We specifically utilize the cached histograms to generate histograms with arbitrary rotations at a fixed translation. In this case, the camera observes the same set of visible points without changes in occlusion or parallax effect due to depth. Therefore the synthetic image is rendered only once and the patch-wise histograms can be closely approximated by our fast variant with $p_{i}=\Pi(R_{\text{rel}}\Pi^{-1}(c_{i}^{n}))$.

Based on the color histogram of the query image and the synthetic views from the point cloud, we generate 2D and 3D score maps to account for possible changes in the measurements. Given a query image $I_{Q}\in\mathbb{R}^{H\times W\times 3}$, we create multiple synthetic views $Y\in\mathcal{Y}$ at various translations and rotations within the point cloud. Specifically, we project the input point cloud $P=\{X,C\}$ and assign the measured color $Y(u,v)=C_{n}$ at the projected location of the corresponding 3D coordinate $(u,v)=\Pi(R_{Y}X_{n}+t_{Y})$ to create the synthetic view $Y$.

We further compare the color distribution of the synthetic views $Y\in\mathcal{Y}$ against the input image $I_{Q}$ and assign higher scores to regions with high consistency. We first divide both the query image and the synthetic views into patches and calculate the color histograms of the patches. Following the notation in Section 3.1, we can denote the patches of the query image as $\mathcal{S}_{Q}=\{S_{i}^{Q}\}$ and $\mathcal{S}_{Y}=\{S_{i}^{Y}\}$ for each synthetic view. Then the color distribution of patch $i$ is recorded into a histogram with $B$ bins per channel: $h_{i}(\cdot):\mathbb{R}^{H\times W\times 3}\rightarrow S_{i}\rightarrow\mathbb{R}^{B\times 3}$. The consistency of two patches is calculated by finding the intersection between two histograms $\Lambda(\cdot,\cdot):\mathbb{R}^{B\times 3}\times\mathbb{R}^{B\times 3}\rightarrow\mathbb{R}$. Finally, we aggregate the consistency values from multiple synthetic views into the 2D score map for the query image $M_{\text{2D}}$ and the 3D score map for the point cloud $M_{\text{3D}}$. We verify the efficacy of the score maps for CPO in Section 4.3.

For the final step, CPO optimizes sampling loss [20] from selected initial poses, as shown in Figure 1. CPO chooses the candidate starting poses by efficiently leveraging the color distribution of the panorama and point cloud. The space of candidate starting poses is selected in two steps. First, we choose $N_{t}$ 3D locations within various regions of the point cloud, and render $N_{t}$ synthetic views. For datasets with large open spaces lacking much clutter, the positions are selected from uniform grid partitions. On the other hand, for cluttered indoor scenes, we propose to efficiently handle valid starting positions by building octrees to approximate the amorphous empty spaces as in Rodenberg et al. [28] and select centroids of the octrees for $N_{t}$ starting positions.

Second, we select the final $K$ candidate poses out of $N_{t}\times N_{r}$ poses, where $N_{r}$ is the number of rotations assigned to each translation, uniformly sampled from $SO(3)$. We only render a single view for the $N_{t}$ locations, and obtain patch-wise histograms for $N_{r}$ rotations using the fast histogram generation from Section 3.1. We select final $K$ poses that have the largest histogram intersections with the query panorama image. The fast generation of color histograms at synthetic views enables efficient candidate pose selection, which is quantitatively verified in Section 4.3.

Here, we compute the patch-wise histogram intersections for $N_{t}\times N_{r}$ poses where the 2D score map $M_{2D}$ from Section 0.C is used to place smaller weights on image patches that are likely to contain scene change. Let $\mathcal{Y}_{c}$ denote the $N_{t}\times N_{r}$ synthetic views used for finding candidate poses. For a synthetic view $Y\in\mathcal{Y}_{c}$, the weighted histogram intersection $w(Y)$ with the query image $I_{Q}$ is expressed as follows,

Conceptually, the affinity between a synthetic view $Y$ and the query image $I_{Q}$ is computed as the sum of each patch-wise intersection weighted by the corresponding patch $M_{i}$ from the 2D score map $M_{\text{2D}}$. We can expect changed regions to be attenuated in the candidate pose selection process and therefore CPO can quickly compensate for possible scene changes.

We individually refine the selected $K$ poses by optimizing a weighted variant of sampling loss [20], which quantifies the color differences between 2D and 3D. To elaborate, let $\Pi(\cdot)$ be the projection function that maps a point cloud to coordinates in the 2D panorama image $I_{Q}$. Further, let $\Gamma(\cdot;I_{Q})$ indicate the sampling function that maps 2D coordinates to pixel values sampled from $I_{Q}$. The weighted sampling loss enforces each 3D point’s color to be similar to its 2D projection’s sampled color while placing lesser weight on points that are likely to contain change. Given the 3D score map $M_{\text{3D}}$, this is expressed as follows,

where $\odot$ is the Hadamard product and $RX+t$ is the transformed point cloud under the candidate camera pose $R,t$. To obtain the refined poses, we minimize the weighted sampling loss for the $K$ candidate poses using gradient descent [21]. At termination, the refined pose with the smallest sampling loss value is chosen.

We assess the robustness of CPO using the OmniScenes [20] and Structured3D  [38] dataset, which allows performance evaluation for the localization of panorama images against point clouds in changed scenes.

We further demonstrate the wide applicability of CPO by comparing CPO with existing approaches in various scene types and input modalities (raw color / semantic labels). The evaluation is performed in one indoor dataset (Stanford 2D-3D-S [4]), and one outdoor dataset (Data61/2D3D [26]). Unlike OmniScenes and Structured3D, most of these datasets lack scene change. Although CPO mainly targets scenes with changes, it shows state-of-the-art results in these datasets. This is due to the fast histogram generation that allows for effective search from the large pool of candidate poses, which is an essential component of panorama to point cloud localization given the highly non-convex nature of the objective function presented in Section 3.

In this section, we ablate key components of CPO, namely histogram-based candidate pose selection and 2D, 3D score maps. The ablation study for other constituents of CPO is provided in the supplementary material.