Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching

Since ground panoramas²²2Although we use panoramic images as an example, the correspondence relationships between ground and aerial images also apply to images with limited FoV. project 360-degree rays onto an image plane using an equirectangular projection, and are orthogonal to the satellite-view images, vertical lines in the ground image correspond to radial lines in the aerial image, and horizontal lines correspond approximately to circles in the aerial image, assuming that the pixels along the line have similar depths, which occurs frequently in practice. This layout correspondence motivates us to apply a polar transform to the aerial images. In this way, the spatial layouts of these two domains can be roughly aligned, as illustrated in Figure 2(b) and Figure 2(c).

To be specific, the polar origin is set to the center of each aerial image, corresponding to the geo-tag location, and the $0^{\circ}$ angle is chosen as the northward direction, corresponding to the upwards direction of an aligned aerial image. In addition, we constrain the height of the polar-transformed aerial images to be the same as the ground images, and ensure that the angle subtended by each column of the polar transformed aerial images is the same as in the ground images. We apply a uniform sampling strategy along radial lines in the aerial image, such that the innermost and outermost circles of the aerial image are mapped to the bottom and top line of the transformed image respectively.

Formally, let $S_{a}\times S_{a}$ represent the size of an aerial image and $H_{g}\times W_{g}$ denote the target size of polar transform. The polar transform between the original aerial image points $(x_{i}^{a},y_{i}^{a})$ and the target polar transformed ones $(x_{i}^{t},y_{i}^{t})$ is

By applying a polar transform, we coarsely bridge the projective geometry domain gap between ground and aerial images. This allows the CNNs to focus on learning the feature correspondences between the ground and polar-transformed aerial images without consuming network capacity on learning the geometric relationship between these two domains.

Applying a translation offset along the $x$ axis of a polar-transformed image is equivalent to rotating the aerial image. Hence, the task of learning rotational equivariant features for aerial images becomes learning translational equivariant features, which significantly reduces the learning difficulty for our network since CNNs inherently have the property of translational equivariance [8]. However, since the horizontal direction represents a rotation, we have to ensure that the CNN treats the leftmost and rightmost columns of the transformed image as adjacent. Hence, we propose to use circular convolutions with wrap-around padding along the horizontal direction.

We adopt VGG16 [16] as our backbone network. In particular, the first ten layers of VGG16 are used to extract features from the ground and polar-transformed aerial images. Since the polar transform might introduce distortions along the vertical direction, due to the assumption that horizontal lines have similar finite depths, we modify the subsequent three layers which decrease the height of the feature maps but maintain their width. In this manner, our extracted features are more tolerant to distortions along the vertical direction while retaining information along the horizontal direction. We also decrease the feature channel number to $16$ by using these three convolutional layers, and obtain a feature volume of size $4\times 64\times 16$. Our feature volume representation is a global descriptor designed to preserve the spatial layout information of the scene, thus increasing the discriminativeness of the descriptors for image matching.

When the orientation of ground and polar-transformed aerial features are aligned, their features can be compared directly. However, the orientation of the ground images is not always available, and orientation misalignments increase the difficulty of geo-localization significantly, especially when the ground image has a limited FoV. When humans are using a map to relocalize themselves, they determine their location and orientation jointly by comparing what they have seen with what they expect to see on the map. In order to let the network mimic this process, we compute the correlation between the ground and aerial features along the azimuth angle axis. Specifically, we use the ground feature as a sliding window and compute the inner product between the ground and aerial features across all possible orientations. Let $F_{a}\in R^{H\times W_{a}\times C}$ and $F_{g}\in R^{H\times W_{g}\times C}$ denote the aerial and ground features respectively, where $H$ and $C$ indicate the height and channel number of the features, $W_{a}$ and $W_{g}$ represent the width of the aerial and ground features respectively, and $W_{a}\geq W_{g}$. The correlation between $F_{a}$ and $F_{g}$ is expressed as

where $F(h,w,c)$ is the feature response at index ($h$, $w$, $c$), and $\%$ denotes the modulo operation. After correlation computation, the position of the maximum value in the similarity scores is the estimated orientation of the ground image with respect to the polar-transformed aerial one.

When a ground image is a panorama, regardless of whether the orientation is known, the maximum value in the correlation results is directly converted to the $L_{2}$ distance by computing $2(1-\max([F_{a}*F_{g}](i))$, where $F_{a}$ and $F_{g}$ are $L_{2}$-normalized. When a ground image has a limited FoV, we crop the aerial features corresponding to the FoV of the ground image at the position of the maximum similarity score. Then we re-normalize the cropped aerial features and calculate the $L_{2}$ distance between the ground and aerial features as the similarity score for matching. Note that if there are multiple maximum similarity scores, we choose one randomly, since this means that the aerial images has symmetries that cannot be disambiguated.

During the training process, our DSM module is applied to all ground and aerial pairs, whether they are matching or not. For matching pairs, DSM forces the network to learn similar feature embeddings for ground and polar-transformed aerial images with discriminative feature representations along the horizontal direction (i.e., azimuth). In this way, DSM is able to identify the orientation misalignment as well as find the best feature similarity for matching. For non-matching pairs, as it is the most challenging case when they are aligned (i.e., their similarity is larger), our DSM is also used to find the most feasible orientation for a ground image aligning to a non-matching aerial one, and we minimize the maximum similarity of non-matching pairs to make the features more discriminative. Following traditional cross-view localization methods [5, 10, 15], we employ the weighted soft-margin triplet loss [5] to train our network

where $F_{g}$ is the query ground feature, $F_{a^{{}^{\prime}}}$ and $F_{a^{\ast^{\prime}}}$ indicate the cropped aerial features from the matching aerial image and a non-matching aerial image respectively, and $\left\|\cdot\right\|_{F}$ denotes the Frobenius norm. The parameter $\alpha$ controls the convergence speed of training process; following precedents we set it to $10$ [5, 10, 15].

We use the first ten convolutional layers in VGG16 with pretrained weights on Imagenet [4], and randomly initialize the parameters in the following three layers for global feature descriptor extraction. The first seven layers are kept fixed and the subsequent six layers are learned. The Adam optimizer [6] with a learning rate of $10^{-5}$ is employed for training. Following [18, 5, 10, 15], we adopt an exhaustive mini-batch strategy [18] with a batch size of $B=32$ to create the training triplets. Specifically, for each ground image within a mini-batch, there is one matching aerial image and $(B-1)$ non-matching ones. Thus we construct $B(B-1)$ triplets. Similarly, for each aerial image, there is one matching ground image and $(B-1)$ non-matching ones within a mini-batch, and thus we create another $B(B-1)$ triplets. Hence, we have $2B(B-1)$ triplets in total.

We carry out the experiments on two standard cross-view datasets, CVUSA [21] and CVACT [10]. They both contain $35,532$ training ground and aerial pairs and $8,884$ testing pairs. Following an established testing protocol [10, 15], we denote the test sets in CVUSA and CVACT as CVUSA and CVACT_val, respectively. CVACT also provides a larger test set, CVACT_test, which contains $92,802$ cross-view image pairs for fine-grained city-scale geo-localization. Note that the ground images in both of the two datasets are panoramas, and all the ground and aerial images are north aligned. Figure 4 presents samples of cross-view image pairs from the two datasets.

Furthermore, we also conduct experiments on ground images with unknown orientation and limited FoV. We use the image pairs in CVUSA and CVACT_val, and randomly rotate the ground images along the azimuth direction and crop them according to a predetermined FoV. The constructed test set with different FoVs as well as our source code are available via https://github.com/shiyujiao/cross_view_localization_DSM.git.

We first investigate the location estimation performance of our method and compare it with the state-of-the-art on the standard CVUSA and CVACT datasets, where ground images are orientation-aligned panoramas. In Table 1, we present our results on the CVUSA dataset with the recall rates reported in other works [10, 15, 5, 12, 2]. We also retrain existing networks [5, 10, 15] on the CVACT dataset using source code provided by the authors. The recall results at top-1, top-5, top-10 and top-1% on CVACT_val are presented in Table 2, and the complete r@$K$ performance curves on CVUSA and CVACT_val are illustrated in Figure 5(a) and Figure 5(b), respectively.

Among those comparison methods, [20, 21, 18] are first explorers to apply deep-based methods to cross-view related tasks. CVM-NET [5] and Siam-FCANet34 [2] focus on designing powerful feature extraction networks. Liu & Li [10] introduce the orientation information to networks so as to facilitate geo-localization. However, all of them ignore the domain difference between ground and aerial images, thus leading to inferior performance. Regmi and Shah [12] adopt a conditional GAN to generate aerial images from ground panoramas. Although it helps to bridge the cross-view domain gap, undesired scene contents are also induced in this process. Shi et al. [15] propose a cross view feature transport module (CVFT) to better align ground and aerial features. However, it is hard for networks to learn geometric and feature response correspondences simultaneously. In contrast, our polar transform explicitly reduces the projected geometry difference between ground and aerial images, and thus eases the burden of networks. As is clear in Table 1 and Table 2, our method significantly outperforms the state-of-the-art methods by a large margin.

In this section, we test the performance of our algorithm and other methods, including CVM-NET [5] and CVFT [15], on the CVUSA and CVACT_val datasets in a more realistic localization scenario, where the ground images do not have a known orientation and have a limited FoV. Recall that Liu & Li [10] require orientation information as an input, so we cannot compare with this method.