Learning Cross-view Geo-localization Embeddings via Dynamic Weighted Decorrelation Regularization

Imaged-based cross-view geo-localization has been tackled as an image retrieval task. Early works (Bansal et al., , 2011; Lin et al., , 2013; Castaldo et al., , 2015; Saurer et al., , 2016) employ hand-crafted operators to extract discriminative features for cross-view image matching. With the development of deep learning, the convolutional neural network (CNN) has received much research attention in the extraction of image representation. The pioneering CNN-based approach (Workman and Jacobs, , 2015) directly deploys pre-trained AlexNet (Krizhevsky et al., , 2012) to extract features for the cross-view geo-localization. Further, Workman et al., (2015) introduce the information of image pairs as the constraint to fine-tune the pre-trained network and acquire a better performance. Following this line of considering object constraints, Lin et al., (2015) borrow knowledge from face verification and harness the contrastive loss (Hadsell et al., , 2006) to guide the optimization of a modified Siamese Network (Chopra et al., , 2005). Vo and Hays, (2016) discuss the limitation of the Siamese architecture in large-scale cross-view matching and provide a soft-margin triplet loss to improve the geo-localization accuracy. Similarly, Hu et al., (2018) propose a weighted soft-margin ranking loss, which not only improves the matching accuracy but also speeds up the training convergence. Cai et al., (2019) mine hard examples in the training batch to strengthen the penalization of the soft-margin triplet loss. Zheng et al., 2020b suggest that images with the same identification can be classified into one cluster and apply the instance loss (Zheng et al., 2020c, ; Zheng et al., , 2017) as the proxy target to learn discriminative embeddings. Another line of works concentrates on addressing the spatial misalignment problem of cross-view retrieval. CVM-Net (Hu et al., , 2018) employs a shared NetVLAD to aggregate the local feature to minor the visual gap between different viewpoints. Liu and Li, (2019) explicitly encode the orientation information into the image descriptors and boost the discriminative power of the learned features. Shi et al., 2020c first attempt to utilize the optimal transport (OT) theory to close the spatial layout information in the high-level feature. Then Shi et al., (2019) directly resort to the polar transform to align the pixel-level semantic information of cross-view images. DSM (Shi et al., 2020a, ) designs a dynamic similarity matching module to solve the cross-view matching in a limited Field of View (FoV). LPN (Wang et al., , 2021) stresses the importance of contextual information and proposes a square-ring partition strategy to improve the performance of cross-view geo-localization. Without any extra annotations, RK-Net (Lin et al., , 2022) automatically detects salient keypoints to improve the model capability against the appearance changes.

In the early study of human perception, the neuroscientist H. Barlow (Barlow et al., , 1961) suggests that the perception system tends to encode the raw sensory input as the low-redundancy representation in which each component possesses statistical independence. This learning principle guides a number of algorithms in machine learning. Bengio and Bergstra, (2009) support that the decorrelation criterion is useful in the context of data and derive a fast online pretraining algorithm to learn decorrelated features for neural networks. Cogswell et al., (2016) design Decov Loss that motivates the network to learn non-redundant representations and demonstrate that decorrelating representations helps to reduce overfitting of the trained deep networks. Sun et al., (2017) utilize Singular Vector Decomposition (SVD) and reduce the correlation between output units by integrating the orthogonality constraint in CNN training. Thus the final descriptor contains lower redundant information about the sample. In self-supervised learning (SSL), Barlow Twins (Zbontar et al., , 2021) proposes a simple yet effective object function to acquire representations with low redundancy and avoid model collapse. The optimization goal of Barlow Twins is to transform a cross-correlation matrix into an identity matrix. SSL does not require the input data with human annotation, and the cross-correlation matrix is computed from two distorted representations of a sample.

Cross-correlation matrix measures the correlation between two matrices. In particular, for two random matrices $\mathbf{X}=(X_{1},X_{2},\cdots,X_{M})^{T}$, $\mathbf{Y}=(Y_{1},Y_{2}\cdots,Y_{N})^{T}$, where $X_{m},Y_{n}\in\mathbb{R}^{a}$ with $a$ dimensions, the cross-correlation matrix of $\mathbf{X}$ and $\mathbf{Y}$ can be defined as:

A component-wise description is:

where $\mathbb{E\left[\cdot\right]}$ refers to the expectation.

Pearson cross-correlation coefficient matrix is similar to the cross-correlation matrix, and it shows a normalized measurement between two matrices. Differently, the value of each element in the Pearson cross-correlation coefficient matrix is between $-1$ and $1$. The score $1$ and $-1$ denote that two vectors are perfectly correlated and anti-correlated. $0$ means that two vectors are completely unrelated. Based on the definition of the cross-correlation matrix, the Pearson cross-correlation coefficient matrix can be formulated as:

where $\mu_{\mathbf{X}}$ and $\sigma_{\mathbf{X}}$ are the mean and the standard deviation of $\mathbf{X}$, respectively. $\mu_{\mathbf{Y}}$ and $\sigma_{\mathbf{Y}}$ are the mean and the standard deviation of $\mathbf{Y}$, separately. $\mathbb{E\left[\cdot\right]}$ is the expectation.

Barlow Twins (Zbontar et al., , 2021) is a widely-used method for self-supervised learning (SSL). To address the issue of representation collapse, the main contribution of Barlow Twins is introducing a new regularization objective, which can be defined as:

where $\lambda$ is a positive hyper-parameter, and $\phi$ is the cross-correlation matrix between two mini-batch of features. Given two batches of features with $d$ dimensions ($d$ is generally set as 2048), we multiply the feature matrix along with the batch dimension. Thus, the dimension of $\phi$ is $d\times d$. The regularization function $L_{BT}$ encourages every feature channel to be independent of others. Specifically, it impels the diagonal elements $\phi_{ii}$ from the same channel to $1$, while pushing off-diagonal elements $\phi_{ij}$ between different channels to $0$. However, in practice, Barlow Twins meets the optimization problem, especially when facing a typical large matrix (e.g., $2048\times 2048$). The model arrives at the plateau after the majority of channels are converged, and it neglects other still-correlated “hard” channels, compromising the training process.

We adopt ResNet-50 (He et al., , 2016) as the backbone and add a new classifier. The classifier consists of a fully-connected layer (FC), a batch normalization layer (BN), a dropout layer (Dropout), and another fully-connected layer (FC). Notably, the backbone can also be other networks such as VGG16 (Simonyan and Zisserman, , 2015) and Swin Transformer (Liu et al., , 2021). The two-branch baseline consists of three forward phases, i.e., feature extraction, feature aggregation, and feature classification. Specifically, we denote the input images from two platforms as $x_{k}$, where $k\in\{1,2\}$. $1$ denotes the satellite platform, and $2$ refers to the drone or ground platform. We first employ two backbones with shared weights to extract feature maps. Then the global average pooling is deployed to aggregate the information of feature maps into the column vectors $f_{k}$. Finally, we harness a classifier to map vectors $f_{k}$ of different platforms into one shared classification space and acquire the predicted logit vectors $z_{k}$. Meanwhile, the cross entropy function is employed to calculate the instance loss $L_{id}$  (Zheng et al., 2020b, ). The instance loss is a classification loss with a shared classifier $\mathcal{F}_{classifier}$. The above process can be formulated as:

The label $y\in[1,C]$, where $C$ indicates the category number of geographic targets in the training set. $z_{k}(y)$ is the logit score of the ground-truth geo-tag $y$. When inference, we remove the final linear classification layer and extract the intermediate feature $f_{k}$ as the visual representation.

In this work, we introduce a dynamic weighted decorrelation regularization (DWDR) to encourage the network to learn low-redundancy visual embeddings. As shown in Figure 2, DWDR is implemented based on a classic two-branch baseline (Zheng et al., 2020b, ). The two-branch baseline harnesses location classification as the pretext (Zheng et al., 2020b, ; Zheng et al., , 2019) to conduct the cross-view geo-localization task. During training, we employ the symmetric sampling strategy to balance examples between different platforms in a training batch. It is worth noting that the symmetric sampling strategy is a by-product of DWDR.

The optimization objective of DWDR is the Pearson cross-correlation coefficient matrix $\rho$ between $f_{k}$ extracted from images of different platforms. Given two batches of extracted vectors $f_{1}$ and $f_{2}$ of size $b\times 2048$, according to Eq. 4, we can gain the objective matrix $\rho$ with the shape of $2048\times 2048$, where $b$ denotes the batchsize. DWDR aims to spur the network by regressing the objective matrix $\rho$ into a sparse matrix, i.e., an identity matrix. We employ Square Loss to constrain the regression of each element. DWDR can be written as:

where $\lambda$ is a hyper-parameter to balance the diagonal and off-diagonal element weight, $\rho_{ii}$ refers to the diagonal elements of the objective Pearson matrix $\rho$, and $\rho_{ij}$ denotes off-diagonal elements. $\rho_{ii}$ is regressed to $1$, which makes visual embeddings of the same geo-tag invariant for different platforms. $\rho_{ij}$ is regressed to $0$ to make the visual embedding channels independent from each other. $\omega_{1}$ and $\omega_{2}$ are two dynamic weights to prevent the optimization plateau, which depends on the regression score. In this way, the dynamic weight adjusts the influence of the loss and adaptively pays attention to the poorly-regressed elements during training. Considering that each element of the Pearson matrix is in $[-1,1]$, we set $\omega_{1}=\left(\frac{1-\rho_{ii}}{2}\right)^{\gamma_{1}},\omega_{2}=\lvert\rho_{ij}\rvert^{\gamma_{2}}$, $\lvert\cdot\rvert$ denotes the absolute value. In this way, given non-negative focusing parameters $\gamma_{1}$ and $\gamma_{2}$, we ensure $\omega_{1}\in[0,1]$ and $\omega_{2}\in[0,1]$. For elements close to the regression result (well-regressed elements), the assigned dynamic weight is near $0$. Conversely, for elements far from the regression target (poorly-regressed elements), the assigned dynamic weight increases to $1$.

Symmetric sampling strategy. In order to compute the Pearson cross-correlation coefficient matrix, a necessary step is to construct a training batch by acquiring different-platform images with same geo-tags (‘positive pairs’). Here we take the University-1652 dataset (Zheng et al., 2020b, ) as an example. We can first randomly sample the satellite-view image as an anchor and then form the positive pair by finding the drone-view image of the same geo-tag. We call this sampling, which takes the satellite-view image as an anchor, the satellite-view based sampling. If we use the drone-view image as the anchor, we call the strategy the drone-view based sampling. We note that the image number of different platforms is usually different due to the capturing difficulty. For example, we can easily acquire multiple drone images while only having one satellite image. If we apply the satellite-view based sampling during training, it will miss sampling some drone-view images. It is because every time we randomly sample one image from 54 drone-view images with putting back. On the other hand, if we apply the drone-view based sampling, it will contain duplicate geo-tags within a mini-batch, which repeatedly samples the same satellite image. Therefore, we propose a symmetric sampling strategy, which combines the satellite-view based sampling and drone-view based sampling. In particular, we sample two mini-batch by the two strategies respectively and combine them together as the new mini-batch to train the model. This combined strategy ensures the model can “see” all the training data while keeping the category sampling relatively balanced.

Discussion. The proposed method is similar to Barlow Twins considering disentangling the correlation matrix, but is different in the following aspects: First, Barlow Twins (Zbontar et al., , 2021) optimizes the cross-correlation matrix, while our method harnesses the Pearson cross-correlation coefficient matrix. The Pearson matrix is preferable, since it normalizes the element in a limited range of [-1,1], which unifies the element scale within the matrix and prevents overflowing. Second, as shown in Eq. 5, Barlow Twins accumulated the error along the whole matrix $\phi$. However, the dimension of $\phi$ is large, e.g., $2048\times 2048$. As training proceeds, the majority of elements converge, and the network arrives at the plateau, since the loss is accumulated by a vast of elements. The optimization of the remaining minority elements is usually ignored. The proposed DWDR also accumulated the error but with dynamic weights for different elements in Eq 9. We leverage the Pearson matrix, which is normalized in the range [-1, 1], to set the corresponding dynamic weights. The design also limits dynamic weights in [0,1], preventing the weight overflow. Therefore, DWDR can focus on the minority elements, even when majority channels are converged. Compared with Barlow Twins, DWDR encourages the network to make still correlated channels independent throughout the training period.

Optimization. We optimize our network by jointly employing the instance loss and DWDR:

The instance loss $L_{id}$ forces different-platform images with the same geo-tag to be close on the high-level features and pushes mismatched images far apart. At the same time, DWDR motivates the learned visual embeddings with independent channels. Thus the network is able to extract more discriminative features. $\alpha$ is a weight to control the importance of the loss function and the regularization term.

We conduct experiments on three geo-localization datasets, i.e., University-1652 (Zheng et al., 2020b, ), CVUSA (Zhai et al., , 2017), and CVACT (Liu and Li, , 2019).

University-1652 (Zheng et al., 2020b, ) is a multi-view multi-source dataset, including data from three different platforms, i.e., drones, satellites and dash cams. As the name implies, this dataset collects 1652 ordinary buildings of 72 universities around the world. 701 of all 1652 buildings are separated into the training set, and the other 951 builds constitute the testing set. Therefore, build images in the training and testing set are not overlapping. For each building, the dataset contains one satellite-view image, 54 drone-view images and 3.38 ground-view images on average. Since dash cams are hard to acquire enough street-view images for some buildings, the dataset also collects 21,099 common-view images from Google Image as an extra training set. The dataset supports two new aerial-view geo-localization tasks, i.e., drone-view target localization (Drone $\rightarrow$ Satellite) and drone navigation (Satellite $\rightarrow$ Drone).

CVUSA (Zhai et al., , 2017) is a large-scale cross-view dataset, which consists of images from two viewpoints, i.e., the ground view and the satellite view. In the dataset, 35,532 ground-and-satellite image pairs are employed for training, and 8,884 image pairs are provided for testing. Noteworthily, ground-view images are the pattern of panoramas, and the orientation of all ground and satellite images is aligned.

CVACT (Liu and Li, , 2019) is a similar dataset to CVUSA. For the ground-to-satellite task, CVACT also contains 35,532 image pairs for training. Different from CVUSA, CVACT provides a validation set with 8,884 image pairs denoted as CVACT_val. Meanwhile, compared with CVUSA, CVACT possesses a larger test set with 92,802 image pairs named CVACT_test. When evaluated in CVACT_val, a query ground-view panorama only match one satellite image in the gallery. However, for CVACT_test, a panoramic query image may correspond to several satellite images within 5 meters from the ground-truth location.

We follow existing works (Wang et al., , 2021; Shi et al., 2020c, ; Shi et al., , 2019) and mainly employ CVACT_val to evaluate our method when training on CVACT. The image number of training and evaluation in these three datasets is shown in Table 1.

Evaluation protocol. In our experiments, the performance of our method is evaluated by two commonly used metrics, i.e., Recall@K (R@K) and the average precision (AP). R@K refers to the proportion of true-matched candidates in the top-K of the ranking list. The value of AP is measured by the area under the Precision-Recall curve. Higher scores of these two metrics denote better performance of the network.

Our method is performed based on a classic two-branch baseline (Zheng et al., 2020b, ). The baseline adopts a modified ResNet-50 (He et al., , 2016) pre-trained on ImageNet (Deng et al., , 2009) to extract visual features. Specifically, we remove the final classification layer of ResNet-50 (He et al., , 2016). Besides, the stride of the second convolution layer and the down-sample layer in the first bottleneck of the ResNet-50 (He et al., , 2016) stage4 is set from 2 to 1. The input image is resized to $256\times 256$, and the image augmentation consists of random cropping, random horizontal flipping, and random rotation. We employ stochastic gradient descent (SGD) with momentum 0.9 and weight decay 0.0005 to update model parameters. The image number of each platform in a mini-batch is 16. The initial learning rate is 0.001 for the modified ResNet-50 (He et al., , 2016) backbone and 0.01 for the classifier module. The dropout rate in the classifier module is 0.75. We train the two-branch baseline for 120 epochs, and the learning rate is decayed by 0.1 after 80 epochs. There are two trade-off parameters $\lambda$ and $\alpha$ in the loss function. We run a simple search and observe the better results for $\lambda=1.3\times 10^{-3}$ and $\alpha=0.9$. Note that when using Swin-B (Liu et al., , 2021) and VGG16 (Simonyan and Zisserman, , 2015) as backbones, $\lambda=2.0\times 10^{-3}$ and $\lambda=3.9\times 10^{-3}$ are best choices, separately. During testing, we deploy the Euclidean distance to compute the similarities between the query and candidates. Our model is implemented on Pytorch (Paszke et al., , 2019), and all experiments are conducted on a single NVIDIA RTX 2080Ti GPU.

Results on University-1652. As shown in Table 2, we compare our method with lots of competitive methods on University-1652. The compared methods are divided into three groups, i.e., baseline-related methods, methods harnessing contextual information and Transformer-based methods. In the first group, the first line reports results of our two-branch baseline, i.e., “Instance Loss (Zheng et al., 2020b, )”. We can observe that “Baseline + Ours” substantially improves the baseline performance. In the drone-view target localization task (Drone $\rightarrow$ Satellite), the accuracy of R@1 increases from $57.09\%$ to $69.77\%$ ($+12.68\%$), and the value of AP raises from $61.88\%$ to $73.73\%$ ($+11.85\%$). In the drone navigation task (Satellite $\rightarrow$ Drone), the accuracy of R@1 goes up from $73.89\%$ to $81.46\%$ ($+7.57\%$), and the value of AP increases from $58.73\%$ to $70.45\%$ ($+11.72\%$). Meanwhile, the performance of our method also has surpassed other baseline-related methods. In the second group, LPN (Wang et al., , 2021) explicitly takes advantage of contextual information during training. Some methods, e.g., “LPN + USAM (Lin et al., , 2022)” and PCL (Tian et al., , 2021), combined with LPN have yielded better results, and we can also implement our method based on LPN. Specifically, we re-implement LPN by utilizing the symmetric sampling strategy to replace the original random sampling and incorporating the dynamic weighted decorrelation regularization during training. Compared with results of LPN, “LPN + Ours” achieves $81.51\%$ R@1 accuracy ($+5.58\%$) and $84.11\%$ AP ($+4.97\%$) on Drone $\rightarrow$ Satellite and $88.30\%$ R@1 accuracy ($+1.85\%$) and $79.38\%$ AP ($+4.89\%$) on Satellite $\rightarrow$ Drone. The feature expression ability of Transformer is stronger than that of CNN, and both Transformer-based methods (Liu et al., , 2021; Dai et al., , 2021) obtain a better performance than CNN-based methods. We further combine our method with “Swin-B (Liu et al., , 2021)”. “Swin-B” indicates the two-branch baseline applying Swin-B as the backbone. “Swin-B + Ours” on University-1652 achieves the state-of-the-art results, i.e., $86.41\%$ in R@1 accuracy and $88.41\%$ in AP for Drone $\rightarrow$ Satellite and $91.30\%$ in R@1 accuracy and $86.02\%$ in AP for Satellite $\rightarrow$ Drone.

Results on CVUSA and CVACT. Comparisons with other competitive approaches on CVUSA and CVACT are summarized in Table 3. CVUSA and CVACT have a similar data pattern, i.e., the satellite-platform images of aerial viewpoint and the ground panoramas. The polar transform considers the geometric correspondence of two-platform images and transforms the aerial-view image to approximately align a ground panorama at the pixel level. The aligned images help to improve the performance of models. Depending on whether or not the polar transform is harnessed, the compared method can be divided into two columns. The first column reports methods without using polar transform, and methods in the second column employ the polar transform during training and testing. Our method does not employ the polar transform. Experiments on CVUSA and CVACT show phenomena similar to that on University-1652. Our method first outperforms a dual-stream baseline (i.e., the method of Zheng (Zheng et al., 2020b, )) by a large margin, i.e., $31.71\%$ R@1 improvement on CVUSA and $35.56\%$ R@1 raising on CVACT. At the same time, our method exceeds most of existing methods in the first column. In particular, our method obtains $75.62\%$ R@1, $90.45\%$ R@5, $93.60\%$ R@10, and $98.60\%$ R@Top1% on CVUSA, and $66.76\%$ R@1, $83.34\%$ R@5, $87.11\%$ R@10, and $95.10\%$ R@Top1% on CVACT. In experiments of University-1652, we observe that our method can combine with LPN (Wang et al., , 2021) and achieve better results. The same experiments are also carried out on CVUSA and CVACT. There are two versions of LPN (i.e., LPN and LPN^‡) in Table 3. LPN^‡ applies the polar transform and has achieved higher performance. We notice that our approach still yields competitive results when complemented with the LPN^‡. For instance, “LPN^‡ + Ours” boosts the R@1 accuracy from $93.78\%$ to $94.33\%$ on CVUSA and $82.87\%$ to $83.73\%$ on CVACT.

The above experimental results on three cross-view geo-localization datasets suggest two points. One is that our method can be flexibly applied in different cross-view settings. The other is that our method is able to encourage existing approaches to mine more diverse patterns, yielding discriminative features.

To further analyze our method, we design several ablation studies. The ablation studies are mainly based on the drone-view target localization (Drone $\rightarrow$ Satellite) and drone navigation (Satellite $\rightarrow$ Drone) of University-1652 (Zheng et al., 2020b, ).

Analysis of parameters $\gamma_{1}$ and $\gamma_{2}$. The main contribution of our paper is the proposed dynamic weighted decorrelation regularization (DWDR). In DWDR, $\gamma_{1}$ and $\gamma_{2}$ are two key parameters that flexibly adjust the rate at which well-regressed elements of the Pearson cross-correlation coefficient matrix are down-weighted. When $\gamma_{1}=0$ and $\gamma_{2}=0$, DWDR does not apply two dynamic weights and can be viewed as Barlow Twins (Zbontar et al., , 2021). We empirically tune $\gamma_{1}$ and $\gamma_{2}$, and the related results are detailed in Table 4. We first observe that applying one dynamic weight, i.e., only $\gamma_{1}=1$ or only $\gamma_{2}=1$, achieves similar results to Barlow Twins. The limited performance improvement reflects that ignoring poorly-regressed diagonal and off-diagonal elements both induce the optimization plateau. When both $\gamma_{1}$ and $\gamma_{2}$ are set to 1, i.e., using two dynamic weights, we obtain the best results. Specifically, compared with deploying Barlow Twins as regularization (“Baseline + BT”), our method (“Baseline + Ours”) boosts R@1 from $67.91\%$ to $69.77\%$ ($+1.86\%$) and AP from $71.99\%$ to $73.73\%$ ($+1.74\%$) on Drone $\rightarrow$ Satellite, and goes up R@1 from $80.17\%$ to $81.46\%$ ($+1.29\%$) and AP from $68.03\%$ to $70.45\%$ ($+2.42\%$) on Satellite $\rightarrow$ Drone. When $\gamma_{1}=2$ and $\gamma_{2}=2$, the performance gains slightly degrades. A reasonable speculation is that large focusing parameters $\gamma_{1}$ and $\gamma_{2}$ cause the importance of poorly-regressed elements in the optimization process to be excessively reduced as well. To further verify the robustness of selected parameters, we conduct the same experiments in “LPN (Wang et al., , 2021) + Ours” and “Swin-B (Liu et al., , 2021) + Ours” and find the same conclusion. That is, when both $\gamma_{1}$ and $\gamma_{2}$ are set to 1, models achieve competitive results. Therefore, we choose $\gamma_{1}=1$ and $\gamma_{2}=1$ as default focusing parameters of DWDR. All three groups of experiments also support that DWDR is more effective than Barlow Twins for motivating networks to learn low-redundancy visual embeddings.

Effect of our sampling strategy and DWDR. Our symmetric sampling strategy is a combination of the drone-view based sampling and the satellite-view based sampling. To discuss the effectiveness of our sampling strategy, we conduct three groups of experiments under the condition of only changing the sampling strategy. Meanwhile, in each group of experiments, we study the effectiveness of DWDR. The experimental results are shown in Table 5. We observe first that utilizing DWDR alone does not give comparable results to the baseline (“Instance Loss (Zheng et al., 2020b, )”) shown in Table 2. However, when applied in conjunction with Instance Loss, DWDR significantly improves the performance of the network, regardless of the sampling strategy. Experiments within each group verify from the side that DWDR concentrates more on the redundant channels of the embedding itself rather than the distance between cross-view embeddings. Second, when only Instance Loss is harnessed, the drone-view based and the satellite-view based sampling acquire similar results to the baseline (“Instance Loss”) applying random sampling. In contrast, the symmetric sampling strategy obtains the best geo-localization accuracy. Furthermore, the symmetric sampling strategy is also the most competitive in the other two experimental settings, i.e., DWDR alone and Instance Loss plus DWDR. The significant performance increment demonstrates that the symmetric sampling strategy as a by-product is effective.

Effect of the dimension of visual embeddings. We deploy the final visual embeddings with different dimensions in geo-localization to investigate the effect of embedding dimensions on retrieval accuracy. The experimental results of the baseline and “Baseline + Ours” are shown in Table 6. We observe that with the increment of the dimension, both the baseline (Zheng et al., 2020b, ) and “Baseline + Ours” have a persistent improvement since the visual embedding possesses more information capacity. However, the performance of the two methods encounters the bottleneck when the feature dimension is $512$. As the dimension of the feature increases to $1024$, the performance of the baseline decreases significantly, and the performance of “Baseline + Ours” tends to stabilize. The experimental results reflect two aspects from the side. One is that features with too high dimensions are prone to redundant channels, which compromise the geo-localization accuracy of models. The other is that our method can encourage networks to learn low-redundancy embeddings and improve the robustness of the model. In addition, as shown in Figure 3, we notice that when the dimension raises from $64$ to $128$, the baseline achieves a higher growth rate than “Baseline + Ours”. The short-dimensional features with small information capacity limit the performance of models. We speculate that our method allows the model to include more primary discriminative patterns in the limited feature dimension to mitigate the negative effects of insufficient information capacity. Therefore, when the feature dimension increases, our method produces fewer performance fluctuations.

Effect of DWDR under different loss functions. Our baseline applies the instance loss (Zheng et al., 2020c, ; Zheng et al., , 2017) to optimize the network while other loss functions are available. The triplet loss and the soft margin triplet loss are broadly utilized in previous works (Hu et al., , 2018; Liu and Li, , 2019; Shi et al., 2020c, ; Vo and Hays, , 2016). We also evaluate our DWDR by deploying baselines adopting these two loss functions. The margin value of the triplet loss is 0.3, and experimental results are shown in Table 7. We notice that both baselines combined with DWDR gain improved retrieval accuracy on the “Drone $\rightarrow$ Satellite” task and the “Satellite $\rightarrow$ Drone” task of University-1652.

Effect of the intra-view DWDR. Our method applies the cross-view DWDR, in which the Pearson cross-correlation coefficient matrix is computed employing cross-view images. The intra-view DWDR means that the Pearson cross-correlation coefficient matrix of DWDR is calculated employing two distorted images from the same platform generated by different data augmentations. In experiments, the method only utilizing the symmetric sampling strategy is treated as the baseline, and the comparison results are shown in Table 8. We observe that the baseline combined with the intra-view DWDR gains a slight increment. Although the intra-view DWDR also encourages the network to learn independent embedding channels, our cross-view DWDR significantly outperforms applying the intra-view DWDR. It is because the cross-view DWDR is aligned with the cross-view retrieval test setting, which considers embeddings from different platforms for the geo-localization task. It also explains the limited performance increase of applying both intra-view and cross-view DWDR, which relies on the cross-view DWDR.

We visualize some heatmaps generated by the baseline and our method as an extra qualitative evaluation. Figure 4 shows the acquired heatmaps in the drone and satellite platforms of University-1652. Images in University-1652 possess an obvious geographic target. Compared with the baseline (Zheng et al., 2020b, ), our method activates a wider range of geographic target regions. In addition, we show some retrieval results on different datasets (see Figure 5). University-1652 supports two tasks. In the drone-view target localization task, the drone-platform image is the query, and in the drone navigation task, the satellite-platform image is the query. The retrieval results of two tasks are shown in Figure 5 (I) and (II). Figure 5 (III) and (IV) show the retrieval results of the ground-to-satellite localization task on CVUSA and CVACT. Given a randomly selected test query, we notice that the proposed method has successfully retrieved the most relevant results from the candidate gallery.