Cross-View Image Retrieval - Ground to Aerial Image Retrieval through Deep Learning

We focus on the formulation of CVIR problem for CrossViewRet dataset $\mathcal{D}$ without losing generality of the topic. Dataset $\mathcal{D}$ contains two subsets: ground-view images $\mathcal{D}_{g}$ and aerial-view images $\mathcal{D}_{a}$. In ground-to-aerial retrieval for the given query image $\mathrm{x}^{g}\in\mathcal{D}_{g}$ we aim to retrieve the set of all the relevant images $\mathcal{D}_{rel}$, where $\mathcal{D}_{rel}\subset\mathcal{D}_{a}$. Similarly, this problem could also be formulated for aerial-to-ground search and retrieval by replacing query image with $\mathrm{x}_{a}$ and search data as $\mathcal{D}_{g}$. For this purpose, we assume a collection of $n$ instances of ground-view and aerial-view image pairs, denoted as $\mathrm{\Psi}=\{{({\mathrm{x}_{i}}^{a},{\mathrm{x}_{i}}^{g})}\}_{i=1}^{n}$, where $\mathrm{x}_{i}^{g}$ is the input ground-view image sample and $\mathrm{x}_{i}^{a}$ is the input aerial-view image sample for the $i$th instance. Each pair of instances${({\mathrm{x}_{i}}^{a},{\mathrm{x}_{i}}^{g})}$ has been assigned a semantic label $\mathrm{y}_{ji}$. If $i$th instance belongs to $j$th class, $y_{ji}=0$, otherwise $y_{ji}=1$.

Representation learning also termed as ”Indexing” in CVIR refers to learn two functions for dual-view images containing same class information: $\mathbf{u}_{i}=f(\mathrm{x}_{i}^{g};\mathrm{\phi}_{g})\in\mathcal{R}^{d}$ for the ground-view image and $\mathbf{v}_{i}=f(\mathrm{x}_{i}^{a};\mathrm{\phi}_{a})\in\mathcal{R}^{d}$ for aerial-view image, where $d$ is the dimensionality of features in their respective feature spaces. $\phi_{g}$ and $\phi_{a}$ in the above two functions are the trainable weights of the street-view and satellite-view feature learning networks. Feature extraction step for the cross-view image pair is influenced by benchmark deep supervised convolutional neural networks including VGG, ResNet-50, and Tiny-Inception-ResNet-v2 pretrained networks [10]. These networks are selected due to their exceptional performance in object recognition and classification task. Unlike traditional Siamese network, here two separate feature learning networks (without weight sharing policy) are employed for extracting features of street and satellite view images. Features acquired through this technique implicitly retain the class information of the images irrespective of their visual viewpoint. Although, these representations might not be projected in the combined feature space for both views still they share same dimensional footprint and could be compared in an embedding space through matching. Figure. 2 (left side) shows the overall indexing procedure in detail.

Features of the cross-view image pair are matched either through distance computation, metric learning, or deep networks with specialized loss functions. Traditionally, matching techniques employ distance computation method of the paired data $(\mathbf{u}_{i},\mathbf{v}_{j})$. For instance, Euclidean distance for feature embeddings of this paired data could be computed as

where ${\|.\|}_{2}$ denotes L2-norm operation. In distance metric learning especially contrastive embedding, a loss function implemented on top of point-wise distance operation, is minimized to learn the association of similar and dissimilar data pairs.It is mathematically computed as

where $\ell_{ij}\in{0,1}$ indicates the labels of the paired data, $0$ representing similar pair and $1$ otherwise. $h(.)=max(0,h)$ is hinge loss function and $D(\mathrm{\Psi})$ is taken from (1). $\alpha$ is used to penalize the dissimilar pair distances for being smaller than this predefined margin using hinge loss in the second part of (2). Similarly, Mahalanobis distance between the cross-view image pair features is computed as

where $\vec{x}_{i}$ and $\vec{z}_{j}$ are two points from the same distribution which has covariance matrix ${\rm\bf C}$. The Mahalanobis distance is the same as the Euclidean distance if the covariance matrix becomes the identity matrix. variation in each component of the point.

For each of the these matching measure if the retrieval score comes out to be less than the given threshold (say 0.5), the feature pair is categorized as similar and dissimilar otherwise. For image retrieval top $k$ images are visualized as relevant to the query image as shown in Figure 2.

The idea of transforming images from feature space to embedding space could be applied by incorporating a deep learning model technically called as deep metric learning network (DML) [16, 11]. We in this research propose a residual deep metric learning architecture optimized with the well known binary cross-entropy loss.

In this research a new dataset CrossViewRet has been developed to evaluate and compare the performance of DeepCVIR framework. Previous cross-view image datasets are deficient in that they (1) lack class information about the content of the image; (2) were originally collected for cross-view image geolocalization task with coupled images; (3) were specifically acquired for the purpose of autonomous vehicles therefore they do not include any images having off-street locations. CrossViewRet comprise of images containing 6 classes including freeway, mountain, palace, river, ship, and stadium with 700 high resolution dual-view images for each class. The satellite-view images are collected from the benchmark NWPU-RESISCS-45 dataset, while respective street-view images of each class are downloaded from Flickr image dataset¹¹1www.flickr.com using Flickr API [2]. The downloaded street-view images are then cross checked by human annotators and images with obvious visual descriptions of the classes are selected for each class. The spatial resolution of satellite-view images is $256\times 256$ and the street-view images are of variable sizes; however, they have been resized before employing for experimentation. The dataset has been made public for future use ²²2https://cvlab.lums.edu.pk/category/projects/imageretrieval.

CrossViewRet is a very complex dataset. Unlike existing cross-view dataset [13, 5] which contain ground and aerial images of the same location. We, on the other hand, do not constraint the images to be of same geo-location. Rather, we focus on visual contents in the scene regardless of any transformation in the images, weather conditions, and variation in day and night time in the scene. As shown in Fig.1, the ground view in sample image of mountain class contains snow whereas its target aerial view image does not. Similarly, the ground view images of palace and stadium class are taken during night and aerial view contains day time drone images. However, the aerial view in river example is satellite image which is totally different than top view drone images.

We use two independent networks for feature learning and embedding learning. In case of feature learning VGGNet, ResNet, and Inception-ResNet-v2 with pre-trained ImageNet weights are fine-tuned. Two independent sub-networks have been employed for learning the discriminating class-wise features of both the views. The architecture of the proposed DML network has been explained in section 3.4. The standard 80/20 train-validation splitting criteria was used for CVIR dataset to fine-tune and train all the feature networks and variants of DML networks respectively. Query images used for evaluation were randomly taken from the validation split of the data.

Deep learning networks have been trained on a Nvidia RTX 2080Ti GPU in Keras. For training feature networks, we employ Stochastic Gradient Descent (SGD) with initial learning rate 0.00001 and a learning rate decay with patience 5. For DML network, ADAM with initial learning rate of 0.001 has been used. Early stopping criteria of 15 epochs has been used to halt training for all the networks.

We evaluated the performance of cross-view image retrieval with not only the standard measures of Precision, Recall, and F1-Score but also evaluated Average Normalized Modified Retrieval Rank (ANMRR), Mean Average Precision (mAP), and P@K (read as Precision at K) [9]. P@K is the percentage of queries which the ground truth image class are in one of the first K retrieved results. Here we only employ P@5 measure for our analysis.

Various deep features have been previously used for the task of same-view image retrieval; however, view-invariant features of multi-modal images plays a pivotal role in CVIR. Table 1 shows that although Inception-ResNet-v2 may outperform the VGGNet and ResNet on ImageNet challenge yet it failed to extract the most optimal features for cross-view image matching. In addition, the performance of various distance computation methods illustrates that the problem is more complex and could not be solved by linear distances i.e. Euclidean or Contrastive loss embedding. Figure 4(a) also confirms the improvement of learning behavior in term of percentage validation accuracy.

The proposed DeepCVIR architecture involves the contribution of DML network which assists the learning process by efficient learning of the embedding space to discriminate similar and dissimilar pairs. However, to evaluate the learning routine of the DML network we tried variants of DML with the single, double and triple combination of the proposed residual blocks termed as S-DeepCVIR, D-DeepCVIR, and T-DeepCVIR, respectively.

Our proposed S-DeepCVIR framework performs equally well on both Str2Sat and Sat2Str tasks. Results in Table 3 shows that although the average ANMRR values remain comparative for all the variants of DeepCVIR architecture still S-DeepCVIR with VGG features achieves minimum average ANMRR of $0.025$ and maximum mAP score.

T-SNE plot is a very effective tool to visualize the data in two dimensional plane for better analysis. We adapted this approach to witness and validate the contribution of DML in transforming features to embedding space. Figure 5(a,f) shows that image features are distributed among the whole region of the plot and hence it is very difficult to measure the correspondence among same and different feature just by using a linear distance. DML separates them into distinguishable clusters.

Although no class information was explicitly provided to the network during training still it successfully clustered the similar pairs into six different classes. It is also observed from the figures that use of different activation functions and multiple residual blocks does not contribute to improvement of the overall result.