Image Retrieval using Multi-scale CNN Features Pooling

The most straightforward approach is to use the activations of fully connected or convolutional layers as descriptors, using the networks as feature extractors. AlexNet FC6 has been used in [25], outperforming local features approaches for instance retrieval in several standard datasets. In [3] the performance of different AlexNet layers and the effects of PCA have been evaluated. More recent approaches use max-pooled activations from convolutional layers [24, 2, 33].

CNN features can be aggregated using techniques like Bag-of-Words, applied to local convolutional features as in [15], VLAD, applied to global features as in [31] and to local patches as in [6, 31], or using Fisher Vectors, e.g. applied to localized local feature maps derived from objectness detectors as in [29]. Component-wise max-pooling of CNN features computed on object proposals has been used in [26]. The approach used to compute CNN features in these methods may have an impact on the computational performance: the approaches used in [6, 25] require to compute CNN features on a large number of sub-patches, a problem that is reduced in [29, 31] where object proposals and “dense sampling” from max-pooling of convolutional layers are used. As a result, faster pooling approaches were introduced. Regional maximum activation of convolutions (R-MAC) aggregation [28] consider a set of squared regions at different scales, and collects the maximum response in each channel. These descriptors are sum-pooled to create the final R-MAC descriptor. Hashing of CNN features, either global [5, 16] or local, based on objectness scores [30], have been used to speed-up image retrieval tasks.

In this class of methods CNN models are fine-tuned on a training set, learning better representations or aggregations of features, and allowing to extract features in an end-to-end manner through a single pass of the model. Typically this results in an improved performance w.r.t. methods based on CNN feature extraction [7, 22].

In [1] has been proposed a layer called NetVLAD, pluggable in any CNN architecture and trainable through back-propagation, that is inspired by the commonly used VLAD representation. This allows to train end-to-end a network, obtaining state-of-the-art results in image retrieval tasks using an aggregation of VGG16 convolutional activations. Simultaneous learning of CNN and Fisher Vector parameters using a Siamese network and contrastive loss has been proposed in [17].

Both the two current state-of-the-art approaches [8, 23] follow an end-to-end approach, one using a three-stream Siamese network with triplet loss and the other using a two-stream Siamese network with contrastive loss.

In [8] an end-to-end learned version of R-MAC descriptor is presented, along with a triplet mining procedure to efficiently train a three streams Siamese Network using triplet loss. In this approach, a region proposal network selects the most relevant regions of the image, where local features are extracted, in three scales of the input images.

In [23] a trainable Generalized-Mean (GeM) pooling layer is proposed, along with learning whitening, for short representations. Two stream Siamese network is trained using contrastive loss. The authors use structure-from-motion information and hard-matching examples for CNN training, and use up to 5 image scales to extract features.

Our proposed method shares similarity with all of these approaches, but in addition to our proposed pooling and triplet mining, it has important subtle differences that increase performance of the resulting system. Differently from [6, 29, 31, 31] our method is fully trainable end-to-end; differently from [31] multiple scales and only one convolutional layer are used; differently from [6] the VLAD aggregation is performed contemporarily at all the scales, and differently from [29] there is no use of region proposals. Differently from [1], our input to the NetVLAD layer is not directly convolutional activations, but the concatenation of two max-pooled sets of activations.

Convolutional features are max-pooled using a $2\times 2$ and $3\times 3$ (both using stride=1) process, so to obtain representations at finer and larger detail. For each location of the two partitions the $f$ activation maps are collected, creating a $1\times 1\times f$ “column feature” (as defined in [32]). This process, shown in Fig.2, is akin to dense grid-based sampling of SIFT descriptors [9]. Sets of column features are concatenated, to provide a multi-scale descriptor of the image.

All the local CNN features are then aggregated using a NetVLAD [1] layer. The activations of this layer are used as a descriptor of the content of the image. The NetVLAD layer is initialized with a K-Means clustering¹¹1In the experiments we performed it on MirFLICKR25K dataset http://press.liacs.nl/mirflickr/mirdownload.html. As in [1] for NetVLAD we use $K=64$, resulting in a 32k-D representation.

The approach can be applied in principle to any CNN network. In the following experiments we have tested VGG16, as it is commonly used in many competing methods and comparisons. An overview of the method is shown in Fig. 3. The figure shows that we have used the penultimate convolutional layer in the $5^{th}$ block, since initial experiments have shown that using the last layer led to a reduced performance.

In this work we use a ranking loss based on triplets of images; the idea is to learn a descriptor so that the representation of relevant images is closer to the descriptor of the query than that of irrelevant ones. The design of the network is shown in Fig. 1: the weights of the convolutional layers of the CNN network and the NetVLAD layer are shared between the streams, since their size is independent of the size and aspect ratio of the images.

At training time the network is provided with image triplets. Given a query image $Q$ with descriptor $q$, a positive image $P$ with descriptor $p$, a negative image $N$ with descriptor $n$, a distance $d()$ (squared Euclidean distance) and a scalar $\alpha$ that controls the margin, the triplet loss used is $L=max(\alpha+d(q,p)-d(q,n),0)$. $\alpha$ is set to $0.1$ as in [1].

An issue that may arise with this approach is due to the sampling of the triplets: e.g. a random approach may select triplets that do not incur in any loss and thus do not improve the model. We note that triplets may have different impact on the learning depending on the difficulty they pose. Some examples may be well separable if they are from different objects and may be easily learnt. In the contrary, similar but different objects may be challenging to be separated correctly. We may classify triplets as:
easy triplets: $d(q,p)<d(q,n)+\alpha<d(q,n)$ do not really improve the model;
semi-hard triplets: $d(q,p)<d(q,n)$ but $d(q,p)+\alpha>d(q,n)$ - these are more useful than easy triplets but may not add enough information;
hard triplets: $d(q,n)<d(q,p)$ - they produce a high loss.

The algorithm shown in Alg. 1 generates semi-hard and hard triplets (with a 0.5 probability, line 14) with the following logic:
case A: searches the index $j$ for the first negative w.r.t. query. If the index is not the first then the index of the positive sample is $j-1$ (line 15), resulting in a semi-hard triplet.
case B: otherwise, searches the index of the first positive after the first negative, resulting in a hard triplet (lines 18-19).
case C: this deals with extremely hard triplets, e.g. due to strong changes in visual content like zoom or very different point of views of the same scene (see Fig. 3.2), so that positives are very far from the query, i.e. the index of the first positive is farther than a threshold $t$ (line 21). In this case triplets are discarded, since they may lead to overfitting or poor generalization.

The number of classes $k$ used in Alg. 1 is $512$, the $mining\_batch\_size$ is $2048$. The procedure select the triplets so that they belong to different classes (line 28), yielding on average $250$ triplets and returned as mini batches composed by $24$.

Training of the network is performed using Google Landmark V2 dataset²²2https://github.com/cvdfoundation/google-landmark. In particular we use the train split of the “cleaned” version³³3https://www.kaggle.com/confirm/cleaned-subsets-of-google-landmarks-v2 presented in [18], that contains 1,580,470 images and 81,313 labels. The mining process is performed every 8 iterations, to account for the fact that descriptors may change greatly, especially during the initial training. The network has been trained using the Adam [13] optimizer, with a starting learning rate of $10^{-5}$, decreased to $10^{-6}$ after few epochs. The training images have been resized to resolution $336\times 336$, regardless to the original aspect-ratio.

We test our approach on three standard datasets: i) Oxford5k dataset [20], ii) Paris6k dataset [21], and iii) INRIA Holidays dataset [10]; the standard evaluation protocol for these datasets is mean average precision (mAP). To be comparable with most CNN-based methods evaluations we manually correct the orientation of the images on the Holidays dataset, evaluating on the corrected images.

In the experiments reported in Tab. 1, we evaluate the effects of the first contribution of this work, i.e. using two max-pooling to obtain multi-scale features before the NetVLAD layer. Results show that using both $2\times 2$ and $3\times 3$ pooling improve the performance. A single resolution image is used as input. It must be noted that all the results improve upon the standard NetVLAD pooling [1] reported in Tab. 3, showing the benefit of the two-step local CNN features aggregation.

Different resolutions may provide different clues regarding the appearance of objects in the scene. Hence, we extract and combine features at different resolutions, improving the performance of the multi-scale pooling. In the experiments reported in Tab. 2 we evaluate using different image resolutions at test time, evaluating the best combination on multiple datasets. Images are resized to $224\times 224$, $336\times 336$, $504\times 504$ and $768\times 768$ pixels, regardless of aspect ratio. The multi-resolution column reports the sizes used. In all these experiments multi-resolution pooling is used. Results show that image multi-resolution improves the performance. It is interesting to note that even the worst performing combination, i.e. without multi-resolution, the proposed method has better results than competing state-of-the-art approaches (see Tab. 3).

In these experiments we evaluate the proposed method with current state-of-the art methods on all three datasets. Results are reported in Tab. 3; all the methods reported in the table use VGG networks. Results of our method have been obtained using multi-resolution (224 + 336 + 504) and power normalization.