Construct Informative Triplet with Two-stage Hard-sample Generation

Our framework is shown in Fig. 3, which mainly consists of three parts: the CNN feature extractor (FE), the hard sample generation (HSG) model, and the embedding learning loss functions. Formally, we denote the feature extraction CNN by $F$, which maps the input images $\mathbf{I}$ into embedding samples $\mathbf{X}$. We then use the proposed HSG model to further produce the hard samples $\hat{\mathbf{X}}$ based on the original samples $\mathbf{X}$. Note that the produced temporary hidden samples $\mathbf{X}^{\prime}$ in HSG are also applied in the loss computing process. The input images $\mathbf{I}$, the original embedding samples $\mathbf{X}$, the hidden samples $\mathbf{X}^{\prime}$ and the generated hard embedding samples $\hat{\mathbf{X}}$ share the same ground truth categories $\mathbf{L}=[l_{1},\cdots,l_{n}]$, where $l_{i}\in{1,\cdots,C}$. The goal is to train the deep embedding network $F$ with parameters $\theta_{m}$. The general metric learning approaches train the parameters $\theta_{m}$ through optimizing the following objective function:

We enhance the training procedure by utilizing the synthetic hard samples. Our final deep embedding learning metric is denoted as:

We rewrite (3) as:

where $\mathcal{L}_{org}$, $\mathcal{L}_{class}$ and $\mathcal{L}_{hard}$ are loss terms corresponding to the original deep distance metric loss for $\mathbf{X}$, the category loss based on $\mathbf{L}$ and the deep distance metric loss for the generated hard sample $\hat{\mathbf{X}}$; $w_{o}$, $w_{l}$ and $w_{h}$ are three weighting factors corresponding to the three terms. Although the embedding model $F$ should be optimized on the whole dataset, we discuss the loss computing in a triplet input images $\mathbf{I}=\{I_{a},I_{p},I_{n}\}$ for convenience. In the following, we will detail the objective function (4).

For the input images $\mathbf{I}=\{I_{a},I_{p},I_{n}\}$, the corresponding triplet embedding samples $\mathbf{X}=\{\mathbf{a},\mathbf{p},\mathbf{n}\}$ will be produced by CNN feature extractor $F$. The hard triplet embedding samples $\hat{\mathbf{X}}=\{\hat{\mathbf{a}},\hat{\mathbf{p}},\hat{\mathbf{n}}\}$ are then further generated by passing $\mathbf{X}$ into HSG model. Besides, the temporary hidden embedding samples ${\mathbf{X}^{\prime}}=\{{\mathbf{a}^{\prime}},{\mathbf{p}^{\prime}}\}$ are also obtained in HSG. The deep embedding network $F$ and the HSG model are simultaneously trained in an end-to-end manner. However, in the early stages of training, the deep model focuses on memorizing the simple training data [30], and thus we can decrease the weight for the generated hard samples. Besides, in the early stage, the embedding space does not have an accurate semantic structure [26], and so the quality of the generated hard embedding samples is very low and they can not provide valid information. Thus the use of the hard generated samples will ruin the learning and damage the embedding space structure. To solve this problem, we designed a weighting parameter adaptively adjusting method by using the training loss $\mathcal{L}_{G}$ of the HSG model. The embedding training objective function is represented as:

where $\beta$ and $\phi$ are pre-defined parameters; $\mathcal{L}_{t}$ and $\mathcal{L}_{sm}$ are the triplet metric loss and the softmax loss, respectively; $e^{-\frac{\beta}{\mathcal{L}_{G}}}$, $\phi$, and $(1-e^{-\frac{\beta}{\mathcal{L}_{G}}})$ are three balancing parameters for the three terms: the original metric loss $\mathcal{L}_{org}$, the category loss $\mathcal{L}_{label}$ and the generated hard metric loss $\mathcal{L}_{hard}$. In the early stage of training, ${\mathcal{L}_{G}}$ is very large and thus the weight $e^{-\frac{\beta}{\mathcal{L}_{G}}}$ for $\mathcal{L}_{org}$ is larger than the weight $(1-e^{-\frac{\beta}{\mathcal{L}_{G}}})$ for $\mathcal{L}_{hard}$. As the training proceeds, ${\mathcal{L}_{G}}$ is decreased and thus high quality hard samples will be produced. The hard synthetics can provide information for training, and higher weight $(1-e^{-\frac{\beta}{\mathcal{L}_{G}}})$ is set to highlight the generated hard samples. Note that ${\mathcal{L}_{G}}$ is set as ${\mathcal{L}_{G_{2}}}$ in the HSG model, which will be discussed in Section 5. In (5), $\mathcal{L}_{t}(\mathbf{X})$ and $\mathcal{L}_{t}(\hat{\mathbf{X}})$ are written as (6) and (7).

The category loss is performed based on the softmax loss,

where $l(\mathbf{x})$ is the one-hot encoding vector of the correct class for sample $\mathbf{x}$; $\hat{l}(\mathbf{x})$ denotes the predicted probability vector.

In (5), the only unknown elements are the generated samples $\hat{\mathbf{X}}$ and ${\mathbf{X}^{\prime}}$. In the following section, we will detail the hard sample generation.

The detailed two-stage hard sample generation architecture is denoted in Fig. 4. In the first stage, the hard level of the input anchor-positive pair $\{\mathbf{a},\mathbf{p}\}$ will first be increased by directly manipulating the distances between them. To achieve this, the piecewise linear manipulation (PLM) is designed. Then the generated hard anchor-positive pair $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$ will be processed by generator $G_{1}$ and thus the hard anchor-positive pair $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$ is obtained. However, just conducting label-preserving synthesis is not enough and this may result in mode collapse, as shown in Fig. 5(a). In this case, the generator only produces samples from part modes of the distribution and ignores the other modes.

To alleviate the mode collapse issue and guarantee the diversity of the generated samples, we encourage the one-to-one relationship between the original sample and the generated sample by adding an extra conditional constraint to the generation process. The discriminator $D_{G1}$ needs to have the ability to identify the results of combining the intermediate embeddings with the original and generated embeddings respectively. In the second stage, the generated anchor-positive pair $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$ and original negative sample $\mathbf{n}$ are passed into a generator $G_{2}$ to produce the final hard triplet $\{\hat{\mathbf{a}},\hat{\mathbf{p}},\hat{\mathbf{n}}\}$. To support the adversarial learning, the discriminator $D_{G2}$ is integrated into Stage 2.

In our architecture, the PLM, the conditional synthesis (in Stage 1) and the hard triplet generation (in Stage 2) are three key parts that will be discussed in the following.

In order to make anchor $\mathbf{a}$ and positive $\mathbf{p}$ more difficult, we need to increase the distance between them. Here, we first linearly stretch the embeddings of anchor and positive samples, as shown in Stage 1 of Fig. 1, thus generating a harder anchor and positive pair $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$, as depicted in Fig. 4. We propose the piecewise linear manipulation (PLM) scheme as shown in (9),

where $\lambda$ is a picecwise variable which controls the hard level of the generated anchor-positive pair $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$. The picecwise variable $\lambda$ plays the key role in our PLM scheme and we will discuss it in the following.

During the stretching process, if the value of $\lambda$ is too large, the anchor-positive samples may be stretched into different classes. Even if the distance is increased, the stretched samples will no longer be anchor-positive pair and thus become meaningless. In order to avoid this issue, we need to limit the scope and size of the $\lambda$. Thus in our PLM, we set the piecewise variable $\lambda$ as:

where $d(\mathbf{a},\mathbf{p})$ denotes the distance between the anchor and positive samples; $d_{t}$ is an adaptive threshold. When $d(\mathbf{a},\mathbf{p})$ is already large enough to be greater than $d_{t}$, we use a exponent function ${\alpha}/{e^{d(\mathbf{a},\mathbf{p})-d_{t}}}$. At this situation, the maximum value of $\lambda$ is $\alpha$, and the minimum value is about 0. When the distance between them is less than $d_{t}$, we use a linear function $\alpha+\gamma(1-\frac{d(\mathbf{a},\mathbf{p})}{d_{t}})$. At this situation, the maximum value of $\lambda$ is $\alpha+\gamma$ and the minimum value is $\alpha$. In the calculation of $\lambda$, $d_{t}$ is related to the distance distribution of samples in different datasets, so it is difficult to manually adjust $d_{t}$ in various datasets. In order to automatically select the best hyperparameters $d_{t}$ in each dataset, we proposed an adaptive method by choosing $d_{t}$ as:

where $M$ is the number of anchor-positive pairs, and $d_{t}$ represents the average distance of anchor-positive pairs in the previous epoch.

After piecewise linearly manipulating the input anchor-positive pair $\{\mathbf{a},\mathbf{p}\}$, we obtain hard anchor-positive pair $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$. However, there is no guarantee that $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$ share the same label with $\{\mathbf{a},\mathbf{p}\}$. Next, we will conduct conditional synthesis based on the generated $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$ to produce valid hard anchor-positive pair $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$.

After piecewise linearly manipulating anchor-positive pairs, in order to ensure the generated pairs are still in the same category domain as the original samples, the recent method requires label consistency to avoid generating invalid embeddings [20]. However, the simple constraint may result in a mode collapse problem, which means the generated samples suffering from insufficient diversity, as shown in Fig 5. Thus except for the label-preserving requirement, we try to enhance the relationship between the original sample and the generated sample by adding an extra restriction. Specifically, we require that the generated samples $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$ based on $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$ have the same distribution as the original samples $\{\mathbf{a},\mathbf{p}\}$. Unlike an unconditional generative adversarial network, both the generator $G_{1}$ and discriminator $D_{G1}$ consider the information of the linear-manipulated anchor-positive pair $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$. To summarize, we encourage the one-to-one relationship between the original sample and the generated sample for our conditional synthesis, and thus the diversity is ensured.

When $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$ passed $G_{1}$, they are mapped to $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$. On the one hand, we synthesize anchor-positive pairs that are further apart based on $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$; On the other hand, no matter where the input $x^{*}$ of $G_{1}$ lies in its category space, the output of $G_{1}$ is located in the same category space as the original one by the constraint of label consistency. Our conditional synthesis is achieved by training $G_{1}$ with the following loss function:

where $\eta$ is the balance factor between softmax and adversarial loss; the first term makes sure the generated embeddings have the same labels with the original ones; the second term is adversarial loss with discriminator $D_{G1}$ which aims to let the generated embeddings look real, not the fake ones; the third term guarantees the generated embeddings can maintain the same hard level as the input embedings. In (12), the first term is denoted as:

where $\mathbf{x}^{\prime}$ denotes embeddings from $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$ and $l_{\mathbf{x}^{\prime}}$ is the ground truth label of $\mathbf{x}^{\prime}$; the $\mathcal{L}_{sm}$ denotes the softmax loss; $C_{F}$ is a shared fully connected layer which is used to classify the generated pair. In (12), the second term is denoted as:

where $\mathbf{x}$ denotes embeddings from $\{\mathbf{a},\mathbf{p}\}$, $\mathbf{x}^{*}$ denotes embeddings from $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$ and ”1” represents the real samples. The discriminator $D_{G1}$ takes $\mathbf{x}^{\prime}_{c}$ the result of concatenating $\mathbf{x}^{\prime}$ and $\mathbf{x}^{*}$ along the last dimension as the input. This adversarial loss is used to fool the discriminator $D_{G1}$ by generating real-like samples; the third term in (12) is reconstruction loss $\mathcal{L}_{rec1}$ between $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$ and $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$, which is calculated as:

With the reconstruction loss $\mathcal{L}_{rec1}$, the relative hardness of anchor-positive synthetic pairs in triplets will be guaranteed. In this way, our generative loss $\mathcal{L}_{G1}$ enables generated anchor-positive pairs to have a greater separation distance while remaining within their original category space.

The discriminator $D_{G1}$ is a two-category classifier for identifying whether a given embedding is a real one or a generated one. The training loss function for $D_{G1}$ is formulated as:

where $\mathbf{x}$, $\mathbf{x}^{*}$ and $\mathbf{x}^{\prime}$ denote embeddings from $\{\mathbf{a},\mathbf{p}\}$, $\{\mathbf{a}^{*},\mathbf{p}^{*}\}$ and $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$, respectively. $\mathbf{x}_{c}$ and $\mathbf{{x}^{\prime}_{c}}$, the input embedding of the discriminator $D_{G1}$, are concatenated of $\mathbf{x}^{*}$ and corresponding embedding in the last dimension of the same labeled instance. ”1” and ”0” denote the real sample and the fake sample respectively.

After the adversarial training of $G_{1}$, our hard anchor-positive pairs have met the requirements of label consistency. At the same time, conditional discriminator $D_{G1}$ ensures that the generated pairs do not have mode collapse with random generation. Next, we will use the hard anchor-positive pair $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$ to generate the hard negative, and produce the final hard samples $\{\hat{\mathbf{a}},\hat{\mathbf{p}},\hat{\mathbf{n}}\}$.

We propose a new hard triplet generator $G_{2}$ to generate hard samples. Generator $G_{2}$ is able to map its input $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$ and $\mathbf{n}$ to $\{\hat{\mathbf{a}},\hat{\mathbf{p}},\hat{\mathbf{n}}\}$. While generating a negative sample, the hard negative sample is not only related to the information of negative sample itself, but also to the anchor-positive pair in the triplet. Therefore the inputs of $G_{2}$ are negative sample $\mathbf{n}$ and hard anchor-positive pair $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$ generated in Stage 1. For our hard triplet synthesis, we propose the loss function to train $G_{2}$ as follows:

where $\eta$ and $\mu$ are the balancing factors. The overall objective loss function of $G_{2}$ is composed of four parts: the adaptive reverse triplet loss (ART-loss) $\mathcal{L}_{ART}$, reconstruction loss $\mathcal{L}_{rec2}$, softmax loss $\!\mathcal{L}_{class2}\!$ and adversarial loss $\mathcal{L}_{DG2,adv}\!$. The ART-loss is used to generate the hard negative; the reconstruction loss is applied to keep hard anchor-positive pair from being affected by reverse triplet loss; the softmax loss is used to ensure label consistency for the generated samples; adversarial loss with discriminator $D_{G2}$ is performed to generate samples like the real ones. These four terms are detailed in the following.

The proposed ART-loss $\mathcal{L}_{ART}$ is defined as follows:

The proposed ART-loss is obtained from the triplet loss by reversing the sign in front of the first two terms in (1). The purpose of ART-loss is to make the distance between $\mathbf{a}^{\prime}$ and $\mathbf{n}$ as small as possible until it is less than the distance between $\mathbf{a}^{\prime}$ and $\mathbf{p}^{\prime}$ with a parameter $\tau_{r}$. When $\mathbf{n}$ satisfies this condition, it will be a hard negative sample. In this paper, we propose to use an adaptive parameter $\tau_{r}$ which increases with the training of the network. We impose more stringent restrictions on reverse triplet loss with larger $\tau_{r}$ when the performance of $G_{2}$ is getting better. Thus, we can gradually generate harder and harder negative samples. We set $\tau_{r}$ as a parameter that changes with $\mathcal{L}_{G2}$ as:

where $\nu$ and $\beta$ are constant parameters. As $G_{2}$ trains better, the $\mathcal{L}_{G2}$ will get smaller, and $\tau_{r}$ will increase adaptively to enhance the difficulty of hard negative.

The reconstruction loss $\mathcal{L}_{rec2}$ aims to keep the synthetic anchor-positive pair with a consistent hard-level by decreasing the distances between $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$ and $\{\hat{\mathbf{a}},\hat{\mathbf{p}}\}$, which is denoted as:

The softmax loss $\!\mathcal{L}_{class2}\!$ makes the generated embeddings have the same labels with the original samples, which is calculated as:

where $\hat{\mathbf{x}}$ denotes embeddings from $\{\hat{\mathbf{a}},\hat{\mathbf{p}},\hat{\mathbf{n}}\}$, and $l_{\hat{\mathbf{x}}}$ denotes the ground truth labels of $\hat{\mathbf{x}}$.

The adversarial loss with discriminator $D_{G2}$ is

Different with the previous adversarial loss, such as (14), we use the ground truth label $l_{\hat{\mathbf{x}}}$ to represent the “real” samples. Thus the adversarial loss (22) is used to fool the discriminator by classifying these generated embeddings as the real samples with ground truth $l_{\hat{\mathbf{x}}}$. Actually, our adopted discriminator $D_{G2}$ is a $(C+1)$-class classifier where $C$ denotes the number of ground truth categories for the real samples and the left one class represents the generated samples. When an embedding is given, $D_{G2}$ can discriminate it into the $C+1$ categories [31] and thus the real-like sample generation and label consistency are achieved at the same time. The training loss for $D_{G2}$ is denoted as:

where $\mathbf{x}^{\prime}$ denotes embeddings from $\{\mathbf{a}^{\prime},\mathbf{p}^{\prime}\}$ or $\mathbf{n}$, $l_{{\mathbf{x}}^{\prime}}$ denotes the ground truth labels of ${\mathbf{x}}^{\prime}$, and $\hat{\mathbf{x}}$ denotes embeddings from $\{\hat{\mathbf{a}},\hat{\mathbf{p}},\hat{\mathbf{n}}\}$.

We evaluated our method under the three datasets below. In order to better compare with other methods, we split the datasets according to [20][26][32][33], and the training set and test set contain image classes with no intersection.

• 1.

  The CUB-200-2011 dataset [34] consists of 200 bird species with 11,788 images. We use the first 100 species (5,864 images) for training and the rest 100 species (5,924 images) for testing.

• 2.

  The Cars196 dataset [35] consists of 196 car makes and models with 16,185 images. We use the first 98 models (8,054 images) for training and the rest 100 models (8,131 images) for testing.

• 3.

  The Stanford Online Products dataset (SOP) [32] consists of 22,634 online products from eBay.com with 120,053 images. We use the first 11,318 products (59,551 images) for training and the rest 11,316 products (60,502 images) for testing.

Configuration. The proposed method is implemented with Python3.7 and PyTorch 1.6.0, using the GoogleNet [36] and ResNet50 [37] architectures, pre-trained on ImageNet ILSVRC dataset [38]. All experiments are performed on individual NVIDIA Tesla T4 GPUs. We trained networks with standard back-propagation, which is performed by Adam optimizer with 4e-4 weight decay. We set the initial learning rate 1e-5 for the feature extractor, 1e-3 for the classifier, 1e-4 for the discriminators, and 1e-3 for the generators. Each generator is composed of two fully connected layers with dimensions 128 and 512. Note that all output embedding vectors of the feature extractor and generators are under normalization. Each training is run over 150 epochs for CUB200-2011/CARS196 and 100 epochs for Stanford Online Products. For a fair comparison with previous methods on deep metric learning, we used GoogLeNet [36] architecture as the CNN feature extractor with output embedding size of 512 for all the three datasets similar to [26] and [20]. While training, we set the batch size to 128, and all the training images are resized to $227\times 227$. During the training process, we will update all discriminators, generators, and the embedding extraction network according to Algorithm 1. We set triplet margin $\tau=0.2$ and set the the maximum value of reverse triple margin $\nu=0.2$ as well. The other related parameters of this work are summarized in Table 2.

Training Flow. We alternately train $F$ and the other models, and mini-batch SGD is applied in the training process. Specifically, we first initialize all the models and then pre-train model $F$. After that, we will iteratively train all the models for $epochs$ times. In each epoch, these models are updated for $batches$ times. For a specific batch, embeddings $\mathbf{X}$ are first produced by $F$, and then $\mathbf{X^{*}}$ and $\mathbf{X^{\prime}}$ can be obtained by using PLM and $G_{1}$. Then model ${D_{G_{1}}}$ can be updated with (16). After that, the generator ${{G_{1}}}$ are updated with (12) accordingly. However, for the training of $G_{2}$ and $D_{G_{2}}$, we will use the samples produced by the latest generative networks. For this reason, we will re-extract embeddings $\mathbf{X^{\prime}}$ after the model updating of Stage 1 for the training of $G_{2}$ and $D_{G_{2}}$ in Stage 2. Then we update them with (23) and (17) accordingly. Finally, we update model $F$ with the re-extracted $\hat{\mathbf{X}}$, $\mathbf{X^{\prime}}$ and the original $\mathbf{X}$ by using (5). With the above training flow, the final embedding model $F$ and the other models related to hard sample generation are obtained at the same time. When our network training flow is over, there is no additional computational effort involved in the final feature extraction for image retrieval. We summarize the whole embedding training flow as Algorithm 1.

We designed experiments to prove the effectiveness of our hard-sample generation.

Evaluation Criteria. We report the performance for both retrieval and clustering tasks. The mAP is widely used in state-of-the-art image retrieval works and it reveals the position-sensitive ranking precision. The definition of mAP is denoted as:

where $i$ is the $i_{th}$ query index and $N$ is the number of total queries [3]. $M^{i}_{\text{relevant}}$ is the number of relevant images corresponding to the $i_{th}$ query, ${r}$ is the rank and $P(r)$ is the precision at the cut-off rank of $r$. The employed Recall@K metric is determined by the existence of at least one correct retrieved sample in the K nearest neighbors [26].

For the clustering task, we employed NMI and $F_{1}$ as performance metrics. The normalized mutual information (NMI) [32] is defined by the ratio of the mutual information of clusters and ground truth labels and the arithmetic mean of their entropy: $\operatorname{NMI}(\Omega,\mathbb{C})=\frac{I(\Omega;\mathbb{C})}{H(\Omega)+H(\mathbb{C})}$, where $\Omega=\left\{\omega_{1},\ldots,\omega_{n}\right\}$ is a set of clusters and $\mathbb{C}=\left\{c_{1},\ldots,c_{n}\right\}$ is the ground truth classes. Here $I(\cdot,\cdot)$ and $H(\cdot)$ denotes mutual information and entropy respectively. $F_{1}$ score is deined as the harmonic mean of precision and recall [32].

Qualitative Analysis. We visualize our image retrieval performance in several selected cases and visually show the advantage of our algorithm. Besides, we depict the learned embeddings for different datasets and qualitatively verify the effectiveness of the proposed scheme.

We conduct full comparisons between our method and the other image retrieval algorithms. The involved strategies include deep pooling features and the recent deep embeddings with hard generation. For a fair comparison, all the involved algorithms are implemented with triplet loss architecture.

Typical Image Retrieval Methods. To comprehensively demonstrate the superiority of our proposed method over existing methods in the retrieval task, we compared our scheme with several typical baseline image retrieval methods built based on deep pooling features, including the average-pooled convolutional features under triplet architecture (Triplet), maximum activations of convolutions (MAC) [7], regional MAC (R-MAC) and Generalized-Mean (GeM) pooling [17]. For a fair comparison, we evaluated all the methods mentioned above using the same CNN model and trained the deep features with triplet loss.

Deep Embeddings with Hard Generation. Our proposed two-stage hard sample generation method is a novel generation method that enables the network to synthesize informative triplet more efficiently, leading to boosts in performance. The state-of-the-art deep embeddings with hard generation are included in this work for comparisons: the hard-negative generation method DAML [20], HDML [26] and symmetrical synthesis (SS) [27].

Comparison Analysis. Table 3, 4, and 5 show the quantitative results of our method and the comparison methods on the CUB-200-2011, Cars196, and SOP datasets. The feature dimension is 512 by default in the three tables above, except for special cases where the dimension is marked with superscript. Bold numbers indicate the best results and blue color represents the second-best performance. As we can see from the tables, our method achieves the best Recall@K and mAP on the CUB-200-2011, Cars196 datasets (Table 3 and Table 4), while obtaining at least the second-best performance on the Stanford dataset (Table 5). It should be noted that the image retrieval accuracy gain (Table 5) introduced by our method on the Stanford Online Products dataset is smaller than the improvement on the other two datasets (Table 3 and Table 4). The possible reason may come from the fact that the Stanford Online Products dataset [32] consists of more diverse images than the other two datasets. The existing data already contains many hard samples and the left improving space for hard sample generation on Stanford Online Products dataset is limited. For studying the performance of our method with varying embedding sizes, we conducted additional experiments on all three using a small embedding dimension. We can find that the performance of our proposed method can outperform the previous schemes even if the feature dimension is reduced to 128.

Compared with the image retrieval schemes with deep pooling features (MAC, R-MAC, GeM), the methods with hard sample generation (DAML, HDML, SS, and our scheme) produce better results. Compared with the DAML, HDML, and SS which improve performance by generating negative samples, our scheme can further boost the retrieval performance. In addition, our scheme can also obtain at least the second-best performance in the result of F1 and NMI. In summary, our proposed two-stage hard sample generation method is highly competitive in both retrieval and clustering tasks.

The motivation of our hard sample generation approach is to synthesize difficult sample pairs from a large number of simple samples to obtain valid information. In contrast, existing hard sample mining strategies directly select the most informative sample pairs from real samples for training. To further explore whether the generated hard samples and real hard samples are complementary, we design to combine the generated samples with the mined samples for training. We perform experiments to analyze the effect of combining our proposed method with a variety of hard sample mining strategies.

Hard Sample Mining Methods. Triplet loss needs to mine training tuples from the available mini-batch. In our study, all of our experiments are based on triplet architecture, so we choose the four most representative tuple mining strategies for our experiments. These strategies include random tuple mining (R) [12], semihard triplet mining (Semi)[25], softhard triplet mining (Soft) [24] and distance-weighted tuple mining (D) [14]. In this experiment, our proposed algorithm combined with random tuple mining fully equivalent to the two-stage hard sample generation method (THSG), which can be regarded as the baseline of our methods.

Implementation Details. In order to warrant unbiased comparability in Table 6, 7, and 8, our training protocol follows settings of [39]. The difference from the previous experimental setting is that we resize images to $224\times 224$ and utilize a ResNet50 architecture with frozen Batch-Normalization layers and embedding dimensionality 128 . When combining two different hard triplet for training, we only replace the original deep distance metric loss $\mathcal{L}_{org}$ with the hard mining deep distance metric loss $\mathcal{L}_{mining}$ and keep the other training steps of ours method unchanged. Therefore, the final deep embedding learning (4) is rewritten as:

Analysis of Results. We can find that experimental results in all evaluation criteria are further boosted by combining our method with a hard sample mining. Both in the retrieval task and in the clustering task, our method combined with hard sample mining results in better performance compared to the individual method. The performance improvement implies there is complementary information in the mined and generated hard samples. In particular, our generation method combined with simple mining strategies, such as random tuple mining or semihard triplet mining, yields an even more obvious improvement in results. Thus, our method does generate hard samples with information that is difficult to mine in the original data.

Visualized Retrieval Cases. We selected several queries from the utilized dataset and depicted the retrieval results in Fig. 6. The depicted methods are our work, the HDML of work [26] and the basic triplet scheme. The visualized results show that our method achieves better search accuracy. It should be noted that for some cases, such as the last two queries in Fig. 6, almost all the methods have many error results.

Embedding Visualization. We visualized the learned embeddings for the CUB-200-2011 and Cars196 datasets by using t-SNE [40], as denoted by Fig. 7 and Fig. 8. In each figure, several selected specific regions are enlarged to highlight the representative classes for easy observation. In each specific region, although the images in the same class suffer from large variations in poses, colors, and backgrounds, our proposed method still can group similar objects. The visualization results denote our learned embeddings have strong representation ability and thus can achieve better retrieval performance in an intuitive manner.

We conducted the ablation study of the proposed method on the Cars196 dataset. In Fig. 10, we show the results based on the Recall@1 criterion of different model settings, including our proposed method (OURS), the proposed method combined with distance-weighted hard mining (OURS^∗), the proposed method without using the second stage of synthesis (w/o $G2$), the proposed method without using the second stage of synthesis and category loss $\mathcal{L}_{class}$ (w/o ($G2$ & $\mathcal{L}_{class}$)), and the proposed method without both the generated hard sample loss $\mathcal{L}_{hard}$ and the category loss $\mathcal{L}_{class}$ (w/o ($\mathcal{L}_{hard}$ & $\mathcal{L}_{class}$)), which is the baseline of using triplet architecture.

We can find that the Recall@1 is further improved (80.2% $\rightarrow$ 81.6%) by combining our method with a hard sample mining. The performance improvement implies there is complementary information in the mined and generated hard samples. We remove the Generator $G_{2}$ and only use the first stage of generation to synthesize the anchor-positive pairs in hard triplets, then some performance loss is introduced (80.2% $\rightarrow$ 76.5%), which reveals the importance of hard sample generation which can extract the potential information from simple samples for embedding training. By further removing category loss $\mathcal{L}_{class}$, the performance suffers an obvious degradation again (76.5% $\rightarrow$ 71.9%), which tells that the category loss still can provide useful information under the metric learning architecture. Moreover, because we train the embedding model $F$ and the other hard sample generation models alternately, the absence of $\mathcal{L}_{class}$ will ruin the original obtained sample $\mathbf{X}$ and in turn impairs the generation of hard samples. Thus, the category loss $\mathcal{L}_{class}$ also plays a great role in our scheme. When the generated hard sample loss $\mathcal{L}_{hard}$ is totally removed (w/o $\mathcal{L}_{hard}$), we see the performance decreases a lot (71.9% $\rightarrow$ 67.8%), which verifies the effectiveness of the proposed stronger conditional synthesis scheme.

To further verify the contributions of each component in our method, we perform the similar experiment on all quantitative criteria including Recall@K, NMI, F1, and mAP, which are depicted as Fig. 10. The quantitative results in Fig. 10 also confirm the analysis as presented above, and the specific performance reduction brought to the model can be observed when each corresponding component is removed.