Dissecting the impact of different loss functions with gradient surgery

Given a triplet, $(\mathbf{f_{a}},\mathbf{f_{p}},\mathbf{f_{n}})$, there are two commonly used triplet losses in the literature, a triplet loss based on Euclidean distance:

where $D_{ap}=\|\mathbf{f_{a}}-\mathbf{f_{p}}\|$, $D_{an}=\|\mathbf{f_{a}}-\mathbf{f_{n}}\|$ are the distances between the anchor-positive and the anchor-negative pairs, and $m$ is a distance margin. A second common triplet loss is the triplet loss based on cosine similarity with NCA [1]:

where $S_{ap}=\mathbf{f_{a}}^{T}\mathbf{f_{p}}$, is the cosine similarity computed as the dot-product of the normalized anchor feature and the normalized feature from the positive example, the anchor-negative is computed in the same way, $S_{an}=\mathbf{f_{a}}^{T}\mathbf{f_{n}}$ and $\tau$ is the scaling parameter.

When comparing these two loss functions, their substantial differences make it challenging to determine how the loss affects performance. One loss is based on the Euclidean distance combined with a hinge function, while the other uses cosine similarity along with a negative log softmax function to combine the anchor-positive and anchor-negative pairs. Looking at the gradients of these loss functions makes the difference more clear. In triplet loss based on Euclidean distance, if the loss is greater than 0, its gradient can be derived from Equation 3 as:

Being explicit about this gradient allows us to name the direction that the positive example is being pulled to anchor example as $\mathbf{e_{p}^{euc}}$, and these are unit vectors defined as: $\mathbf{e_{p}^{euc}}=\frac{\mathbf{f_{p}}-\mathbf{f_{a}}}{\|\mathbf{f_{p}}-\mathbf{f_{a}}\|}$, with corresponding directions for the negative example, $\mathbf{e_{n}^{euc}}=\frac{\mathbf{f_{a}}-\mathbf{f_{n}}}{\|\mathbf{f_{a}}-\mathbf{f_{n}}\|}$.

The gradient of the triplet loss function based on cosine similarity can also be derived from Equation 4 to give a unit direction and magnitude:

where $\mathbf{e_{p}^{cos}}=-\mathbf{f_{a}}$, $\mathbf{e_{n}^{cos}}=\mathbf{f_{a}}$, $\mathbf{e_{a_{p}}^{cos}}=-\mathbf{f_{p}}$ and $\mathbf{e_{a_{n}}^{cos}}=\mathbf{f_{n}}$ are the unit gradient directions.

Though both $L_{euc}$ and $L_{cos}$ contain different gradient components, those components can be categorized into two major parts: unit gradient direction for moving the feature and a scalar weight that affects the length of the gradient in that direction. The weight itself can be divided into two sub-parts: the weight related to all three features in a triplet $\mathbf{f_{a}}$, $\mathbf{f_{p}}$ and $\mathbf{f_{n}}$ (Triplet Weight), and the weight related to the positive pair $\mathbf{f_{a}}$ and $\mathbf{f_{p}}$ or negative pair $\mathbf{f_{a}}$ and $\mathbf{f_{n}}$ in a triplet (Pair Weight).

With the categorizations of the gradient components, it becomes easy to compare the effects of each component. Before the comparison, we first show how recently proposed loss functions can be characterized by computing the direction and weights of the different gradient terms in Sections 3.2, 3.3 and 3.4 and then perform comparisons of the isolated effects of each gradient component in Sections 5.1, 5.2 and 5.3

The first gradient component is the unit vector in the direction of the gradient, derived from how the loss function moves the relative configuration of the anchor, positive and negative features. We refer to the unit gradient direction of the two most common metrics Euclidean distance and cosine similarity as Euclidean direction $\mathbf{e^{euc}}$ and cosine direction $\mathbf{e^{cos}}$. Recent work [9], also suggests to other directions, Euclidean orthogonal direction $\mathbf{e^{euc-orth}}$ and cosine orthogonal direction $\mathbf{e^{cos-orth}}$.

We define the pair-weight $P$, for the anchor-positive pair $P_{+}$ and anchor-negative pair $P_{-}$. The pair weight of cosine similarity $P^{cos}$ based triplet loss is a constant scaling parameter. This is useful as a baseline for comparison. For this case where both pair weights are set with constant $1$, as:

In Euclidean distance based triplet loss, the pair weight $P^{euc}$ is different for the anchor-positive and anchor-negative pairs:

and indicates the pair weight is proportional to the distance between the anchor and the other element of the pair.

Recent works [18, 19, 22] argue that the weight for anchor-negative pair should be large when they are close to each other. Otherwise, as mentioned in  [22], the optimization will quickly converge to bad local minima. The solution in Circle loss [18] is to apply a linear pair weight $P^{lin}$: for negative pairs, the weight is large if the similarity is large and small if the similarity is small; for positive pairs, the weight is large if the similarity is small and small if the similarity is large:

Early work binomial deviance loss [24] uses a similar pair weight but with a nonlinear sigmoid form $P^{sig}$:

where $\alpha$, $\beta$ and $\lambda$ are three hyper-parameters.

Multi-similar(MS) loss [19] combines ideas from the lifted structure loss [17] and binomial deviance loss [24], which includes not only the self-similarity of a selected pair but also the relative similarity from other pairs.

The MS paper [19] tries to find a loss function whose derivative fits the proposed pair weight. Because the relative similarity term involves additional examples (outside the triplet), this creates additional gradients relative to those examples, even though the stated purpose is to weigh the selected pair. Therefore, it’s difficult to understand if the performance gain is coming from the proposed pair weight or from the gradients affecting the feature location of these other examples. By casting their work within our framework, we can decouple the pair-weighting and explore the impact of this term in isolation.

We follow the MS paper to cast their weighting function $P^{sig-ms}$ in our framework. Given a triplet, the self-similarity of the selected positive pair and negative pair are $S_{ap}$ and $S_{an}$. The similarity of other positives and negatives to the anchor is considered as relative-similarity, noted as ${R_{ap}}^{i}$ and ${R_{an}}^{j}$. In addition, [19] also defines $\mathcal{P}$ and $\mathcal{N}$ be the sets of selected ${R_{ap}}^{i}$ and ${R_{an}}^{j}$, where

where

When $m_{+}^{sig}=m_{-}^{sig}=1$ the pair weights simplify back to the sigmoid form in equation 11.

In practice, training MS loss needs to tune four hyper-parameters $\alpha$, $\beta$, $\lambda$ and $\epsilon$ to fit different datasets, making the training not convenient and not efficient. With analysis on relative-similarity terms $m_{+}^{sig}$ and $m_{-}^{sig}$ in the appendix, we define a clearer and parameter free version of pair weight called linear MS pair weight $P^{lin-ms}$, which behaves similar to the original MS weight:

where

The triplet weight contains the similarity of both positive and negative pairs of a triplet, measuring whether a triplet is well separated or not. In Euclidean distance based triplet loss, the triplet weight (denoted as $T$) is a constant indicating that every triplet will be treated equally. For the fair comparison for other triplet weights, we set constant weight $0.5$.

In cosine similarity based triplet loss, the triplet weight is:

$T^{cos}$ is rely on the difference of $S_{ap}$ and $S_{an}$. When a triplet in a correct configuration, $S_{ap}-S_{an}>0$, the triplet weight is small. Otherwise, the triplet weight will be large.

In Circle loss [18], the triplet weight is:

Because $T^{cos}$ only considers the similarity difference $S_{ap}-S_{an}$, some corner cases such triplet with both large $S_{ap}$ and $S_{an}$ or both small $S_{ap}$ and $S_{an}$ are not well treated. The idea of $T^{cir}$ is to introduce a non-linear mapping for $S_{ap}$ and $S_{an}$ in the exponential term in order to weight more on the corner cases.

Figure 3 shows the triplet weight diagram, a triplet visualization tool from [22], for $T^{cos}$ and $T^{cir}$ with $\tau=1$. The equal weight line in $T^{cos}$ is straight lines with form $S_{ap}-S_{an}=\text{const.}$. And the equal weight line in $T^{cir}$ is circular lines with form $(S_{ap}-1)^{2}+S_{an}^{2}=\text{const.}$.

Selectively Contrastive Triplet(SCT) loss [22] selects triplets with hard negatives (the negative example in a triplet is closer to anchor than the positive example) and applies only contrastive loss to the hard negative pairs during the batch training. At gradient level, this approach is to remove the gradients from the anchor-positive pairs for triplets with hard negatives. We treat the selection as a masking operator on positive pair weight:

Because the decision boundary of triplets selection $S_{an}=S_{ap}$ is a 1st order straight line, we note this masking operator is noted as $T^{sc1}$. Besides, we continue to extend the selection idea with Circle loss. The triplets in the corner cases can be also selected to only separate the negative pairs. Then, the decision boundary of the selection operator becomes a 2nd order circular line. We note it as $T^{sc2}$,

Figure 3 right shows the difference decision boundaries of $T^{sc1}$ and $T^{sc2}$.

In this section, we have derived ways to represent many previous loss functions in terms of their gradients. We have explicitly defined the gradients in terms of how the anchor, positive and negative are moved, defined them in terms of a unit vector in the direction of motion, a weight of anchor-positive term and the anchor negative term and weight of the triplet overall. Table 1 shows how to map different combinations of gradient components into currently proposed loss functions. Section 5 gives explicit experiments to understand the isolated effects of these three parts of gradient component.

To understand the behavior of unit gradient directions in section 3.2, we set constant pair and triplet weight $T^{con}=0.5$ and $P^{con}=1$, and vary the choice of Euclidean, cosine, Euclidean-orthogonal and cosine-orthogonal direction.

In Table 2, we find the following trends. First, the cosine and cosine-orthogonal direction have better Recall@1 accuracy than other directions. Second, the cosine-orthogonal gradient direction gives an improvement for both datasets compared to the cos direction. More analysis will be discussed in section 5.4.

To understand the behavior of the pair weights, we set triplet weights with constant form $T^{con}=0.5$ and gradient direction with $\mathbf{e^{cos}}$ and $\mathbf{e^{euc}}$ for two sets of results respectively. As for baseline results, the pair weights are set with constant form $P^{con}=1$.

In Table 3, all pair weights provide a clear performance gain to their baseline results. Also, the performance gap of gradient direction $\mathbf{e^{euc}}$ and $\mathbf{e^{cos}}$ after applying the pair weight is greatly reduced.

Both relative-similarity methods $P^{lin-ms}$ and $P^{sig-ms}$ perform better than the method with only self-similarity on CAR dataset across different learning rates. Due to the property of well separation on In-shop dataset as shown in Figure 4, the relative-similarity term will less likely exist during the train because few positive and negative examples will be in $\mathcal{P}$ and $\mathcal{N}$ set as mentioned in equation 12. $P^{lin-ms}$ is performance almost as same as $P^{lin}$, but $P^{sig-ms}$ shows some computation instability effect. We put a further analysis of this effect in the appendix.

In summary, Table 3 shows several features related to the pair weight. First, pair weight causes substantial improvement in recall@1 accuracy. Second, in most cases, the linear and sigmoid pair weight outperforms the default Euclidean pair weight. Third, the linear version of the multi-similarity gradient direction is much more robust to different learning rates than the sigmoid version(see in appendix), and gives better performance and Recall@1 accuracy.

Table 4, we show two groups of experiments to compare the seven triplet weights. One group sets the pair-weight to be constant $P^{con}=1$. Another group uses the linear pair weight $P^{lin}$. All experiments use cosine gradient direction.

Comparing to the baseline method where triplet weight $T^{con}=0.5$, $T^{cos}$ and $T^{cir}$ has minimal but slight boost in performance; $T^{sc1}$ and $T^{sc2}$ has bigger impact on CAR data set than In-shop dataset. This is due to the properties of these two datasets as shown in Figure 4. CAR dataset has low inter-class variance(images from different classes may look similar) while In-shop dataset has high inter-class variance(images from different classes look not similar). The major challenge of CAR dataset is to distinguish similar images with different labels, and this is the purpose of triplet operators $T^{sc1}$ and $T^{sc2}$ because they concentrate on separating triplets with hard negative in training. And In-shop dataset is to relatively easy to separate images with different labels, the goal is to continue separating the images better, which is the impact of $T^{cos}$ and $T^{cir}$

Therefore, we can conclude that the performance gain in Circle loss is largely from the pair weight not from the triplet weight. Selective Contrastive operator benefits the training tasks which need to separate triplets with hard negative and is not helpful for training tasks which easily separate triplets during the training.

In Section 3.2 and Figure 2, the different gradient behaviors of $\mathbf{e^{euc}}$ and $\mathbf{e^{cos}}$ have been showed. But additional discussion will highlight the performance difference shown in Sections 5.1 and 5.2.

We first decompose the unit gradient to move positive and negative features into two directions: the direction along positive and negative feature $d_{\parallel}$ and the direction orthogonal to positive and negative feature $d_{\bot}$. As shown in Figure 2, only the gradient component along $d_{\bot}$ effectively contributes to the angle change of anchor-positive and anchor-negative pair which directly affect the similarity score. The effective gradient projection strength for $\mathbf{e^{euc}}$ and $\mathbf{e^{cos}}$:

where $S$ is the similarity of a positive or negative pair. The derivation of the above projection length is shown in the appendix.

Figure 2 right shows the change of the effective gradient strength for $\mathbf{e^{euc}}$ and $\mathbf{e^{cos}}$ varying as a function of pair similarity. Because most pairs during the training have positive similarity, we focus on projection length when similarity is positive.

The Euclidean gradient has stronger force to pull positive close and push negative away than the cosine direction when two features are close to each other. Therefore, Euclidean gradient continues to force features together even when they are already relatively close, unlike the cosine gradient. In Figure 5 left column, we show the triplet diagram plot of triplets extracted from the last 5 epochs of training (epoch 55-60) on CAR dataset. The Euclidean direction clusters the same label feature more tightly than the cosine direction because there are more triplets along the right edge of the triplet diagram comparing to the scatter of cosine direction.

However, the tight clustering behavior in training leads to even the triplet with nearest positive and the nearest negative to be compact. In the middle and right column of Figure 5, we plot just these triplets for the whole training set (middle), and testing set (right). The Euclidean gradient has more triplets very close to the top right corner, indicating that point have very similar same class and different class neighbors, while cosine gradient creates triplets that are more spread out. The spread out effect indicates the feature learned by the deep model is distinguishable [26, 21, 23]. Because these Euclidean feature are more compressed (for both the anchor-positive and anchor-negative pairs), it is harder for the network to learn distinguishable features that if it is using the cosine gradient.

One more piece of evidence to support the analysis above is the pair weight result in section 5.2. When the Euclidean pair weight $P^{euc}$ is applied to Euclidean direction and cosine direction, the performance gap between these two methods is almost disappeared. This is because the Euclidean pair weight $P^{euc}$ reduces the weight de-emphasizes positive pair when they are already close, and therefore avoid its tight clustering behavior, making Euclidean direction behave similarly as cosine direction.