Multi-Agent Semi-Siamese Training for
Long-tail and Shallow Face Learning

The straightforward solution to enlarge the diversity of samples is data generation. Hariharan and Girshick [9] propose to hallucinate additional samples of few-shot classes by category-independent transformations. Dixitet et al. [25] present an approach guided by attributes for data augmentation. Delta-encoder [26] is designed to synthesize new samples for unseen categories even if seeing few samples from them, by extracting transferable intra-class deformations and applying them to the few provided examples of a novel class. Yin et al. [10] assume a Gaussian prior of the variance to transfer variations of normal classes to few-shot classes, which can augment the feature space of few-shot classes. Ding et al. [27] propose a generative adversarial one-shot face recognizer to synthesize data for one-shot classes. Specifically, a knowledge transfer generator and a general classifier are combined to guide the generation of effective data, which contributes to promoting the underrepresented classes.

Besides, to regularize the learning process, some methods adjust the conventional settings such as weight of samples and evaluation of loss. Guo et al. [11] develop the classifier to recognize few-shot samples by aligning the norms of the weight vectors of few-shot classes and the normal classes. For few-shot learning in face recognition, Ma et al. [28] adopt a hierarchical regularization method which utilizes coarse class labels for training and fine class labels for refining to avoid overfitting, Cheng et al. [13] propose the enforced softmax to guide model to produce a more compact vectorized representation by optimal dropout, selective attenuation, $L_{2}$ normalization, and model-level optimization. Simon et al. [29] introduce a dynamic classifier that is constructed from few samples to design a framework for few-shot learning. With such modeling, the results on the task of supervised and semi-supervised few-shot classification are competitive.

Arising from the long-tail distribution of nature data, the imbalance problem has been extensively studied [30, 31, 32, 33, 34]. Some metric-based methods add desirable constraints on the distance between samples in the feature space. Huang et al. [35] propose quintuplet sampling with triple-header hinge loss to maintain both inter-cluster and inter-class margins for solving imbalanced distribution in vision classification tasks. Center invariant loss [36] aligns centers of the minority classes to the majority. Range loss [18] is developed to overcome the impact of long-tail distribution by simultaneously minimizing the harmonic mean of $k$ largest intra-class distance and maximizing the shortest inter-class distance within the mini-batch. Ring loss [37] applies soft feature normalization to augment standard loss functions. In addition, fair loss [38] is proposed to balance different classes by using reinforcement learning.

In terms of data, some other methods have been proposed for improvement. Classical strategies include under-sampling head classes, over-sampling tail classes, and data instance re-weighting, which aim to balance the number of samples. By SMOTE [21], over-sampling of the few-shot class and under-sampling of the normal class can be combined to achieve better classifier performance than replication-based methods. BalanceCascade [39] trains the learners sequentially. The majority of class examples classified correctly are not taken into consideration further. Unfortunately, these methods can easily suffer from over-fitting and introduce undesirable noises. The under-sampling also might discard a large portion of samples so that missing valuable information. Some of these solutions have been further extended to solve the data imbalance problem in deep learning [40]. In order to remove representation bias, REPAIR [41] learns weights for different classes to re-sample data. Zhong et al. [42] attempt to treat the head data and the tail data of long-tail distribution in an unequal way to make full use of their respective characteristics. Accompanying with noise-robust loss functions, these two training streams offer complementary information to deep feature learning. Zhang et al. [43] focus on the optimization of dataset structures and build a medium-scale face recognition training set BUPT-CBFace, which can alleviate the problem of recognition bias. Huang et al. [44] conduct extensive and systematic experiments to validate the effectiveness of classic schemes mentioned above. Furthermore, CLMLE is proposed to reduce the class imbalance inherent in the data neighborhood, by enforcing a deep network to maintain inter-cluster margins both within and between classes. As an extension of [28], the two-step training schema [45] is designed, which aggregates similar classes in tail parts and then selectively disperses those aggregated superclasses, to address the long-tail problem.

The major issue in shallow face data is the lack of intra-class information. Its adverse effect on model training has been discussed in Du et al. [7]. Here, we elaborate the detailed behavior of model when trained on shallow data with the current state-of-the-art methods, and introduce a novel perspective for understanding the difficulty of training on shallow face data and, more general cases, long-tail face data.

The state-of-the-art training methods are mostly developed from the softmax loss and its variants. Without loss of generality, taking softmax as an example, the formulation can be written as

where ${n}$ is the class number, $s$ is the scaling parameter, $\boldsymbol{x}$ is the sample feature with the L2 normalization ($\boldsymbol{x}=\boldsymbol{x}/\|\boldsymbol{x}\|_{2}$), $\boldsymbol{w_{j}}$ is the $j$-th class weight with the L2 normalization ($\boldsymbol{w_{j}}=\boldsymbol{w_{j}}/\|\boldsymbol{w_{j}}\|_{2}$), $y$ is the ground-truth label of the sample, and the output of the last FC layer is the inner product of the sample feature $x$ and $j$-th class weight $\boldsymbol{w_{j}}$. In the training process, the goal is maximizing the intra-class pair similarity $\boldsymbol{{w}_{y}}^{\mathrm{T}}\boldsymbol{x}$ and minimizing the inter-class pairs $\boldsymbol{{w}_{j}}^{\mathrm{T}}\boldsymbol{x}$. If the sample number per class is large enough (a.k.a. deep data), $\boldsymbol{x}$’s can provide abundant intra-class diversity. There are, however, only two samples per class (i.e., the gallery $\boldsymbol{x_{g}}$ and probe $\boldsymbol{x_{p}}$) in shallow data. As the prototype $\boldsymbol{{{w}_{y}}}$ is determined by $\boldsymbol{x_{g}}$ and $\boldsymbol{x_{p}}$, the three vectors will easily converge close ($\boldsymbol{w_{y}}\approx\boldsymbol{x_{g}}\approx\boldsymbol{x_{p}}$) that the loss value in this class approaches to the lower bound. Therefore, in each iteration of batch-wise training, the network is well-fitted on a small number of classes and badly-fitted on the remaining ones. As a result, the total loss value will be oscillating and the training will be harmed. Moreover, due to the lack of the intra-class diversity in features space of all the classes ($\boldsymbol{x_{g}}\approx\boldsymbol{x_{p}}$), the prototype $\boldsymbol{{{w}_{y}}}$ is pushed to zeros in most dimensions so that it is unable to span an effective feature space. Specifically, given a network and loss function in conventional training, the loss landscape will become much more winding when the data depth becomes shallow. One can refer to Du et al. [7] for detailed discussion.

In this context, SST is designed to improve the stability of training on shallow data. We denote gallery and probe images by $I_{g}$ and $I_{p}$, and their features $x_{g}=\phi(I_{g})$ and $x_{p}=\phi(I_{p})$, where $\phi$ is the backbone. $\phi$ should be Semi-Siamese, coined gallery network $\phi_{g}$ and probe network $\phi_{p}$, to maintain the distance between $\boldsymbol{x_{g}}$ and $\boldsymbol{x_{p}}$. $\phi_{g}$ and $\phi_{p}$ have the same architecture and close (but non-identical) parameters, $\phi_{g}=\phi_{p}+\boldsymbol{\epsilon^{\prime}}$, to prevent features from collapse, $\phi_{g}(I_{g})=\phi_{p}(I_{p})+\boldsymbol{\epsilon}$. To implement the Semi-Siamese networks, a common approach is to add a network constraint $\|\phi_{g}-\phi_{p}\|<\epsilon^{\prime}$ in the training loss. As suggested by MoCo [46], we choose a better approach that updating gallery network in the momentum way, which is defined as

where $m$ is the weight of moving-average, and the probe network $\phi_{p}$ updates with SGD.

The prototype constraint is deployed in the training objective to enlarge the entries of prototype, such like $\mathcal{L}+\beta(\alpha-\|\boldsymbol{w_{y}}\|)$ with parameters $\alpha$ and $\beta$, avoiding the vanishing issue in dimensions of $\boldsymbol{w_{y}}$. This technique, however, does not show effective improvement (Table II), since the entries are enlarged uniformly. To handle this issue, SST replaces $\boldsymbol{w_{y}}$ with the gallery feature $\boldsymbol{x_{g}}$ as the prototype rather than manipulating $\boldsymbol{w_{y}}$ itself. The feature-based prototype avoids the vanishing issue, and thereby keeps the discriminative components in the prototype.

The experiments in Du et al. [7] show that SST has remarkable advantage over the conventional training on shallow data. Inspired by Markov-Lipschitz theory [47], we analyze the advantage of SST from the perspective of the bi-Lipschitz continuity in the following.

A function $\eta:\mathbb{R}^{H\times W}\rightarrow\mathbb{R}^{n}$ is locally bi-Lipschitz if for all $x_{j}\in\mathcal{N}_{i}$ (the neighborhood of $x_{i}$), there exists a real constant $K\geq 1$ satisfying

$K$ is termed as bi-Lipschitz constant. According to Convex Optimization theory [48], a differentiable loss function $\mathcal{L}(x)$ is called smooth if its gradient $\eta(x)$ is Lipschitz or bi-Lipschitz continuous. The optimization of smooth function will be more stable and robust if $K$ is smaller [47].

First, we compare the bi-Lipschitz constant $K$ of the conventional training and SST to compare their stability of optimization process. For conventional training, according to Eq. 1, when the gallery $I_{g}$ and probe $I_{p}$ are fed into the backbone $\phi$ separately, we have the gradient

As the training carries on, for $I_{g}$ and $I_{p}$ that belong to the same ID, $\phi(I_{g})\approx\phi(I_{p})$ and $\eta_{Convt.}(I_{g})\approx\eta_{Convt.}(I_{p})$. This means $K$ is very large. Thereby, the optimization of conventional training on shallow data is difficult to be smooth and steady [47].

As for SST, $\boldsymbol{w_{y}}$ is replaced with the gallery feature $\boldsymbol{x_{g}}=\phi_{g}(I_{g})$, and the gradient with respect to $\boldsymbol{x_{g}}$ becomes to

Given the shallow data including a pair of gallery and probe per ID, which are the input of SST, Eq. 5 is developed to

where $\boldsymbol{x_{g}}=\phi_{g}(I_{g})$ and $\boldsymbol{x_{p}}=\phi_{p}(I_{p})$.

For the clarity of the following deduction, we denote $f$ as a function of $\boldsymbol{x_{g}}$ and $\boldsymbol{x_{p}}$ that

Then, the upper bound of the variation of $\eta_{x_{g}}$ is computed as

where $\boldsymbol{x_{g}^{\prime}}=\boldsymbol{x_{g}}+\epsilon_{1}$. Then, we can obtain the ratio to the input variation is

One can refer to the supplementary material for the deduction details.

This indicates that there always exists a constant $K$ making SST satisfies the right side inequality in Eq. 3. Besides, the ratio of variation also has a lower bound. This is because that $\eta_{x_{g}}([\boldsymbol{x_{g}};\boldsymbol{x_{p}}])\neq\eta_{x_{g}}([\boldsymbol{x^{\prime}_{g}};\boldsymbol{x_{p}}])$ always holds owing to the Semi-Siamese structure and the momentum-updating method.

In most of the existing experiments, the parameter $s$ is set around 30, so $\frac{s^{2}}{2}>1$. Therefore, there exists $K\geq 1$ that constrains the variation range of ${\|\eta_{x_{g}}([\boldsymbol{x_{g}};\boldsymbol{x_{p}}])-\eta_{x_{g}}([\boldsymbol{x_{g}^{\prime}};\boldsymbol{x_{p}}])\|}_{2}$. In contrast to the conventional training, SST can satisfy the bi-Lipschitz continuity, which is the basic reason that SST achieves stable and robust convergence on shallow data.

Based on the advantage of SST on shallow data, we further extend it to handle a more general problem that is long-tail face learning. Unfortunately, with the non-uniform distribution of training samples in long-tail face data, SST suffers from bad convergence. The tail classes have faster convergence speed due to the fewer samples, but the head classes including large number of samples converge slower. The power of SST is undermined on long-tail data.

In order to better satisfy the locally bi-Lipschitz continuity, we propose MASST to strengthen the stability and robustness with smaller value of $K$. The single gallery network in SST is replaced with a gallery network stack including multiple agents ($\phi_{g_{1}},\phi_{g_{2}},\cdots,\phi_{g_{S}}$). The framework of MASST is depicted in Fig. 2. In the training process, we sequentially rotate one agent from the stack as the gallery network. The update method of the $i$-th agent is defined as

where $i=1,2,\cdots,S$, $S$ is the number of agents in the gallery network stack, and $a$ is a parameter that adjusts the weight of the first and second terms. With the second term, the agents in the gallery network stack can maintain distance intentionally. Therefore, MASST can have smaller value of $K$ to achieve smoother optimization process.

Similar to SST, we have the gradient in MASST

where $\boldsymbol{{x_{g}}^{ma}}=\phi_{g_{i}}(I_{g})$ and $\boldsymbol{{x_{p}}^{ma}}=\phi_{p}(I_{p})$.

Without loss of generality, ignoring the scaling parameter $s$, the ratio of the variation of $\eta_{x_{g}}([\boldsymbol{x_{g}};\boldsymbol{x_{p}}])$ to the variation of $\eta_{x_{g}}([\boldsymbol{{x_{g}}^{ma}};\boldsymbol{{x_{p}}^{ma}}])$ can be written as

where $\boldsymbol{{x_{g}^{\prime}}^{ma}}=\boldsymbol{{x_{g}}^{ma}}+\epsilon_{2}$, and $\lVert\nabla_{\boldsymbol{{x_{g}}}}\frac{e^{\boldsymbol{{x_{g}}}^{T}\boldsymbol{{x_{p}}}}}{e^{\boldsymbol{{x_{g}}}^{T}\boldsymbol{{x_{p}}}}+e^{\boldsymbol{{x_{g}^{\prime}}}^{T}\boldsymbol{{x_{p}}}}}\rVert_{2}$ can be developed to

Compared with SST, the update method of the gallery network in MASST increases differences among agents in the stack, i.e., $\epsilon_{2}>\epsilon_{1}$. Then, we can find that

According to the regional monotonicity of Equation LABEL:monotone w.r.t. $\boldsymbol{{(x_{g}-x_{g}^{\prime})}}^{T}\boldsymbol{{x_{p}}}$ (more details are shown in the supplementary material), we obtain:

for $\boldsymbol{{(x_{g}-x_{g}^{\prime})}}^{T}\boldsymbol{{x_{p}}}\leq\boldsymbol{{({x_{g}}^{ma}-{x_{g}^{\prime}}^{ma})}}^{T}\boldsymbol{{{x_{p}}^{ma}}}$. By further comparing the ratio to the input variation in SST and MASST,

We can verify that there exists a smaller constant $K$ making MASST better satisfies the bi-Lipschitz continuity than SST. In Section III-C, the result of experiment also proves this property of MASST.

It is worth noting that, although multiple gallery agents are involved for training, we only utilize the probe network for test inference. Thereby, the test accuracy is improved while keeping the inference cost unchanged.

With the non-uniform increase of samples, the training suffers from data imbalance and intra-class diversity dearth simultaneously, which leads to the instability of the convergence process.

In order to verify the effectiveness of MASST in long-tail face learning, we construct long-tail data by the existing identity-based sampling method. It is expressed as follows,

where $index$ is the ID index sorted in descending order according to the number of intra-class samples that $index\in N^{*}$, $Num_{org}$ and $Num_{new}$ are the numbers of intra-class samples before and after sampling, and $r$ is an adjusting parameter.

With the increase of $r$, the degree of long-tail distribution is intensified and the sample size becomes smaller correspondingly. $Num_{new}\geq 2$ guarantees that there are at least two images as input for training in tail classes. Different from deep data, the distribution of synthetic long-tail face data is non-uniform (as shown in Fig. 3 (a)). Obviously, shallow face data is an extreme case of long-tail face data, and long-tail face learning is a more general case, which suffers from data imbalance and the lack of intra-class diversity simultaneously.

To compare the optimization process, the key problem here is measuring the loss landscape’s curvature. By means of the Taylor expansion, we have

then,

where $H$ is the Hessian matrix of loss function $\mathcal{L}(x)$.

Therefore, the Lipschitz norm is given by ${\|\eta\|}_{Lip}$ = sup$\sigma(\nabla\eta)$ = sup$\sigma(H)$ = $\sigma(H)$, where $\sigma(A)$ is the spectral norm of the matrix $A$ ($L_{2}$ matrix norm of $A$). The gradient $\eta$ satisfies the Lipschitz continuity better with the smaller spectral norm $\sigma(H)$. Naively forming the Hessian matrix to compute the spectral norm has a prohibitive computational cost. If there are $q$ parameters (weights and biases) in the network, then the Hessian matrix has dimensions $q\times q$. The computational effort needed to evaluate the Hessian will scale like $O(q^{2})$, and it also requires storage that is $O(q^{2})$. In order to compute the principal eigenvalue of the Hessian matrix efficiently, we use the power method, which is a matrix-free algorithm [49]. For a random vector $v$, whose dimension is the same as $\eta$, as it is independent to $\theta$, we have

where $H$ is the Hessian matrix. The curvature can be computed efficiently in this way.

The algorithm is shown in Alg. 1. Specifically, we first compute the gradient $\eta$, and initialize a random vector $v$, followed by the operation of the inner product. Thus, we simply backpropagate $\eta^{T}v$ rather than computing the full Hessian matrix. When the error between two calculations of principal eigenvalue is less than the threshold value ($thr=1e-3$), or the iterations exceed the maximum iterations ($max=50$), the loop is stopped. In this way, the quantity of interest is not the Hessian matrix $H$ itself but the product of $H$ with some vector $v$. The Hessian vector product $Hv$ that we wish to calculate, however, has only $q$ elements, so instead of computing the Hessian as an intermediate step, we find an efficient approach to evaluating $Hv$ directly that requires only $O(q)$ operations.

For the conventional training, SST and MASST with MobileFaceNet on shallow face data and synthetic long-tail face data, we calculate the principal eigenvalue of the Hessian matrix during the training process. As shown in Fig. 4, we can find: (1) the curvature of the conventional training becomes too small on shallow face data, which means it converges to the local minimum and optimizes invalidly; (2) on long-tail face data, the curvature of the conventional training gradually increases with the increase of iteration times, which means the training process becomes more tortuous; (3) compared with conventional training, the curvatures of SST and MASST are maintained at the medium range whether on shallow face data or long-tail face data. We also calculate the average value and standard deviation of curvature during the training process in Table I. On both shallow face data and long-tail face data, MASST has a smaller average value and standard deviation of curvature to maintain the stable optimization process.

According to the above experimental results, we further intuitively summarize the optimization processes of conventional training, SST, and MASST in Fig. 5. The optimization process of conventional training on deep data is smooth. However, when encountering long-tail data, the process to obtain an optimal solution becomes tortuous. This issue has been greatly improved in SST. Fortunately, with the combination of the gallery network stack, MASST achieves the smoothest optimization process.

To analyze comprehensively, we compare SST and MASST with the different choices aforementioned, such as the network constraint ($\|\phi_{g}-\phi_{p}\|<\epsilon^{\prime}$) and the prototype constraint ($\beta(\alpha-\|w_{y}\|)$). Table II compares their performance with four basic loss functions (softmax, A-Softmax, AM-Softmax and Arc-softmax). In this table, “Convt.” denotes the plain training, “A” denotes the prototype constraint, “B” denotes the network constraint, “C” denotes the gallery queue, “D” denotes the combination of “B” and “C”, “SST” denotes the ultimate scheme of Semi-Siamese Training which includes the moving-average updating Semi-Siamese networks and the training scheme with gallery queue, “MASST-MA” denotes the ultimate scheme of Multi-Agent Semi-Siamese Training which replaces the single gallery network in SST with a gallery network stack that involves multiple agents and updates by the moving-average. “MASST” denotes the gallery network stack in MASST updates in a composite way. From Table II, we can conclude: (1) enlarging $w_{y}$ in every dimension indiscriminately is not effective in shallow face learning, as the prototype constraint “A” leads to decrease in most terms; (2) compared with the network constraint “B” and the gallery queue “C”, the combination of them “D” obtains further improvement; (3) SST employs moving-average updating and gallery queue, and gets better performance; (4) finally, MASST sets up the gallery network stack including multiple agents and designs a more appropriate update mode, which achieves the best results by most of the terms. The comparison indicates MASST can address the problem in shallow face learning and obtain a slight improvement in most test accuracy compared with SST.

To prove the effectiveness of MASST on shallow data learning, we not only train the network with various loss functions, but also employ it to train different CNN architectures.

First, we set $r$ at 0.02 intervals between 0 and 0.1 to compare the performance of SST and MASST on LFW, AgeDB-30, CFP-FP, CALFW, CPLFW, BLUFR, and MegaFace. In order to balance performance and the time costs, we choose softmax as the loss function and use the MobileFaceNet to conduct this experiment. The result is shown in Table III. We can find that MASST gains higher accuracy in most of the test sets such as LFW, AgeDB-30, CFP-FP, CALFW, and BLUFR. In particular, it shows greater advantages than SST on MegaFace.

Then, we compare the performance of conventional training, SST and MASST on IJB-B and IJB-C. The results of experiments are shown in the supplementary material. With MASST, there are some improvements for different cases of loss functions both on IJB-B and IJB-C.

To observe the performance trends of MASST, SST, and conventional training when $r$ is changed, we set $r$ at 0.05 intervals between 0 and 0.3 (if $r>0.3$, too few samples to get satisfied training results). The number of images in the synthetic long-tail face data for different $r$ is shown in Fig. 3 (b). We employ Attention-56 and train the network with various loss functions such as softmax, AM-softmax, and Arc-softmax. Specifically, compare the performance of conventional training, SST, and MASST on the verification task of MegaFace.

As shown in Table IV, with the increase of $r$, i.e. the degree of long-tail distribution is intensified, the verification rate of conventional training decreases significantly. By contrast, MASST is more adaptable and achieves better performance than SST in long-tail face learning.

The previous experiments show MASST has well tackled the problems in shallow face learning and long-tail face learning and obtained significant improvement in test accuracy. To further explore the advantage of MASST for wider application, we adopt MASST scheme on the deep data (full version of MS1M-v1c), and make comparisons with the conventional training and SST. Table V shows the performance on LFW, AgeDB-30, CFP-FP, CALFW, CPLFW and MegaFace. MASST gains the leading accuracy in most of the test sets, and also the competitive results on CALFW and the face identification rank1 accuracy with 1M distractors. MASST (softmax) achieves about two percent improvement than conventional training (softmax) on AgeDB-30, CFP-FP, CALFW, and CPLFW which include the hard cases of large face pose or large age gap. Compared with SST, MASST shows better network property. Notably, MASST and SST reduce a large number of FC parameters by which the classification loss is computed for the conventional training.

The public training datasets, such as MS-Celeb-1M and VGGFace2, are different from the captured face images in real-world scenarios of face recognition. They are consist of well-posed face images collected from the internet. However, in real-world applications, the external conditions are more complicated. To address this issue, the typical routine is to pretrain a network on the public training datasets and fine-tune it on real-world face data. Although MASST is proved to be superior in shallow and long-tail face learning, we still look forward to extending it to the challenge on domain transfer. So, we conduct an extra experiment with pretraining on MS1M-v1c and finetuning on QMUL-SurvFace in this subsection. From Table VI, we can find that, no matter for classification learning or embedding learning, MASST boosts the performance significantly in both verification and identification, compared with the conventional training and SST.