Achieving Better Kinship Recognition Through Better Baseline

We could use images from FIW dataset without the special preparations to obtain the image embedding, but facial recognition models work differently based on the different face alignment techniques that were used during their training. Moreover, better face detection and registration further improve model performance [1]. Knowing this, we re-detected faces in the challenge’s dataset and aligned them with landmarks from the RetinaFace [1] detector. At this step, some faces were not detected with the selected confidence threshold and were removed from the training and validation set. For the test set, such images were just resized to fit into the face recognition model. There were a total of $4$ images removed from the training set, $3$ images from validation, and $4$ problematic images that occurred in the test set.

The ArcFace [2] model was used for features extraction. This model was pre-trained on cleaned MS-Celeb-1M [3] dataset ³³3Clean dataset can be downloaded from https://github.com/deepinsight/insightface/wiki/Dataset-Zoo and has the embedding dimension of 512.

To compare two images $u_{i}$ and $u_{j}$ we used cosine distance between their computed embeddings $x_{i}$ and $x_{j}$:

The main difference between face and kinship recognition tasks is in the relative distance between different people. While face recognition cares mostly about pictures of the same person being close in embedding space, kinship recognition is trying to achieve a much harder task. In the later different people must be closer to each other than the other groups of people, ideally forming family clusters. The difference in the available labeled data and similarity between the tasks makes face recognition a great source domain for transferring to the kinship recognition domain.

In the previous iterations of RFIW, the metric learning was used as a transfer learning approach and achieved a great performance [10]. However, it requires pairs of images to be carefully sampled to confidently estimate the distribution of all distances in the metric space. Another concern is that it requires a significant amount of time to train the final model. Given the information for each person about their family association, we can construct a family classification problem similar to the recent methods in face recognition [16, 2, 17] and metric learning for image retrieval [23, 24]. With this our loss function looks like:

where $B$ is the batch size, $N$ is the number of families, $x_{i}$ is the image embedding of a member of the $y_{i}$-th family, $W$ is the weight matrix (with $W_{j}$ denoting it’s $j$-th column) and $b$ is the bias from classification layer.

In track 1 of RFIW Data Challenge images are divided between families. As the distribution between persons and between families in challenge’s data is non-uniform, we need to be careful with sampling the pairs, as validating model offline is crucial given the limited number of submissions. We sampled $5\ 000$ positive and $5\ 000$ negative pairs selected uniformly between all families from the validation set using algorithm 1.

Using this approach validates our model without the issue of popular families that have lots of images. We used AUC ROC metric on the validation pairs (see fig. 1) for choosing the best model for submission Another problem occurs though: as we select uniformly between the families, families with low member count are given higher priority. We chose to resolve this issue with a higher binarization threshold.

Given a comparison of two image embeddings using cosine distance (1) we need some binarization function to get the needed result. In our case, a simple binarization by threshold was used. The threshold can be chosen based on the trade-off between the false positive rate and the true positive rate. As we had no prior knowledge of how the test pairs were selected we chose target false positive rate based on our three submissions for the final phase.

Given the information about the pair’s possible kind of kinship relation, we could have chosen the threshold for every kind separately. Alas, we could only submit $15$ results and some kinds of relations had a low number of pairs to confidently select threshold. We further discuss this in section IV-C.

The retrieval task can be reduced to a series of verification tasks. Every probe image is matched with every gallery image to construct a retrieval matrix (every row contains a retrieval result for the probe image). Our task also has more than one image for every probe person, and we can solve this in different ways. One way is creating an aggregated feature for each probe person. We can do this by averaging all the embeddings from their images. Thus we reduced our task to a single feature per probe and can form a retrieval matrix.

The other way is using all matching results from all images. We need to add aggregation function $g(\cdot)$ that would consolidate distances between images to adapt to this. Given a gallery image $u$ and a probe person $P$ with $m$ images $\langle p_{1},\ \dots,\ p_{m}\rangle$ we can sort the gallery images using the distance:

Aggregation function should take a vector with an arbitrary number of elements and return a single real number. There are different options for such function but we tested only mean and max in this challenge.

With this approach, we need to compare embeddings for every probe image with every gallery image. It can be difficult to compute with reasonable resources when there is a large number of images per person. There is a variety of approaches [22] for that purpose: from reducing latent space [25, 26] to constructing special structures [27] but in this work we will only show (see IV-D) that getting a mean embedding for a person is comparable with searching using the aggregated function $g(\cdot)$.

The Recognizing Families In the Wild Data Challenge (RFIW2020) focuses on determining blood relations based on visual facial similarities. For that, it has $20867$ images from $763$ families for training and validation sets. This year there were a total of three tracks: one-to-one kinship verification (track 1), two-to-one kinship verification (track 2), and family search and retrieval (track 3). Our team chose to participate in tracks 1&3 so we will focus only on them.

In track 1 there were $39743$ pairs for the final testing. Methods were evaluated based on the average accuracy of binary classification (kin, non-kin) over all of the testing pairs. Additional measurements were provided separately for every kinship type.

Track 3 is the image retrieval problem with one family member as a query (or probe) and other family members with distractors as a gallery. There were $190$ probe subjects (each with a different number of images) and $3897$ images in the gallery. Methods were evaluated based on mean average precision (mAP) and rank@k metrics.

We used Mxnet [28] for the implementation of our pipeline. For detection and feature extraction insightface python package was used. In particular, retinaface_r50_v1 which is RetinaFace implementation with ResNet50 as the backbone and arcface_r100_v1 which is modified ResNet101 trained with ArcFace loss on cleaned MS-Celeb-1M dataset.

Re-detected and aligned (as described in III-A) faces were given to the feature extractor model to obtain image embeddings. Performance of this approach (pretrained on fig. 1) was used as a baseline to test our hypotheses.

First, we tried to add a simple classification layer and finetune the whole model on the train set with stochastic gradient descent with base learning rate of $0.0001$, momentum $0.9$, linear warmup for first $200$ batches of size $64$, linear cooldown for last $400$ batches, multiplying learning rate by $0.75$ on epochs $8,\ 14,\ 25,\ 35,\ 40$ and gradient clipping $1.5$ for $50$ epochs. Random color jitter and random lightning with parameter $0.15$ were used for the data augmentation. No random cropping or similar technique was used to not confuse the model that was trained on similar aligned images. After that, we added $L_{2}$ normalization of the embeddings and retrained the model starting with pre-trained weights. Performance of these two models on our sampled validation pairs (see III-C) can be seen in fig. 1.

We needed to binarize our predictions to submit for test verification on track 1. We chose threshold such that we had true positive rate (TPR) of $0.75$ on our sampled validation images for our first submission (pretrained in table I). Next submissions were tested with different thresholds and between several strategies, the one that showed the best average performance was to choose the threshold such that the method would have a false positive rate (FPR) of $0.2$. Other entries in table I are provided with that strategy of choosing the binarization threshold.

We tested the simple fine-tuning using a classification layer (+classification) with a similar approach where the embeddings are normalized to have a unit $L_{2}$ norm before the classification layer (+normalization). Both on our validation data and test set the second approach was superior. This indicates that consistency with our cosine distance metric that we use for image comparison is crucial for fine-tuning the model for kinship verification.

From comparison table I we can see that our approach performs poorly on grandparents-grandchildren type of kinship because there is a small number of images with this type of relationship in the training set, but we can mitigate this bias through a different threshold for every kind of relationship. Sadly, we could not test this idea due to the lack of time, but we provide a proof of this hypothesis with +different thresholds submission where we improved the performance for grandmother-grandson relationship by lowering binarization threshold.

We should note that though our approach scores first on average it is not by a far margin and mostly due to our great performance on sibling pairs. Having an $0.78$ average accuracy at best, automatic kinship recognition still needs to be improved to be considered for usage in real-world applications. For the reference, the best face verification models perform with FPR around $10^{-6}$ [7].

In track 3 we needed to aggregate several embeddings per probe person to rank gallery images. We used average consolidated embedding for our baseline submission pretrained and compared it to every gallery image using the cosine metric to get a resulting retrieval matrix. The same procedure was used with our best model from track 1 (+norm+class) and we can see that our approach gives improvements not only to the verification task but also to the retrieval. Then we compared this search method with different aggregation functions $g(\cdot)$ (see III-E). We can see that averaging embeddings for probe subjects perform worse than searching using all available embeddings with aggregation function but is still comparable. Furthermore, we can see that max aggregation function, which searches for the image from the gallery that is closest to any of the query images, has a higher rank@K metric than mean aggregation but lower mAP.

Table II shows that even the pre-trained ArcFace model performs much better than the other competitors and our pipeline improved this performance even further. But even such great performance is too low for trying to use this pipeline in a real-world scenario.