Unsupervised Manga Character Re-identification via Face-body and Spatial-temporal Associated Clustering

Manga-related researches have increased with the popularity of e-manga. Some studies such as [23] focus on comic images, managing to generate comic-style images for real people using GAN. Some articles concentrate on manga translation. For example, [11] designs a fully automated manga translation system via a multi-modal and context-aware translation framework. There are also studies focusing on the content of comics. [13] analyzes the closure-driven narratives of the comics, and [9] and [24] focus on the extraction and order of manga frames(or storyboards). Some of the well-known cartoon-related datasets include Manga109 [20], Danbooru [22], iCartoonFace [33], etc. Manga109 provides 109 manga books with marks of the face and body images of characters, manga frames and texts, which provides a basis for us to study the clustering of manga characters from the perspective of the spatial-temporal relationship and face-body combination.

Prior to our focus on the comics world, retrieval of the pedestrian images in the real world had received a lot of attention. Person re-identification aims to retrieve the image of a specific pedestrian across the camera and is widely used in surveillance scenes.

Among many deep learning methods, unsupervised domain adaptation (UDA) methods have attracted much attention because they do not rely on manual annotation. [5, 14] provide a baseline of this approach. These methods usually consist of three steps: 1) They extract features of images in the target domain using models pre-trained in the source domain, 2) the generated features are used for clustering to generate pseudo-labels, and 3) pseudo-labels are used to further fine-tune the model. However, this method produces hard labels with noise. How to solve the noise problem is the key. On this basis, [7] designs a mutual mean-teaching framework and achieved effective results in unsupervised domain adaptive problems with open sets (i.e., the number of classes of images in the target domain is unknown).

In addition, problems such as image occlusion, appearance change, and pose transformation also plague person re-identification research.

Various methods have been proposed to cope with the appearance change problem in person re-identification research. [12] designs a Domain-Transferred Graph Neural Network to obtain robust representation for the group image. The proposed transferred graph addressed the challenge of the appearance change, while the graph representation overcomes the challenge of the layout and membership change. [29] proposed a cloth-Clothing Change Aware Network (CCAN) and addressed the appearance change issue by separately extracting the face and body context representation. And similar idea is applied in [27].

Some patch-based methods learn the partial/regional aggregation properties to make them robust against pose transformation and occlusion. [31] segmented the human body through a patch generation network and developed a PatchNet to learn discriminative features from patches instead of the whole images. [6] proposed a self-similarity grouping (SSG) approach, which exploits the potential similarity (from the global body to local parts) of unlabeled samples to build multiple clusters from different views automatically. These methods of combining the part and the whole have achieved effective results, and also provide ideas for us to migrate in the manga.

To date, a limited amount of published literature has reported research on manga character identification. The existing research can be broadly classified into supervised and unsupervised methods. Some supervision methods, such as [33], draw on the experience of face recognition methods and propose a method that uses domain data of real-world faces to help classify hyperplanes in the manga domain. This supervised approach can achieve satisfactory performance, but it requires a lot of labeling work, which brings limitations in the manga context.

Unsupervised methods such as [18] and [19] proposed a multiscale histogram of edge directions for sketch-based manga retrieval. However, these methods do not consider the diversity among individual manga volumes. [25] improves the clustering performance by adapting the manga face representation to the target volume, while manual rules are used in constructing pseudo-label pairs, which lack some generality.

In unsupervised re-identification based on clustering, given a target training set $X^{t}=\{x_{1}^{t},x_{2}^{t},...,x_{N^{t}}^{t}\}$ of $N^{t}$ images, the goal is to learn a feature embedding function $F(\theta;x_{i}^{t})$ from $X^{t}$ without any manual annotation, where parameters of $F$ are collectively denoted as $\theta$.

For UDA methods, however, an additional source dataset $X^{s}=\{x_{1}^{s},x_{2}^{s},...,x_{N^{s}}^{s}\}$ of $N^{s}$ images is typically used for pre-training, aiming at learning domain-invariant features by both the source and target domains. The UDA approach applies the classifier $F(\cdot|\theta)$ learned from the source domain data to the target domain data with no or little labeling in the target domain. The source domain images’ and target domain images’ features encoded by the network are denoted as $\left\{F\left(x_{i}^{s}|\theta\right)\right\}|_{i=1}^{N_{s}}$ and $\left\{F\left(x_{i}^{t}|\theta\right)\right\}|_{i=1}^{N_{t}}$, respectively.

To learn the feature embedding, traditional clustering-based UDA methods usually consist of three steps:

1) Features $\left\{F\left(x_{i}^{t}|\theta\right)\right\}|_{i=1}^{N_{t}}$ of images in the target domain are extracted using the model $F(\cdot|\theta)$ pre-trained in the source domain;

2) By using the generated features $\left\{F\left(x_{i}^{t}|\theta\right)\right\}|_{i=1}^{N_{t}}$, $x_{i}^{t}$ images are clustered into $M^{t}$ classes. Let $\tilde{y}_{i}^{t}$ denote the pseudo label generated for image $x_{t}$;

3) Using the generated pseudo-labels, the model parameters $\theta$ and a learnable classifier $C^{t}:\rightarrow f^{t}\left\{1,\cdots,M_{t}\right\}$ of the target domain dataset are fine-tuned by the classification loss and triplet loss denoted as:

where $\left\|\cdot\right\|$ denotes $L2$-norm distance, subscripts ${i,p}$ and ${i,n}$ represent the positive and negative feature index in each mini-batch for the sample $x_{i}^{t}$, and $m$ denotes the triplet distance margin.

This feature embedding function can be applied to the target gallery set, $X_{g}^{t}=\{x_{g_{1}}^{t},x_{g_{2}}^{t},...x_{g_{N_{g}}^{t}}\}$ of $N_{g}^{t}$ images, and the target query set $X_{q}^{t}=\{x_{q_{1}}^{t},x_{q_{2}}^{t},...x_{q_{N_{q}}^{t}}\}$ of $N_{q}^{t}$ images. During the evaluation, we use the feature of a target query image $F(\theta;x_{q_{i}}^{t})$ to search similar image features from the target gallery set. The query result is a ranking list of all gallery images according to the Euclidean distance between the feature embedding of the target query and gallery data, i.e.,

The feature embeddings are supposed to assign a higher rank to similar images and keep the images of a different person a low rank.

In this section, we present the proposed Face-body and Spatial-temporal Associated Clustering (FSAC) method. As mentioned above, manga characters have special features, such as spatial-temporal relationships and unique signatures of the characters, which may help clustering.

Therefore, our core idea is to make use of the two special features to make up for the shortage of clustering. To achieve this goal, we propose two modules: the face-body combination module and the spatial-temporal relationship correction module.

Overall, the framework of our method is shown in Figure 3. The target dataset is divided into related face-set and body-set, where each face image is derived from a crop of body images and they are one-to-one correspondence. Formally, we denote the body data as $\mathbb{D}_{B}=(x_{i}^{B},k_{i}^{B})|_{i=1}^{N_{B}}$, where $x_{i}^{B}$ denotes the $i$-th body training sample, $k_{i}^{B}$ is the data associated with the spatial-temporal information of $x_{i}^{B}$, indicating the manga frame index of the image $x_{i}^{B}$, and $N_{B}$ stands for the number of body images. We also denote the face data as $\mathbb{D}_{F}=(x_{i}^{F},k_{i}^{F})|_{i=1}^{N_{F}}$, where $x_{i}^{F}$ and $k_{i}^{F}$ denote the $i$-th face training sample and the manga frame index of the training image, and $N_{F}$ stands for the number of face images.

For each part, we first extract features which are denoted as $\left\{F_{B}\left(x_{i}^{B}|\theta_{B}\right)\right\}|_{i}^{N_{B}}$ and $\left\{F_{F}\left(x_{i}^{F}|\theta_{F}\right)\right\}|_{i}^{N_{F}}$ through their networks, respectively. Then, we employ a clustering algorithm on the unlabeled data and the pseudo labels $\tilde{y}_{i}^{B^{\prime}}$ and $\tilde{y}_{i}^{F^{\prime}}$ are generated under the constraint of a face-body graph. After that, based on these pseudo labels, we optimize the network parameters $\theta$ and learnable classifier $C^{t}:f^{t}\rightarrow\left\{1,\cdots,M^{t}\right\}$ jointly by the classification loss and the spatial-temporal triplet loss described in Equation 9 and Equation 10. The above steps are repeated until convergence.

In the beginning of clustering, each image is assigned to a different cluster for initialization, and then similar images are gradually clustered together by distance. The cluster ID is used as the training set label, and we expect the network to minimize the intra-cluster variance and maximize the inter-cluster variance.

Following [28, 15, 16], we generate soft pseudo labels for face and body after clustering, which are expressed in terms of probability. For each image, the probability that it belongs to the $m$-th class is defined as

where ${\{a_{i}\}}_{i=1}^{N}$ is the reference agent that stores the feature of each class and $N$ is the number of classes.

The core component of the face-body combination module is the face-body graph, which builds a map of the body and face images of the same character from the same manga frame. For face images $x_{i}^{F}$ and body images $x_{i}^{B}$ with mapping relationship, their soft labels are respectively expressed as $\tilde{y}_{i}^{F}$ and $\tilde{y}_{i}^{B}$. The reference agents corresponding to the soft label with the maximum probability are denoted as $m_{i}^{F}$ and $m_{i}^{B}$. The graph enables us to synthesize the face and body results in the following way:

The refined labels $\tilde{y}_{i}^{B^{\prime}}$ and $\tilde{y}_{i}^{F^{\prime}}$ will be further used in the fine-tune network.

In this module, we add spatial-temporal constraints on the basis of the classic triplet loss [10] and optimize network parameters jointly with the classification loss (i.e., the cross-entropy loss).

The classic triplet loss and classification loss are shown in Equation 1 and Equation 2. In the traditional triplet loss, the positive and negative sample pairs are selected based on the feature distance from the anchor pictures. However, in the UManga-ReID task, we find that spatial-temporal information may help clustering, so we set up spatial-temporal correlation rules to help select positive and negative sample pairs.

In order to add spatial-temporal constraints on the triplet loss, we define a spatial-temporal distance $d^{st}$ to find the positive and negative pairs of the sample $x_{i}$. The spatial-temporal distance between $x_{i}$ and $x_{j}$ is denoted as

where $\sigma$ indicates the threshold of manga frame index difference and $\eta$ denotes the penalty factor of characters in the same manga frame in the form of

The sum of the spatial-temporal distance and the $L2$-norm distance of images constitutes the total distance $d_{i,j}$, denoted as

which is used to search for pseudo positive and negative pairs in the triplet loss.

Meanwhile, we still use the $L2$-norm distance of the pseudo positive and negative pairs to calculate the triplet loss.

Taking the face dataset as an example, the spatial-temporal triplet loss and classification loss shown below are are still calculated in the classical way, but differ in some details.

In classification loss, $\tilde{y}_{i}^{F}$ uses the refined label after face-body combination, and in spatial-temporal triplet loss, positive and negative pairs are selected based on spatial-temporal distance $d_{st}$.

We use the labeled real-person face and body datasets as source domains for pre-training. These two pre-trained models $F_{F}\left(\cdot|\theta_{F}\right)$ and $F_{B}\left(\cdot|\theta_{B}\right)$ will be used as the CNN initialization for the face and body in the subsequent loop iteration steps, respectively.

In the iterative process, we first extract image features and generate soft pseudo-labels after clustering. Then, following Equation 5 in the face-body combination module, we generate the same hard pseudo-label $\tilde{y}_{i}^{B^{\prime}}=\tilde{y}_{i}^{F^{\prime}}$ for the corresponding face and body images. After that, we employ the spatial-temporal triplet loss and the classification loss to update their respective networks.

Such operations of generating pseudo labels by clustering and learning features with pseudo labels are alternated until the training converges. The entire process is summarized in Algorithm 1.

The experimental dataset includes four parts: the source face dataset and source body dataset for pre-training face and body networks, as well as the target face dataset and target body dataset for fine-tuning on the manga domain. The details are as follows.

The comparisons with the state-of-the-art methods on Manga109 are shown in Table II.

The baseline method in the table uses the general Unsupervised Domain Adaptation(UDA) pipeline illustrated in Sec. III-A in [7], which adopts features extracted from the pre-trained network to cluster and uses the classification loss and classic triplet loss to update the network, excluding modules for face-body combination and temporal & spatial relationship modification. Strong-baseline is the follow-up update of this method, changing the steps of pre-training to synchronous training of shared weights.

Although these methods achieve outstanding performance in person re-identification, our manga-content-based method surpasses them by a large margin in the manga context in terms of mAP and CMC. On the face dataset, we achieve the best results with mAP of 32.2% and rank-1 score of 78.4%, which outperforms the state-of-the-art result by 7.8% and 14.5% respectively. On the body dataset, we also surpass the state-of-the-art result by 4.3% and 8.3% on mAP and rank-1 score respectively. When tested with a mixed dataset of Manga109Face and Manga109Body, we achieve 5.0% and 14.0% of improvement in rank-1 accuracy and mAP, respectively.

In addition, although [25] also studies this topic, their results were not compared because the dataset is pre-screened and we have no access to their code and experimental data.

We conduct ablation studies on the two key parts: the spatial-temporal triplet loss and the face-body graph. For the model without the spatial-temporal triplet loss, the classic triplet loss [10] is used. As shown in Table IV, the comparison results on the two datasets validate the effectiveness of the spatial-temporal module, which can be seen from the mAP difference of 5.2% on the face dataset and 2.2% on the body dataset.

In addition, for the model without the face-body graph, we adopt the hard pseudo labels generated by clustering instead. The results in Table IV also show the effectiveness of the face-body combination module.

To make the experimental analysis more comprehensive and to explore the details of the effects of the spatial-temporal relationship, we conducted experimental tests on the parameters $\sigma$ and $\eta$ in Equation 6. The experimental results are displayed in Figure 7.

We visualize the query results of some images from the face dataset in Figure 5 and displayed the rank-1 to rank-5 results of the search image (or query image). Frame sequence indices below images express their temporal positions. From this figure, we may further discover the detailed impact of our approach and discuss the following questions:

Have spatial-temporal relationships helped clustering? As we can see, most negative results are far away from search images in the temporal dimension. From the perspective of method design, we shorten the distance of neighboring frame images while lengthening the distance of the images in the same frame. Therefore, we can avoid the clustering of remote images and images in the same frame as much as possible, so as to overcome the style limitation to a certain extent. The results of lines 1 and 2 in Figure 5 validate the effectiveness of our method.

Has face-body combination helped clustering? Characters in lines 3 and 4 in the Figure 5 have similar clothes, and they are wrongly clustered together in the baseline without face-body combination. The face-body combination module considers two parts of the body, making it less possible to miss the signatures of characters. Hence, the experimental results corroborate our idea.

Moreover, we utilize t-SNE [26] to visualize the feature embeddings of the clusters by plotting them to the 2D map, which is illustrated in Figure 6. Although it is still difficult to distinguish some very similar images, our method has a satisfactory effect on the whole.