Domain Adaptive Person Search

With the development of deep learning and large scale benchmarks [39, 44], person search [4] has recently become a popular research topic. Existing fully supervised person search models can be divided into two-step and one-step frameworks. Two-step frameworks typically consist of separately trained detection and ReID models [34, 21]. Zheng et al. [44] make a systematic evaluation on different combination of detection and ReID models. Wang et al. [34] solve the inconsistency between detection and person ReID tasks. One-step frameworks [6, 26, 41] design a unified model to jointly solve detection and ReID tasks in an end-to-end manner, making the pipeline more efficient. Yan et al. [43] introduce a graph model to explore the impact of contextual information for identity matching. Chen et al. [6] disentangle the person representation into norm and angle to eliminate the cross-task conflict. Li et al. [26] develop a sequential structure to reduce the low-quality proposals. Several recently studies [40, 20] adopt the weakly supervised setting without no accessible person ID labels. In this work, we explore a novel person search setting to generalize labeled source to unlabeled target domain without any bounding boxes and ID labels annotation.

Unsupervised domain adaptation (UDA) ReID [7, 10, 28, 15, 17, 32, 42] typically trains a model with labelled source domain and transfers to the target domain under the unsupervised setting. Mainstream UDA ReID methods can be divided into two categories. The first category employs generative adversarial networks [19] to mitigate the style discrepancy and translate the labelled source domain data into the target domain [7, 10, 28]. For the second category, they generate pseudo labels by clustering [15, 17, 32] or assigning soft labels [35] on target domain, and use these pseudo labels to further supervise target domain training. Recently, pseudo label-based methods raise more attention due to their superior performance. However, UDA ReID requires the cropped images, which cannot be directly extended to adaptive person search due to the lack of bounding boxes on target domain. To address this, we propose a dynamic clustering strategy to generate high-quality pseudo boxes to facilitate target domain training.

Existing Domain Adaptive approaches object detection can be categorized into three main branches, including adversarial-based methods [33, 8, 45, 37, 31], discrepancy-based methods [24, 2, 3] and reconstruction-based methods [27, 1, 11]. Adversarial-based methods utilize a domain discriminator to distinguish the domain of input data, the adversarial training is performed to encourage domain confusion between the source domain and the target domain. The discrepancy-based strategy utilizes the unlabeled target domain images to fine-tune the detector, and further followed by mean-teacher learning [2] or auto-annotation [3]. The reconstruction-based approaches bridge the domain gap by reconstructing the source or target samples, which is usually realized by image-to-image translation [1, 27]. In this work, we consider the conflicts between sub-tasks of person search, and develop a task-sensitive alignment module to alleviate such conflicts.

The general pipeline of the proposed DAPS framework is illustrated in Fig. 2. Given the input images from both the source and the target domain, the image-level feature maps are extracted with a backbone network. Then, these features are input into the Region Proposal Network (RPN) to generate candidate bounding boxes, which are subsequently fed into the ROI-Align layer to represent instance-level feature maps. To close the domain gaps for the downstream detection and ReID tasks, we design a domain alignment module (DAM) to align both image-level and instance-level features from different domains.

Subsequently, the domain-aligned instance-level feature maps are input into both the detection and the ReID branch. Since the ground-truth bounding boxes are not available in the target domain, the model will generate different pedestrian detection results for each training epoch. Therefore, it is infeasible to follow the traditional UDA ReID methods, which generally perform clustering on a fixed size of instances to generate pseudo labels. To address this issue, we design a novel dynamic clustering strategy, which continuously associates the bounding boxes generated from consecutive epochs, to guarantee the stability of instance-level ReID features. Based on dynamic clustering strategy, we further introduce the hybrid hard case mining and the target domain detection refinement to sufficiently take advantage of the unlabeled training data.

Image-level Alignment. As discussed in [36, 8, 7, 10], minimizing domain discrepancy is beneficial for both sub-tasks of person search, and an effective way is to guide the model to learn domain-invariant representation. Motivated by the recent progress in domain adaptive detectors [8, 45, 37, 31], where intermediate features are imposed with image-level alignment constraints, we introduce a domain alignment module into our DAPS framework. As shown in Fig. 2, DAM employs a patch-based domain classifier to predict the domain where the input feature comes from. A min-max formulation is adopted to misdirect the domain classifier and encourage domain-invariant representation learning.

Suppose we have $N$ training images $\{{I}_{1},...,{I}_{N}\}$ with corresponding domain labels $\{{d}_{1},...,{d}_{N}\}$. Particularly, ${d}_{i}=0$ indicates that image ${I}_{i}$ comes from the source domain, while ${d}_{i}=1$ denotes the target domain. We denote the backbone of DAPS as $\mathrm{\Phi}$ and the image-level domain classifier as $\mathrm{D}_{g}$, and further represent the domain prediction result of input ${I}_{i}$ as ${p}_{i}$. We apply a cross entropy loss to perform domain alignment in an adversarial training manner:

We have tried to conduct image-level alignment on different intermediate features and multi-scale alignment, but achieve no better results.

Task-sensitive Instance-level Alignment. As illustrated in Fig. 3, our framework consists of two head networks, where the detection performance mainly depends on the first standard Faster R-CNN [18] head, while the NAE [6] head is highly relevant to ReID. When the scale of the source domain is much smaller than the unlabeled target, the target pseudo bounding boxes predicted by detector trained on the source can be severely overfitted to the smaller domain, but no reliable target detection guidance can relieve this issue. When the target is much smaller, pseudo target ID labels can be easily obtained by clustering, but these might provide insufficient generalizing for the ReID sub-task.

According to the characteristics of the up- and down-stream tasks, we propose the task-sensitive instance-level alignment module by balancing the alignment weight on instance-level features for both sub-tasks. Suppose we have ${K}_{1}$ instances in the standard head and ${K}_{2}$ instances for the NAE head, two domain classifiers $\{{D}_{i}^{d},{D}_{i}^{r}\}$ are built in the same way with image-level alignment, and the domain predictions of the two local classifiers are denoted as $\{{p}_{i,1}^{d},...,{p}_{i,K_{1}}^{d}\}$, $\{{p}_{i,1}^{r},...,{p}_{i,K_{2}}^{r}\}$, respectively. The instance-level loss can be formulated as:

where $j\in\{1,...,{K}_{1}\}$, and $k\in\{1,...,{K}_{2}\}$. The source and target domain contains $N_{s}$ and $N_{t}$ images respectively, and the balancing factor $\lambda$ is obtained by

where the $\sigma\left(\cdot\right)$ is Sigmoid function to normalize the domain scale ratio. Moreover, we impose a L2-norm regularizer to ensure the consistency between image-level and instance-level classifiers.

Dynamic Clustering. UDA ReID models typically employ the clustering strategy (e.g., DBSCAN) to generate pseudo labels for the target domain instances, and employ memory-based losses [17] for metric learning. However, without ground-truth bounding boxes on target domain, the instances can be only generated from the detection results, which varies with the training process. This makes it infeasible to directly apply typical clustering approach to DAPS. To address this issue, we propose a novel dynamic clustering strategy to make full use of the detection results for continuous ReID training.

As illustrated in Fig. 4, an asynchronized training strategy is introduced to progressively update pseudo bounding boxes with the selected proposals as ground-truth boxes on the target domain. Specifically, for the beginning $\alpha$ epochs, DAPS is trained only on the source dataset labeled with both bounding boxes and ID labels. After that, we maintain a bounding box memory $\mathbf{M_{B}}=\{{B}_{1},...,{B}_{N_{t}}\}$ and a feature vector memory $\mathbf{M_{V}}=\{{V}_{1},...,{V}_{N_{t}}\}$, corresponding to each of $N_{t}$ target domain images. At the start of each subsequent epochs, DAPS filters out high-confidence candidate proposals $\{{c}_{1},...,{c}_{m}\}$ from $x_{i}^{t}$, and employ them to match pseudo bounding boxes in box memory $B_{i}=\{{b}_{1},...,{b}_{n}\}$ according to IOU scores. Every proposal is assigned to the most relevant box in memory if their IOU score is above the threshold, and the boxes which fail to match any qualified proposal will be removed from memory $B_{i}$. The remaining boxes in the memory are continuously updated in the Exponential Moving Average (EMA) method.

For example, suppose the proposals $c_{j1},c_{j2},c_{j3}$ are mapped to the box $b_{k}$, then $b_{k}$ is updated by:

where $\gamma\in\left[0,1\right]$ controls the update rate. Eventually, the proposals without any matched box will also be fed into the memory $B_{i}$, and further, the feature memory $\mathbf{M_{V}}$ is updated in the same way. Afterwards, we perform clustering upon $\mathbf{M_{V}}$ to obtain $N_{t}^{c}$ clusters $\{{C}_{1},...,{C}_{N_{t}^{c}}\}$ with centroids $\mathbf{W}=\{{w}_{1},...,{w}_{N_{t}^{c}}\}$, and $N_{t}^{o}$ instances $\mathbf{F}=\{{f}_{1},...\,{f}_{N_{t}^{o}}\}$ not belonging to any cluster. By extracting the identity features $\mathbf{V}$ in the source domain, we eventually build a unified memory $\mathbf{M}=\{\mathbf{V},\mathbf{W},\mathbf{F}\}$ for ReID training. The loss function can be expressed :

where ${w}$, ${f}$, and ${v}$ represents the target domain clusters, the independent instances and the source domain classes, respectively. ${z}^{+}$ is the corresponding class prototype of the input feature $x$, and $\cdot$ denotes the inner product to measure the feature similarity. The features in the memory will be updated in a momentum way during backward stage:

where $z_{t}$ is the $t$-th prototype in the memory bank $\mathbf{M}$.

Hybrid Hard Case Mining. A significant challenge for dynamic clustering is to generate reliable bounding boxes. We treat those boxes with lower confidence than a threshold as negative samples. In order to sufficiently exploit target domain information, we explore the potential of adding these “negative” samples to the ReID training. Proposals with relatively low confidence scores can be divided into highly overlapped with high-confidence boxes, the undetected persons and the background clutters. It is undesirable to enhance the ReID sub-task by treating all these proposals as negative samples. As a result, we design a hierarchical scheme to categorize the candidate proposals, and employ both of the low-confidence person proposals and non-trivial background clutters to enhance the discrimination of the ReID branch.

Specifically, proposals with confidence score in the range of $\left(\epsilon_{h},\epsilon_{p}\right)$ defined by upper and lower bound thresholds are regarded as non-trivial cases. We exclude highly overlapped duplicates by further screening IOUs with positive proposals, while the hybrid of undetected persons and the negative clutters are reserved for training. The features of these hard cases will be added to $\mathbf{M}$, and be used for the contrastive learning process. The memory loss in Eq. 5 is modified as:

where $h$ denotes the hybrid hard cases. It is noteworthy that the hybrid hard cases will be involved into the dynamic clustering before the next epoch. Once a hard case is matched with new qualified proposals, it will be treated as a positive sample and updated in a momentum way.

Target Detection Training. Although DAM can minimize the domain discrepancy, the over-fitting towards the source domain is still likely to take place, especially when the source domain data is extremely less complex and comprehensive than the target domain images. To this end, simultaneously training detection with both of the source and the target domain data is beneficial for the generalization ability of model. DAM and dynamic clustering provide relatively reliable pseudo bounding boxes, and specifically, we employ such pseudo bounding boxes after the $\alpha$ epoch to supervise detection on the target domain. In this way, the potential of unlabeled target domain images is released for both ReID and detection training.

Datasets. We employ two large-scale benchmark datasets, CUHK-SYSU [39] and PRW [44] in our experiments. CUHK-SYSU is one of the largest public datasets for person search, composed of 18,184 images and 96,143 bounding boxes from 8,432 different identities. It is divided into a training set of 11,206 images with 5,532 identities, and a test set with 6,978 gallery images and 2,900 query images. The widely used PRW dataset contains 11,816 images, 43,110 annotated bounding boxes from 932 identities. The training set includes 5,704 images and 482 labelled persons, while the other 6,112 images and 2,057 probe persons from 450 identities are adopted as test set.

Evaluation Protocols. Our experiments employ the default splits for both datasets. For domain adaptation settings, the annotations of dataset used as the source domain is accessible, while neither bounding boxes nor identity labels of datasets as the target domain are available. All evaluations are performed on the test set of target domain. We adopt the widely used mean average precision (mAP) and cumulative matching characteristic (CMC) top-1 accuracy as evaluation metrics for ReID sub-task, while average precision (AP) and recall rate are adopted as the metrics for detection.

We adopt ResNet50 [22] pretrained on ImageNet-1k [9] as our default backbone network. DBSCAN [14] with self-paced learning strategy [25] is employed as the basic clustering method, we set default hyper-parameters $\epsilon_{p}=0.95$, $\epsilon_{h}=0.8$ and $\lambda_{t}=0.1$. During training, the input images are resized to $1500\times 900$, and random horizontal flip is applied for data augmentation. Our model is optimized by Stochastic Gradient Descent (SGD) for 20 epochs. We set a mini-batch size of 4, and an initial learning rate of 0.0024, which is reduced by a factor of 0.1 at epoch 16 with warmed up in the first epoch. The momentum and weight decay are set to 0.9 and $5\times 10^{-4}$, respectively. We set the momentum factor $\gamma$ for memory updating to 0.2. The starting epoch of $\alpha$ is set to 8 when PRW is chosen as target domain, and 0 for CUHK-SYSU. All experiments are implemented with one NVIDIA Tesla A100 GPU. We also plan to support this project with MindSpore in our future work.

We perform analytical experiments to verify the effectiveness of each detailed component in our proposed framework. In table 1, we compare the baseline method with different combinations of proposed components, and report the results on both CUHK-SYSU and PRW datasets. For example, when we use CUHK-SYSU as the target domain dataset, the directly transferring baseline model achieves 52.5% mAP and 54.8% top-1. After individually adding the domain adaptive module (DAM) and dynamic clustering (DC), the performance improves 9.3% and 18.4% in terms of mAP. When combining DAM and DC, the mAP is further promoted to 72.6%, surpassing the 52.5% of baseline by a large margin. Furthermore, to make full use of the unlabeled target data, we implement the hybrid hard case mining (HM) and detection on target domain (DTD). HM improves the ReID performance by 0.6% in mAP, and DTD prominently enhances the detection branch with a 7.0% gain for AP. Eventually, DAPS achieves 77.6% mAP and 79.6%top-1 with all designed modules, outperforming the baseline by 25.1% in mAP, 24.8% in top-1, 22.5% in recall, and 14.8% in AP.

Effectiveness of task-sensitive instance-level alignment. To validate the effectiveness of our task-sensitive instance-level alignment design, we compare it with normal instance-level alignment, which conducts instance alignment on both head networks without balancing between them. As observed in Table 2, the task-sensitive design successfully alleviates the inner task conflicts and outperforms normal strategy by a large margin.

Effectiveness of dynamic clustering. As aforementioned, the key to utilizing unlabeled target domain data is generating reliable pseudo bounding boxes. To validate the quality of the pseudo bounding boxes we use, we compare different strategies of obtaining bounding boxes, and the results are reported in Table 3. We first measure the performance achieved by using ground-truth bounding boxes for training the ReID task. Furthermore, we report the performance achieved by directly employing the qualified proposals before each epoch, which is denoted as ‘static’ in Table 3. The results reveal that our proposed dynamic clustering strategy can generate trustworthy pseudo bounding boxes to achieve comparable performance with using ground-truth boxes.

Effectiveness of asynchronized training. We conduct experiments for influences by the training stage hyper-parameter $\alpha$ on final performance. As shown in Table 4, when PRW is adopted as the target domain, the best performance is achieved with $\alpha=8$, while with $\alpha=0$ for CUHK-SYSU. The results might be counterintuitive but indeed validate our task-sensitive motivation. For smaller source dataset, even limited additional target information might be helpful for cross-domain generalization. In contrast, for larger source dataset, unreliable target proposals can be harmful for domain gap bridging.

Analysis on hyper-parameter $\epsilon_{p}$. We visualize the influence of hyper-parameter $\epsilon_{p}$ in Fig. 5. We observe that the value of $\epsilon_{p}$ influences the ReID performance to a large extend, and the best performance is achieved with $\epsilon_{p}=0.95$. From Fig. 5b, it can be observed that the selection of $\epsilon_{p}$ is a trade-off between recall rate and proposal quality. Setting it to a extremely high value leads to discarding useful proposals, while a low threshold will induct clutters to undermine the quality of clustering.

Since no existing person search methods with such domain adaptation settings can be directly compared, we further compare DAPS with fully supervised methods in Table 5, including both of the two-step methods and one-step ones. It is surprising that our framework even surpasses some supervised methods. For example, DAPS outperforms MGTS [5], OIM [39], IAN [38], NPSM [29] and CTXGraph [43] on PRW. The comparison with the state-of-the-art fully supervised methods indicate that there exists a large performance gap, and we hope our work will encourage more explorations for this setting. Moreover, to make measure the theoretical upper limit of DAPS setting, we train some state-of-the-art method with both datasets in a supervised manner, and more details are described in the supplementary material.

The comparisons with existing weakly supervised methods are shown in Table 6, and we also present the results of training R-SiamNet with both datasets in the weakly supervised manner. When evaluated on the PRW dataset, DAPS outperforms all existing weakly supervised methods by a significant margin. For the CUHK-SYSU dataset, DAPS still underperforms the state-of-the-art weakly supervised models, which is mainly caused by the limitation brought by detection capabilities. As mentioned in Sec. 4.1, the images and identities in PRW are prominently fewer than those in CUHK-SYSU, and this further leads to the poor detection performance of adopting CUHK-SYSU as target domain.

To better illustrate the distributions of our hybrid hard cases, we visualize some qualitative results from both datasets in Fig. 6. As is observed, the hybrid hard cases consist of undetected persons (column a), highly overlapped human crops (column b) and background clutters(column c,d). These qualitative results demonstrate the diversity of our hybrid hard cases, and validate the rationality of adding such cases to the memory bank.