Making Person Search Enjoy the Merits of Person Re-identification

In the proposed TDN, to make the one-step person search model learn more discriminative Re-ID features from the powerful Re-ID teacher model, we design a new one-step base framework, namely the Partially Disentangled Framework, and propose a Knowledge Transfer Bridge (KTB) module. As shown in Fig. 2, the proposed TDN consists of a Re-ID branch, a detection branch and a teacher branch. The Re-ID branch and detection branch form the Partially Disentangled Framework which carries out the person detection and re-identification subtasks. The KTB module is introduced between the Re-ID branch and the teacher branch to help the one-step person search model learn from the teacher model more easily. The details are introduced in the following parts.

Partially Disentangled Framework. For the Re-ID branch, we adopt the ResNet50-IBN-a [34] as the backbone. The ResNet50-IBN-a is divided into two parts, Part-1 and Part-2. The Part-1 is composed of layers from the conv1 to conv3_x, while the Part-2 includes layers from conv4_x to conv5_x. Following the Part-2, a Global Average Pooling (AVG) layer and a Batch Normalization (BN) layer are used to generate the final Re-ID features. Finally, a softmax layer classifies person identities in the training phase.

For the detection branch, we do not adopt the widely-used one-step framework [24] shown in Fig. 1(b). As analyzed in the introduction, the detection and Re-ID subtasks have a conflict of optimization targets in this widely-used one-step framework [24] which shares all convolutional networks between two subtasks. The conflict prevents the Re-ID subtask from learning discriminative Re-ID features for each detected person RoI. To address this problem, we propose to partially disentangle the two branches to retain both relevance and independence between them. Specifically, in the proposed architecture, only the Part-1 network is shared between the detection and Re-ID branches (as illustrated in Fig. 2), while Part-2 is only kept for the Re-ID branch to learn discriminative features for person RoI detected by the detection branch.

Based on the feature maps generated by the Part-1, a standard RPN [23] is built to generate possible person RoIs. And the RoI Align [35] operation is adopted to crop the feature maps of the RoIs into fixed-size feature maps ($7\times 7\times 512$). These 3-D feature maps are then flattened into 1-D feature vectors before fed into the RoI Head. The RoI Head is composed of two fully connected (FC) layers with 1024 units. Finally, a classification layer and a coordinate regression layer are employed to predict the probability that the RoIs are persons and refine their box coordinates, respectively. The Non Maximum Suppression (NMS) is applied to the proposals generated from the RoI Head to output the final predicted person RoIs. The Part-2 only extracts Re-ID features for the final predicted person RoIs rather than all the RoIs from the RPN, which improves the running speed.

Teacher-guided Learning with KTB. For the teacher branch, we choose a state-of-the-art Re-ID model, the strong Re-ID baseline model [15], as the default teacher model. To pre-train the Re-ID teacher model, we construct a Re-ID style training set by cropping all labeled person patches from the panoramic images and resizing them to $256\times 128\times 3$ ones with the bilinear interpolation. Then, we pre-train the strong Re-ID teacher model following its training settings [15]. During the training of the TDN, the pretrained teacher model is utilized to extract features for each predicted person RoI to supervise the learning of the one-step person search model. Please note that the teacher branch is only used in the TDN training phase, which will not bring an extra computational burden during inference.

In the one-step person search framework, different from traditional fixed-size Re-ID inputs, features are extracted from predicted person RoIs with arbitrary sizes. As analyzed in [3], the CNN model is not robust against scale variations, making it difficult to directly transfer the strong prior knowledge learned on fixed-size inputs to the Re-ID branch in the one-step person search model. To ensure effective knowledge transfer from the strong Re-ID teacher model to the Re-ID branch, we further propose the KTB module for the Re-ID branch to reduce scale variations between the Re-ID teacher model and the Re-ID branch. Specifically, denote a predicted person RoI with varying sizes as $x$, we introduce $\hat{x}$ as a supplementary input of the Re-ID branch, where $\hat{x}$ is a fixed-size ($256\times 128\times 3$) image interpolated from $x$ by the RoI Align operation. Then, the KTB module processes the supplementary input $\hat{x}$ to generate the $32\times 16\times 512$ RoI transferring feature maps. The KTB has the same structure as Part-1 in the Re-ID branch, but does not share parameters with Part-1 to avoid entanglement with the detection task.

Next, we combine the RoI base feature maps extracted from the Part-1 with the RoI transferring feature maps generated from the KTB module. As the RoIs predicted by the detection branch are of variable sizes, to allow fusion with the fixed-size RoI transferring feature maps from the KTB, the RoI base feature maps are interpolated to $32\times 16\times 512$ ones using the RoI Align operation (as shown in Fig. 2). Then, the RoI base feature maps and the RoI transferring feature maps can be fused by pixel-wise addition. The fused feature maps contain feature maps extracted from the same inputs as the teacher model, and consequently can help the Re-ID branch to learn from the teacher model more easily by reducing scale variations.

Finally, in the Re-ID branch, the Part-2 together with the ”GAP+BN” layer is utilized to further process the fusion feature maps to generate the final 2048-D Re-ID features, followed by a softmax classifier. Additionally, as shown in Fig. 2, there are two RoI Align operations in the Re-ID branch. One is performed on the base feature maps, and the other is conducted on the input scene image. Please kindly note that these two RoI Align operations use the same person RoIs predicted by the detection branch in both the training and inference stages.

Training Loss. Following the Faster R-CNN [23], we employ the RPN training losses ($L_{cls}^{rpn}$ and $L_{reg}^{rpn}$) and RoI Head training losses ($L_{cls}$ and $L_{reg}$) to train the detection branch. The total detection loss $L_{det}$ is defined as:

For the Re-ID branch, we adopt the following loss function to perform knowledge transfer.

where $K$ is the number of the final predicted person RoIs in a mini-batch, $f_{t}(\hat{x})$ is the L2-normalized Re-ID supervision feature from the teacher model, and $f_{s}(x,\hat{x})$ is the L2-normalized Re-ID feature from the Re-ID branch. Besides, we also introduce the cross-entropy loss $L_{ce}$. Overall, the total loss for the Re-ID branch is as follows:

where $w$ is a weight factor to balance between the two components of $L_{reid}$. In the training phase, to let the teacher model lead the learning of Re-ID branch, we empirically set $w$ as follows:

There are labeled and unlabeled person RoIs in person search datasets. Please kindly note that the $L_{t}$ loss is applied to both the labeled and unlabeled person RoIs, while the $L_{ce}$ loss is only applied to the labeled person RoIs.

The total training loss for the proposed TDN is the combination of the detection loss $L_{det}$ and the Re-ID loss $L_{reid}$, which is mathematically formulated as follows,

Different from the traditional Re-ID task, the person search task provides the panoramic scene images which contain context persons (co-travelers). Mazzon et al. [36] point out that persons are likely to walk in groups, which means that context persons of the query person are likely to simultaneously appear with him in more than one gallery scenes (as shown in Fig. 3). Thus, the context persons of the query person can be regarded as additional cues to help search for the query. In this paper, we propose an effective ranking method to take advantage of the context cues for better ranking results.

Given a query scene image containing the query person $q$ and context persons $q_{1},\cdots,q_{N}$ and a gallery scene including gallery persons $g_{1},g_{2},\cdots,g_{M}$, it is a conventional practice to take the most similar gallery person (denoted as $g$) to the query person $q$ as the candidate for this gallery scene image and rank all candidates from all gallery scene images based on the individual similarity. This common procedure ignores the valuable information of context persons. It is intuitive that if more context persons of the query person $q$ appear in the gallery scene image with him, it is more likely that the $g$ is a right candidate of $q$. However, it is still unknown how to utilize the context persons to rectify the matching degree of the target query-gallery pair $(q,g)$.

In fact, one one hand, when the target query-gallery pair $(q,g)$ is not the same person, any context person should make few contributions to the rectification, even if this context person appears in both the query and gallery scenes. On the other hand, when the target pair $(q,g)$ is the same person, the contributions a context person makes to the rectification should depend on the possibility that this context person also appears in the gallery scene. To model this relationships between the query person $q$ and a context person $q_{i}$, we propose the co-occurrence index score based on the target query-gallery pair $(q,g)$ and a context query-gallery pair $(q_{i},g_{j})$. The co-occurrence index score $S_{co}(q,q_{i})$ for the query person $q$ and a context person $q_{i}$ is defined as follows:

where $S_{I}(q,g)$ is the cosine similarity between person $q$ and $g$, the gallery person $g_{j}$ is the most similar person to the context person $q_{i}$ in the gallery scene image, and $b$ is a positive threshold to keep confident context query-gallery pair $(q_{i},g_{j})$.

Based on the co-occurrence index score, we define the context similarity score $S_{C}(q,g)$ for the target query-gallery pair $(q,g)$ as follows:

During the ranking phase, the context similarity score $S_{C}(q,g)$ is regarded as an additional cue to rank the gallery persons. Specifically, we combine it with $S_{I}(q,g)$ to define the final similarity score $S(q,g)$.

where $\lambda\in(0,1)$ balances the relative importance between $S_{I}(q,g)$ and $S_{C}(q,g)$.

PRW dataset [19] is collected in a university with six cameras. It provides 11,816 video frames with 43,110 bounding boxes. Among them, 5,704 frames are split into the training set, 2,057 person bounding boxes are taken as the query set, and 6,112 frames are taken as the gallery set. The training set contains 15,575 bounding boxes from 482 labeled identities and many unlabeled bounding boxes. Both the training set and gallery sets include plenty of unlabeled bounding boxes which have different identities from those labeled ones. Different from the CUHK-SYSU, the search scope is the whole gallery set, making it more challenging than the CUHK-SYSU.

CUHK-SYSU dataset [24] contains video frames from the street snap and movies. It provides 18,184 frames with total 96,143 bounding boxes containing both labeled and unlabeled identities. The training set consists of 11,206 frames containing 15,080 bounding boxes from 5,532 labeled identities and many unlabeled bounding boxes, while the testing set is composed of 2,900 query persons and 6,987 gallery frames. For each query person, it provides several gallery subsets with various gallery sizes.

The widely-used Cumulative Matching Characteristic (CMC) and mean Average Precision (mAP) in the Re-ID task are adopted to evaluate the performance of person search. For each query, its AP is scaled by its recall rate. Then the mAP is computed as the average of all APs across all query persons.

Following the settings proposed in [15], we pre-train the strong Re-ID teacher model. When training the TDN, the Re-ID teacher model is frozen. The TDN is trained for total 40 epochs with batch size 16 using the SGD optimizer. The initial learning rate is 0.01, decayed to 0.001 and 0.0001 in the 25-th and 30-th epochs, respectively. For the RPN in the detection branch, the anchor sizes are set to 4, 8, 16, and 32 for each location on feature maps. Since the height of a person is not less than the width in a scene image, the anchor aspect ratios are empirically set to 1, 2, and 3. The height and width of an input scene image are scaled by the same factor to make the shorter side not less than 640 pixels or the longer side not more than 960 pixels. During reference, the predicted person RoIs with foreground scores lower than 0.5 are removed, and only person RoIs whose Intersection over Union (IoU) with ground truth bounding boxes larger than 0.5 are regarded as true detection results. In addition, the parameters $b$ in (6) and $\lambda$ in (8) are 0.3 and 0.2, respectively.

In this section, we compare the proposed method with some state-of-the-art methods. Experimental results are reported in Table I. In this section, the strong Re-ID model [15] is taken as the default teacher model in our method.

Results on PRW. As shown in Table I, the proposed method achieves 70.2% mAP and 93.5% top-1 recognition rate, outperforming all the compared state-of-the-art methods. Among one-step state-of-the-art methods, the IGPN obtains the highest performance (47.2% mAP and 87.0% top-1). Compared to the IGPN, the proposed method achieves 23.0% mAP and 6.5% top-1 improvement. We also make comparisons with some two-step methods among which the TCTS method achieves the best performance. According to Table I, our proposed method outperforms TCTS by 23.4% in mAP and 6.0% in top-1 recognition rate respectively. It is observed that the proposed method surpasses all the listed state-of-the-art methods by a large margin, especially in mAP, which shows that the proposed method can retrieve more positive gallery persons at the top of the ranking list.

Results on CUHK-SYSU. As shown in Table I, the proposed method achieves 94.9% mAP and 96.3% top-1 recognition rate, surpassing all the compared state-of-the-art methods. For example, compared to TCTS which achieves the best performance (93.9% mAP and 95.1% top-1) among all the compared state-of-the-art methods, the proposed method achieves 1.0% gains in mAP and 1.2% gains in top-1 performance.

Besides, we also conduct experiments to evaluate the influence of various gallery sizes. The gallery sizes vary from 50 to 4,000. Detailed comparison results are presented in Fig. 4. On one hand, we observe that the performance of all methods decreases with gallery size increasing. On the other hand, the proposed method generates much higher and more stable performance compared with other state-of-the-art methods, especially when the gallery size is large. The robustness against various gallery sizes is valuable in practical scenarios.

In this section, experiments are first conducted to validate the effectiveness of each proposed component. Then, the comparison results between the one-step and two-step frameworks are reported. Next, the running speed of the previous one-step framework and the proposed TDN is compared. Finally, the parameter analysis is conducted to study the influence of hyper-parameters in the proposed RCP.

Effectiveness of TDN. To validate the effectiveness of the proposed TDN, we re-implement the widely-used one-step person search framework (denoted as OIM*) proposed in [24]. For a fair comparison, we replace the original ResNet50 backbone with the ResNet50-IBN-a, and divides the backbone into two parts from the conv3_4. As shown in Table II, on the PRW dataset, the proposed TDN surpasses the OIM* by 45.9% in mAP and 42.8% in top-1. On the CUHK-SYSU dataset, the proposed TDN outstrips the OIM* by 12.3% in mAP and 11.7% in top-1. This demonstrates that the proposed TDN one-step person search framework can significantly boost the one-step person search performance by integrating the advanced Re-ID model.

In the proposed TDN, to help the one-step person search model learn discriminative Re-ID features from the powerful Re-ID teacher model, we propose to partially disentangle the detection and Re-ID subtasks and introduce the KTB module. To validate their effectiveness, we construct a baseline model by removing the teacher branch and KTB module in the proposed TDN. The baseline is also trained end-to-end. Experimental results are reported in Table III. Compared to the OIM*, the proposed baseline achieves better results on both datasets, especially on the more challenging PRW dataset (15.2% mAP and 28.3% top-1 improvement). This demonstrates that the proposed Partially Disentangled Framework can help the one-step person search model learn more discriminative Re-ID features, and can be used as a new strong baseline framework for the one-step person search model in place of the OIM one-step framework. Besides, we also employ the Re-ID teacher model to guide the learning of the OIM* model. As shown in Table III, compared to the “OIM* only w/ teacher”, the “Baseline only w/ teacher” achieves much higher performance on both datasets. For example, on the PRW dataset, the “Baseline only w/ teacher” surpasses the “OIM* only w/ teacher” by 12.0% in mAP and 7.5% in top-1. This further validates that the proposed Partially Disentangled Framework is effective to help the one-step person search model learn from the powerful Re-ID teacher model.

Without the KTB module, the performance of TDN drops from 68.7% to 63.0% in mAP on the PRW dataset and from 93.8% to 93.1% in mAP on the CUHK-SYSU dataset. This validates that the KTB module is crucial to help the one-step person search model learn from the teacher model. When the Re-ID teacher branch is removed, the performance of TDN suffers heavy losses. On the PRW dataset, the mAP drops from 68.7% to 55.2% and the top-1 drops from 91.9% to 89.5%. On the CUHK-SYSU dataset, the mAP drops from 93.8% to 90.0% and the top-1 drops from 94.9% to 91.8%. This shows that it is effective to improve the one-step person search performance by learning from the powerful Re-ID model.

Effectiveness of RCP. As shown in Table II, by introducing RCP in the ranking phase, the person search performance further witnesses a gain of 1.5% in mAP and 1.6% in top-1 recognition rate on the PRW dataset, and 1.1% in mAP and 1.4% in top-1 recognition rate on the CUHK-SYSU dataset. It demonstrates that the proposed RCP can effectively improve performance by exploiting context information.

Enjoying the merits of person Re-ID. In this paper, the TDN is proposed to make the end-to-end person search enjoy the merits of person Re-ID researches. To demonstrate that the proposed TDN is able to realize this purpose, besides employing the strong Re-ID baseline model proposed by [15] as the teacher model, we also adopt some other state-of-the-art Re-ID models as the teacher model, the MGN [44] model and AGW model [45]. Experimental results are reported in Table IV. It is observed that the proposed TDN achieves much higher performance on both person search datasets with any one of the three state-of-the-art Re-ID models as the teacher model when compared to the baseline model.

The MGN model is designed to extract multiple granularity locals features as well as a global feature for a person patch. The final feature of a person patch is obtained by concatenating the global feature and multiple granularity global features. Although the TDN has no local branches to learn local features for person RoIs, it can still learn discriminative Re-ID features with the guidance of the MGN teacher model. The AGW model integrates the Non-local Attention Block into the backbone networks to capture non-local relations and replaces the widely-used max-pooling or average pooling with a learnable generalized-mean pooling layer to capture the fine-grained discriminative features. Even if the TDN has no Non-local Attention Blocks and the generalized-mean pooling layer to learn the non-local relations between pixels and fine-grained features, it can also learn discriminative Re-ID features from the AGW teacher model. These experimental results demonstrate that the proposed TDN can significantly boost the performance of the end-to-end person search by enjoying the merits of state-of-the-art person Re-ID models.

One-step vs. two-step. A straightforward solution to person search is the two-step methods which employ a person detector to detect person RoIs and use a person Re-ID model to extract feature representations for the predicted person RoIs. To validate that the proposed one-step person search model TDN is a better solution to person search task compared to the two-step ones, we construct several two-step methods by training a Faster R-CNN person detector and employing different Re-ID teacher model to extract features for the predicted person RoIs. Comparison results are reported in Table V. All the three two-step methods achieve pretty high performance on both datasets with the advanced Re-ID models as the feature extractor. It is also observed that these two-step methods outperforms most state-of-the-art person search methods, which shows that the state-of-the-art Re-ID models are very powerful when applied to person search task. However, when adopting any one of these advanced Re-ID models as the teacher model, the proposed TDN realizes higher performance than the corresponding two-step method. It validates that the proposed TDN can not only learn from the powerful Re-ID teacher models but also can be more excellent than the teacher by exploiting the correlations between two subtasks in the end-to-end one-step framework.

Running Speed. Most of the previous one-step methods adopt the Faster R-CNN-based one-step framework (denoted as OIM*) and design more additional modules based on it. This means that the running speed of these methods is slower than the OIM*. Thus, in Table VI, we only compare the running speed of the OIM* and the proposed TDN. It is observed that the proposed TDN runs three times as fast as the OIM*. In the OIM* framework, the second part has to process the RoI feature maps from all the RoIs generated by the RPN (e.g. 128 RoIs per image). However, the RoIs generated by the RPN contain a lot of invalid proposals which do not contain persons or are removed after the NMS. The computation on invalid RoIs greatly lowers the running speed. In contrast, in the proposed TDN, the Part-2 in the Re-ID branch only processes the final detections generated by the lightweight detection branch for each scene image. Assuming 10 final predicted person bounding boxes in each frame on average, we quantitatively compare the computation complexity of main parts for two one-step frameworks. As shown in Table VII, the computation of the proposed TDN is much lower than the previous one-step framework OIM*. Thus, the proposed TDN can run much faster than the OIM*.

Parameter Analysis. We conduct experiments to evaluate the impact of parameters $b$ in (5) and $\lambda$ in (6).

As shown in Fig. 5, on the PRW dataset, both the mAP and top-1 can obtain stable gains when the threshold $b\leq 0.6$, and when $b>0.6$, the mAP and top-1 begins to decrease slightly. On the CUHK-SYSU dataset, when $b\leq 0.3$, the performance achieves the highest improvement, and when $b\geq 0.3$, the improvement becomes smaller. Generally, the proposed RCP ranking method is not sensitive to the threshold $b$. In a large reasonable range, any value of $b$ can bring performance gains. It is recommended to set $b$ to 0.3, considering the performance on both datasets.

Fig. 6 shows the impact of $\lambda$. On the PRW dataset, when $\lambda\in[0.1,0.3]$, the proposed RCP brings performance improvement. On the CUHK-SYSU dataset, any value of $\lambda\in[0.1,0.9]$ can improve the performance. Experimental results show that the proposed RCP is not sensitive to $\lambda\in[0.1,0.3]$ on two datasets. In this paper, we set $\lambda$ to 0.2 for both datasets.

To further validate the effectiveness of the proposed TDN model as well as the RCP ranking method, some visualization results are reported in Fig. 7 and Fig. 8. Compared to the previous widely-used OIM*, the proposed TDN can retrieve more true candidates at the top of the ranking list. When the proposed RCP ranking method is applied, the quality of the ranking list is further improved. These visualization results demonstrate the effectiveness of the proposed TDN and RCP.