Self-similarity Driven Scale-invariant Learning for
Weakly Supervised Person Search

Different from the fully supervised person search, only the bounding box annotations are accessible in the weakly supervised setting. Firstly, we propose the scale-invariant loss to address the scale variance problem by hard exemplar mining. In addition, because the existing cluster based methods may incur noisy pseudo labels, we further propose a dynamic threshold based method to obtain pseudo labels, which is jointly used with cluster based method.

The general pipeline of the framework is illustrated in Fig. 2. Our detection part is based on Faster R -CNN [31], a widely used object detection baseline. As aforementioned, scale variation problem is a severe obstacle and will further affect the procedure of the pseudo label prediction and the subsequent person matching . To address it, we propose the SSL that consists of multi-scale exemplar branch and main branch. The main branch locates the persons firstly, and extracts the RoI features by the RoI-Align layer with the localization information. The multi-scale exemplar branch is used to obtain the features of the same person with different scales. Specifically, the multi-scale cropped images with background filtering [23] and scene images are fed into the two branches for scale-invariant feature learning.

To enhance the reliability of the pseudo labels, we propose a dynamic threshold based method to obtain the instance level pseudo labels. At the same time, the DBSCAN based method is also used to obtain complementary cluster level pseudo labels.

In this section, we adopt a Scale Augmentation strategy to obtain multi-scale exemplars and propose a Scale-invariant Learning to learn scale-invariant features by hard exemplar mining.

Scale Augmentation. Given a scene image $I$, we obtain the cropped image of the $i$-th person $x^{i}_{o}$ with the given localization annotation $gt_{i}$.

Then, we apply a binary mask [23] filtering the background to obtain person’s emphasized region:

$$x^{i}_{s}\longleftarrow\mathcal{T}(x^{i}_{o}\odot\rm mask,s),$$

(1)

where $\odot$ means pixel-wise multiplication, and $\mathcal{T}_{s}$ is the bilinear interpolation function that transforms the masked image to the corresponding scale $s$.

Scale-invariant Loss. When we have obtained multi-scale exemplars by scale augmentation, we use them to learn the scale-invariant features in the guidance of multi-scale exemplar branch which takes the exemplars as the inputs and aims to extract multi-scale features. At the same time, our main branch takes the scene image as input, aiming to detect persons and extract corresponding re-id features.

In a scene image, assuming there are $\mathbf{P}$ persons, we can get corresponding cropped images by the bounding box annotations, and we augment each bounding box to $\mathbf{K}$ different scales. Thus, we totally obtain $\mathbf{P(K+1)}$ cropped images with different scales. The scale-invariant loss can be formulated as follows:

$$L_{scale}^{b}=\frac{1}{P}\sum_{i=1}^{P}([\ m+\max\limits_{s=1\dots K}D_{s}^{i\rightarrow{i}}-\min\limits_{\begin{subarray}{c}j=1\dots P\\ s=1\dots K+1\\ j\neq i\end{subarray}}D_{s}^{i\rightarrow{j}}]_{+}+\gamma D_{o}^{i\rightarrow{i}}),$$

(2)

with

$$\begin{split}D_{s}^{i\rightarrow{i}}&=D(f_{i},f_{i}^{s}),\\ D_{s}^{i\rightarrow{j}}&=D(f_{i},f_{j}^{s}),\\ D_{o}^{i\rightarrow{i}}&=D(f_{i},f_{i}^{o}),\end{split}$$

(3)

$$L_{scale}=\frac{1}{B}\sum_{b=1}^{B}L_{scale}^{b},$$

(4)

where $B$ denotes the batch size, $D(\cdot)$ measures the squared euclidean distance between two features, $f_{i}$ stand for the instance feature extracted from the main branch. $f_{j}^{s}$ and $f_{i}^{s}$ stand for the multi-scale features of the $i$-th and $j$-th persons, respectively. And $f_{i}^{o}$ means for the original scale feature of the $i$-th person. $m$ denotes the distance margin and $\gamma$ denotes the regularization factor. Furthermore, the obtained features are processed by $l2$-norm.

An intuitive illustration of $L_{scale}$ is presented in Fig. 3. $L_{scale}$ tries to select the most difficult positive and negative ones from multi-scale exemplars for a query image. It learns the scale-invariant features by decreasing the distance with the corresponding hardest positive exemplar and increasing the distance with the corresponding hardest negative exemplar. Meanwhile, it is notably that the original scale cropped image could be aligned better with the instance feature. Thus, we additionally constrain the distance between the instance feature and its corresponding original scale feature to make the model focus on the foreground information and extract body-aware features.

In weakly supervised settings, we do not have the ID annotations of each person. Thus, it is very important to predict pseudo labels and its quality will affect the subsequent training for discriminative re-id features.

As shown in Fig. 2, we maintain two extra memory banks $\mathcal{M}_{I}\in\mathbb{R}^{N\times d}$ and $\mathcal{M}_{M}\in\mathbb{R}^{N\times d}$ to store the features extracted from the main branch and multi-scale exeplar branch, separately. Where $N$ is the number of samples in the training data set and $d$ is the feature dimension. For the latter branch, we extract features $[f_{i}^{o},f_{i}^{1},f_{i}^{2},f_{i}^{3}]$ corresponding to scale-o, scale-1, scale-2, and scale-3, and obtain the average feature $f_{i}^{h}=\mathrm{mean}(f_{i}^{o},f_{i}^{1},f_{i}^{2},f_{i}^{3})$. For the main branch, we extract the instance feature $f_{i}$ from the scene image. After each training iteration, $\mathcal{M}_{M}$ and $\mathcal{M}_{I}$ are updated as:

$$\begin{split}\mathcal{M}_{M}[i]&=\lambda\cdot f_{i}^{h}+(1-\lambda)\cdot\mathcal{M}_{M}[i],\\ \mathcal{M}_{I}[i]&=\lambda\cdot f_{i}+(1-\lambda)\cdot\mathcal{M}_{I}[i],\end{split}$$

(5)

where $\lambda$ is the momentum factor and set to 0.8 in our experiments.

To obtain reliable pseudo labels, we use the multi-scale features to generate pseudo labels for clustering and multi-label classification. The multi-scale features are obtained as follows:

$$\mathcal{M}=\textrm{mean}(\mathcal{M}_{M},\mathcal{M}_{I}).$$

(6)

Dynamic Pseudo Label Prediction. Suppose we have a training set $\mathcal{X}$ with $N$ samples, we treat the pseudo label generation as an N-classes multi-label classification problem. In other words, the $i$-th person has an N-dim two-valued label $Y_{i}=[Y_{i1},Y_{i2}\dots,Y_{iN}]$. The label of the $i$-th person can be predicted based on the similarity between its feature $f_{i}$ and the features of others. Based on the $\mathcal{M}$, the similarity matrix can be obtained as follows:

$$S=\mathcal{M}\mathcal{M}^{T}=\left[\begin{array}[]{ccc}s_{11},&\dots,&s_{1N}\\ \vdots&\ddots&\vdots\\ s_{N1},&\dots,&s_{NN}\end{array}\right],$$

(7)

and with it, we can get two-valued labels $Y$ matrix with a threshold $t$:

$$Y_{i,j}=\left\{\begin{array}[]{cc}1&s_{ij}\geq t\\ 0&s_{ij}<t\end{array}\right..$$

(8)

However, the multi-label classification method is sensitive to the threshold. An unsuitable threshold can seriously affect the quality of label generation, i.e., the low threshold will introduce a lot of noisy samples, while the high threshold omits some hard positive samples. Thus, we further adopt an exponential dynamic threshold to generate more reliable pseudo labels. That is,

$$t=t_{s}+\alpha\cdot e^{\beta\cdot e},$$

(9)

where $t_{s}$ is the initial threshold, $\alpha$ and $\beta$ are the ratio factors, and $e$ stands for current epoch number. So far, we can use the dynamic threshold to get the label vector for each person at each iteration by Eq. 8.

We define the positive label set of the $i$-th person $P_{i}$ and negative label set $N_{i}$. To make the pseudo label more reliable, we further process the label based on the hypothesis: persons in the same image can not be the same person. For the $i$-th person, we can get its similarity vector $S_{i}$ by Eq. 7. We sort the $S_{i}$ in descending order and get the sorted index:

$${SI}_{i}=\mathop{\arg\mathrm{sort}(s_{ij})}_{j\in P_{i}}\ \ \ \ \mathrm{w.r.t.},1\leq j\leq n.$$

(10)

Then, we traverse the label $\mathcal{Y}_{i}$ by the $SI_{i}$. If the $j$-th person is predicted to be the same person with the $i$-th person, i.e., $\mathcal{Y}_{i,j}=1$. We consider the other persons belong to the same image with the $j$-th person can not have the same ID with the $i$-th person, and set these labels to 0.

Besides, for the cluster level, based on the $\mathcal{M}$, we adopt DBSCAN with self-paced strategy [12] to generate cluster-level pseudo labels.

$\mathcal{M}_{M}$ and $\mathcal{M}_{I}$ based re-id feature learning. As aforementioned, we use the $\mathcal{M}$ to generate reliable pseudo labels and calculate the loss function on the two branches with $\mathcal{M}_{I}$ and $\mathcal{M}_{M}$, separately. The instance level multi-label learning loss function can be formulated as:

$$\begin{split}L_{ml}(\mathcal{M}^{*},f^{*})=\sum\limits_{i=1}^{q}[\frac{\delta}{|P_{i}|}\sum\limits_{p\in P_{i}}||\mathcal{M}^{*}[p]^{T}\times f^{*}+(-1)^{Y_{i,p}}||^{2}+\\ \frac{1}{|N_{i}|}\sum\limits_{v\in N_{i}}||\mathcal{M}^{*}[v]^{T}\times f^{*}+(-1)^{Y_{i,v}}||^{2}],\end{split}$$

(11)

where $\mathcal{M}^{*}\in\{\mathcal{M}_{I},\mathcal{M}_{M}\}$, $f^{*}\in\{f_{i},f_{i}^{h}\}$, $q$ is the number of persons in a mini-batch and $\delta$ is used as a balance factor of the loss. The total dynamic multi-label learning loss can be formulated as follows:

$$L_{DML}=L_{ml}(\mathcal{M}_{I},f_{i})+L_{ml}(\mathcal{M}_{M},f_{i}^{h}).$$

(12)

In general, our SSL is trained in an end-to-end manner by using the following loss function:

$$L=L_{scale}+(L_{cluster}+L_{DML})+L_{det},$$

(13)

where $L_{det}$ stands for the detection loss used in SeqNet [24] and $L_{cluster}$ denotes to the contrastive learning loss used in CGPS [40].

In the inference phase, we only use the main branch to detect the persons and extract the re-id features which are further used to compute their similarity score.

CUHK-SYSU [39] is one of the largest public datasets for person search. It contains 18,184 images, including 12,490 frames from street scenes and 5,694 frames captured from movie snapshots. CUHK-SYSU provides 8,432 annotated identities and 96,143 annotated bounding boxes in total, where the training set contains 11,206 images and 5,532 identities, and the test set contains 6,978 images and 2,900 query persons. CUHK-SYSU also provides a set of evaluation protocols with gallery sizes from 50 to 4000. In this paper, we report the results with the default 100 gallery size.

PRW [47] is collected in a university campus by six cameras. The images are annotated every 25 frames from a 10 hours video. It contains 11,816 frames with 43,110 annotated bounding boxes. The training set contains 5,401 images and 482 identifies, and the test set contains 6,112 images and 2,507 queries with 450 identities.

Evaluation Protocol. We adopt the Cumulative Matching Characteristic (CMC), and the mean Averaged Precision (mAP) to evaluate the performance for person search. We also adopt recall and average precision to evaluate person detection performance.

We adopt ResNet50 [17] pre-trained on ImageNet [7] as our backbone. We set the batch size to 2 and adopt the stochastic gradient descent (SGD) algorithm to optimize the model for 26 epochs. The initial learning rate is 0.001 and is reduced by a factor of 10 at 16 and 22 epochs. We set the momentum and weight decay to 0.9 and $5\times 10^{-4}$, respectively. We set the hyperparameters $m=0.3,\ t_{s}=0.6,\ \alpha=0.1,\ \beta=-0.1$ and $\gamma=0.05$. We employ DBSCAN [10] with self-paced learning strategy [12] as the basic clustering method, and the hyper-parameters are the same as [40]. In our experiments, the scale-1, scale-2, scale-3 are set to 112$\times$48, 224$\times$96 and 448$\times$192, respectively. For inference, we resize the images to a fixed size of 1500 $\times$ 900 pixels. Furthermore, We use PyTorch to implement our model, and run all the experiments on an NVIDIA Tesla V100 GPU.

Baseline. We adopt a classical two-stage Faster R-CNN detector as our baseline model. Following SeqNet [24], we adopt two RPN structure to obtain more quality proposals. Furthermore, we adopt DBSCAN [10] with self-paced strategy [12] to generate pseudo labels and optimize the model with the contrastive learning loss in [40].

Effectiveness of Each Component. We analyze the effectiveness of our SSL framework, and report the results in Tab. 1, where DML denotes dynamic multi-label learning and SL means scale-invariant learning.

Firstly, we can see that the baseline model achieves 18.8% mAP and 67.0% top-1 on PRW. With the SL, baseline obviously improves the mAP and top-1 by 7.4% and 9.4% on PRW, respectively. This improvement indicates that the proposed SSL is effectiveness to handle pedestrian scale variations. Secondly, we can observe that DML improves baseline model by 2.0% in mAP and 3.8% in top-1 on PRW dataset, and DML further improves baseline + SL by 4.5%/4.2% in mAP/top-1 and improves baseline + SL by 1.2%/1.1% in mAP/top-1 on CUHK-SYSU. This improvement illustrates the effectiveness of our dynamic multi-label classification strategy for unsupervised learning.

Effectiveness of Scale-invariant Loss. In the Sec. 3.3, we introduce using the multi-scale features to generate more reliable pseudo labels. For a fair comparison, we only use the memory bank $M_{I}$ to generate pseudo labels in our experiments. The results are reported in Tab. 2. In the method SSL w/ Original Scale, we only use the original size of the cropped person images, which is obtained by bounding box annotations directly. In the method SSL w/ One Scale, we resize the cropped persons images to 224$\times$96 pixels. We observe that the One Scale method just surpasses the Original Scale method by 0.4% and 0.3% in mAP and top-1. The SSL w/ Multi-Scale method significantly improves the performance, which achieves 27.1% in mAP and 77.4% in top-1. ${\dagger}$ means filtering the background with the method in Sec. 3.2, which makes the model concentrate more on the foreground information and obtains more discriminative features. Filtering the background information further improves the performance by 1.5% and 2.2% in terms of mAP and top-1.

Effectiveness of Pseudo Label Generation Method. To illustrate the effectiveness of DML, we count the precision and recall of the pseudo labels, and report the statistics result in Fig. 4. Compared with clustering method, the higher precision indicates that DML is more reliable, while the higher recall indicates that DML is more valid. This is because that cluster method (i.e., DBSCAN) introduces within-class noisy. Our DML can alleviate this issue. Additionally, we shows the limitation of the multi-label classification with a fixed threshold in  Tab. 3, where ML denotes generating the pseudo multi-labels with a fixed threshold. It shows that the ML is not adaptive for different datasets. Although ML with a fixed threshold achieves comparable results on PRW (29.2% mAP and 79.0% top-1), it performances poorly on CUHK-SYSU. Our dynamic threshold significantly alleviates this issue and achieves promising results on both datasets.

In this section, we compare our method with current state-of-the-art methods including fully supervised methods and weakly supervised methods.

Results on CUHK-SYSU. Tab. 4 shows the performance on CUHK-SYSU with the gallery size of 100. Our method achieves the best 87.4% mAP and 88.5% top-1, outperforming all existing weakly supervised person search methods. Specifically, we outperform the state-of-the-art method R-SiamNet by 1.4 in mAP% and 1.4% in top-1 accuracy. We also evaluate these methods under different gallery sizes from 50 to 4,000. In Fig. 6, we compare mAP with other methods. The dashed lines denote the weakly supervised methods and the solid lines denote the fully supervised methods. It can be observed that our method still outperforms all the weakly supervised with gallery increasing. Meanwhile, Our method surprisingly surpasses some fully supervised methods, e.g., [39], [42],[2] and [4]. However, there still exists a significant performance gap. We hope our work could give some inspiration to others to explore weakly supervised person search.

Results on PRW. As shown in Tab. 4, among existing two weakly supervised methods, CGPS [40] and R-SiamNet [14] achieve 16.2%/68.0% and 21.2%/73.4% in mAP/top-1. Our method achieves 30.7%/80.6% in mAP/top-1, surpassing all existing weakly supervised methods by a large margin. We argue that, as shown in Fig. 1, PRW has large variations of pedestrians scales, which presents multi-scale matching challenge, and our scale-invariant feature learning(Sec. 3.2) significantly alleviates this problem. As shown in Tab. 1, even the baseline model with our scale-invariant feature learning still outperforms CGPS 10.0%/8.4% in mAP/top-1 and outperforms R-SiamNet in 5.0%/3.0% in mAP/top-1.

Visualization Analysis. To evaluate the effectiveness of our method, we show several search results on CUHK-SYSU and PRW in Fig. 5. Specifically, the first two rows show that our method has stronger cross-scale retrieval capability compared to the baseline method. Additionally, the third row shows that our SSL extracts more discriminative features and retrieve the target person correctly among the confusing persons gallery.

Moreover, we visualize the feature distribution with t-SNE [33] in Fig. 7. The circle denotes to the small scale persons whose resolution less than 3600 pixels, the square denotes to the large scale persons whose resolution larger than 45300, and the cross denotes to the medium scale persons whose resolution is between 3600 and 45300. Different colors represent different person identities. It illustrates that our method generates more consistent features across different scales. For more search results and visualizations, please refer to the supplementary materials.