Quality-aware Part Models for Occluded Person Re-identification

One main challenge for occluded ReID is to identify visible body regions. Most existing works utilize visibility cues provided by outside tools [21, 22, 6, 23, 24, 25]. Here, we divide occluded ReID methods into two categories depending on whether or not outside tools are required during training and testing.

The first category of methods employs outside tools in both the training and testing stages [14, 15, 16]. For example, Miao et al. [14] utilized pose landmarks to identify visible local patches and only adopt commonly visible local patches of one image pair for matching. Wang et al. [15] utilized pose landmarks to learn high-order relation and topology information of the visible local features, so as to better match probe with gallery images. Gao et al. [16] employed graph matching and utilized pose landmarks to self-mine part visibility scores. They then match probe and gallery images by calculating the part-to-part distances in visible regions. However, in addition to its extra computational cost, another key downside of this approach is that external tools may not be reliable when encountering complex occlusions, as illustrated in Fig. 1(c-d).

The second category of methods avoids using outside tools in the testing stage [26, 27, 28, 29]. For example, under the guidance of human masks, He et al. [26] designed an occlusion-sensitive foreground probability generator that enables the model to focus on non-occluded human body parts. He et al. [27] further combined pose landmarks and human masks to generate spatial attention maps that guide discriminative feature learning. These approaches can reduce the impact of occlusions on the extracted features. Despite the convenience this affords during testing, this approach still relies on the visibility information of body part for each training image.

Moreover, two very recent works [30, 31] try to predict visible scores without outside tools. VPM [31] classifies each pixel in the image into one body part. However, it is designed for the partial ReID problem and cannot be directly used to solve the occluded ReID problem. Specifically, VPM classifies each pixel in the partial image into one body part, assuming there is no occlusion in the partial image. ISP [30] performs cascaded clustering on feature maps to generate the pseudo-labels of human parts for each pixel. However, ISP contains no strategy to handle the occlusion problem between pedestrians. In addition, the clustering process is time-consuming.

In a departure from existing works, our approach aims to predict the part quality rather than visibility in an end-to-end framework. And we do not employ any outside tools or require any quality annotations in either the training or inference stages. Moreover, our approach is robust to complex occlusions. In conclusion, QPM is a powerful and efficient model.

Due to their powerful representation ability, part-based methods are popular for ReID. Depending on the way to obtain body part locations, we divide existing works into three categories.

In part-based ReID methods, it is a common practice to concatenate part features as the final representation [23, 3, 33, 36, 39, 40, 41, 42]. However, it is less effective for occluded ReID, as it ignores the impact of features from occluded parts. In this paper, we accordingly propose to jointly learn part features and predict part quality. For simplicity, we extract part features from fixed part locations. If part locations are provided by outside tools or attention modules, the performance of our approach can be further promoted.

As there is no annotation of part quality scores provided, we cannot impose a direct supervision signal on the predicted part quality. Recent works for video-based recognition have revealed that framewise features and quality scores can be jointly optimized with the same identification [44, 45] or metric learning loss [46]. In these works, e.g., QAN [46] and MG-RAFA [44], the quality scores are utilized to aggregate multiple frame-level features into a single video-level feature. However, this approach cannot be directly used in image-based ReID, as recognition is based on each single image. Inspired by these works, we propose to jointly optimize part features and part quality scores using triplet loss. Accordingly, the quality scores in QPM are imposed on part distances instead of frame features. Moreover, both pair-wise part distances and pair part quality scores are included in the triplet loss, instead of using a sample’s own quality score only. In this way, our approach optimizes pair-wise part distance and pair part quality scores for occluded ReID in an end-to-end manner.

Specifically, given a probe image $I^{p}$ and a gallery image $I^{g}$ to be compared, we first calculate their part-wise cosine distances $d_{k}^{pg}$ (1$\leq$ $k$ $\leq$ $K$). Next, the part-wise distances are summed via weighted average, as follows:

where ${q}_{k}^{p}$ and ${q}_{k}^{g}$ are the quality score of $k$-th part for probe image $I^{p}$ and gallery image $I^{g}$, respectively. In this way, body parts with low quality scores will contribute less to $D^{pg}_{part}$, which weakens the impact of occlusion.

To sample sufficient triplets during training, we randomly sample $A$ images in each of $P$ random identities to create a mini-batch, the batchsize N of which is equal to $P\times A$. The triplet loss is formulated as $\mathcal{L}^{tp}_{part}=$

where $\alpha$ is the margin of the triplet constraint, while $N_{tp}$ is the number of triplets in a batch that violate the triplet constraint and $[\cdot]_{+}$ denotes the hinge loss. $D^{a_{i}p_{i}}_{part}$ and $D^{a_{i}n_{j}}_{part}$ are calculated by Eq. 2 and denote the distances of the positive and negative image pairs in a triplet, respectively. In order to reduce triplet loss, the part quality predictors have to predict lower scores to occluded parts.

Fig. 3 illustrates the structure of ISA. ISA makes use of a coarse identity-aware feature to generate spatial attention for each image. This attention suppresses occlusions caused by objects and other pedestrians, meaning that only features of spatial regions relevant to the target pedestrian are highlighted.

In more detail, we feed $\mathbf{F}$ into another $1\times 1$ Conv layer and obtain the feature maps ${\mathbf{G}}$. Global features are then extracted based on $\mathbf{G}$. Similar to the part branch, we uniformly partition $\mathbf{G}$ into $K$ parts and obtain $K$ part-level features that are denoted as $\mathbf{g}_{k}$ (1$\leq$ $k$ $\leq$ $K$) via RAP operations. We then obtain a coarse identity-aware global feature $\mathbf{h}$ by fusing $\mathbf{g}_{k}$ via weighted averaging as follows:

where ${\hat{q}}_{k}$ is the normalized part quality score produced in the part branch in Section III-A. Formally,

To further reduce the impact of occlusions, we process $\mathbf{h}$ using a simple two-layer network inspired by the squeeze-and-excitation network [47]. As illustrated in Fig. 3, the dimension of $\mathbf{h}$ is first reduced via a $1\times 1$ Conv layer which is followed by a ReLU layer, then recovered to the original dimension by means of another $1\times 1$ Conv layer. The reduction ratio of the first Conv layer is set to 4; the output of the two Conv layers are denoted as $\mathbf{\hat{h}}$ and $\mathbf{\widetilde{h}}$, respectively.

Moreover, to ensure that noisy elements in $\mathbf{h}$ are suppressed via the reduction operation, we impose a cross-entropy loss on $\mathbf{\hat{h}}$ as follows:

where ${\mathbf{\hat{h}}}^{l}$ represents the feature produced by the first Conv layer for the $l$-th image in a batch, while $\mathbf{W}^{g}$ denotes the parameters of the classification layers.

By adopting this approach, information in $\mathbf{\widetilde{h}}$ is identity-aware and noise-free. Accordingly, we employ $\mathbf{\widetilde{h}}$ to generate the spatial attention map $\mathbf{M}$ for the feature maps $\mathbf{G}$. Formally:

where $\sigma$ is a sigmoid function, while $\ast$ represents the inner product between $\mathbf{\widetilde{h}}$ and the feature vector of each pixel in $\mathbf{G}$. Accordingly, $\mathbf{M}$ is a matrix with the same height and width as $\mathbf{G}$. As identity-relevant pixels obtain a high response value in $\mathbf{M}$, we apply $\mathbf{M}$ to weigh $\mathbf{G}$ and produce new feature maps denoted as $\widetilde{\mathbf{G}}$. The above process can be summarized as follows:

where $\odot$ signifies the element-wise multiplication between $\mathbf{M}$ and each channel of $\mathbf{G}$.

As indicated in Fig. 5, the responses on $\widetilde{\mathbf{G}}$ focus primarily on the body of the pedestrian to identify after the processing of ISA. However, this does not mean that it is reasonable to extract global-level features directly from $\widetilde{\mathbf{G}}$; this is because the two images being compared may differ in terms of their occlusion locations. To ensure semantic consistency, it is essential to adaptively extract global features from the common non-occluded regions for each image pair. We designed the Adaptive Global Feature Extraction (AGFE) module with the help of the part quality score to achieve this goal.

For example, given a probe image $I^{p}$ and a gallery image $I^{g}$, we first obtain their feature maps ${\widetilde{\mathbf{G}}^{p}}$ and ${\widetilde{\mathbf{G}}^{g}}$ from the output of the ISA module. In the next step, we equally partition each of them into $K$ parts and apply the RAP operation on each divided feature maps. In this way, we obtain a set of $K$ feature vectors for $I^{p}$ and $I^{g}$, denoted as $\widetilde{\mathbf{g}}_{k}^{p}$ and $\widetilde{\mathbf{g}}_{k}^{g}$ respectively. We then adopt the part quality scores from the part branch to aggregate $\widetilde{\mathbf{g}}_{k}^{p}$ and $\widetilde{\mathbf{g}}_{k}^{g}$ and obtain the global-level features $\mathbf{h}^{p}_{g}$ and $\mathbf{h}^{g}_{p}$ for $I^{p}$ and $I^{g}$, respectively. More specifically,

where ${\widetilde{q}}_{k}$ denotes the weight for $\widetilde{\mathbf{g}}_{k}^{p}$ and $\widetilde{\mathbf{g}}_{k}^{g}$. ${\widetilde{q}}_{k}$ is computed as follows:

The classification loss for the final global representations can thus be formulated as follows:

We also apply the triplet loss to ensure that the intra-class distances are smaller than the inter-class distances. This triplet loss is similar to Eq. 3:

where $D^{a_{i}p_{i}}_{global}$ and $D^{a_{i}n_{j}}_{global}$ represent the cosine distances of global features which are obtained by Eq. 9 and Eq. 10 for the positive and negative image pairs in a triplet, respectively.

Obtaining global features from the common non-occluded regions of each image pair is usually ignored by existing works. As shown in Table V, the AGFE module significantly promotes performance for occluded ReID. This is because the AGFE module extracts semantically aligned global features.

During training, the overall objective function of QPM can be written as follows:

During the training process, the overall loss is optimized together. The parameters of the network, including those of the fully connected layers, are optimized together via gradient descent.

Many existing works design local and global branches [50, 51, 14, 15] for multi-source structural information integration. Similar to [14], there are two parts in QPM that make up the final distance between one pair of query and gallery images: namely, the distance between part-level features and the distance between global-level features. In our approach, the distance between the part-level features is computed according to Eq. 2. The global-level features with respect to the image pair are obtained according to Eq. 9 and Eq. 10. Formally,

where $\gamma$ is the weight that balances the contributions from $D^{pq}_{part}$ and $D^{pq}_{global}$. $\gamma$ is consistently set to 0.6 in this work.

In the inference stage, we first compute the distance between a query image and each of the gallery images using the part features according to Eq. 2. The body parts with low quality scores will contribute less to the distance, which weakens the impact of occlusion. In this way, we efficiently obtain the top $n$ nearest neighbors for the query image. Then, we compute the final distance according to Eq. 15 between the query image and each of the $n$ nearest neighbors. Therefore, the AGFE module hardly increases the inference time cost.

Moreover, we provide stability analysis on the performance of QPM in the supplementary material.

For the two large-scale datasets, i.e. Occluded-Duke [14] and P-DukeMTMC [20], we train QPM using their own training sets, respectively.

One recent method $\rm{ISP}^{*}$ [30] achieves strong performance with a deeper backbone model, i.e. HRNet-W32 [60]. With the same backbone model, $\rm QPM^{*}$ achieves higher Rank-1 and mAP performance than $\rm{ISP}^{*}$.

Following [16, 15, 31, 18], ReID model in this setting are trained using the Market-1501 database [61]. Then, the model is directly evaluated on the Partial-REID, Partial-iLIDS, and P-DukeMTMC databases.

As is evident from the table, for Partial-iLIDS, it can be seen that QPM outperforms all other methods that also evaluate on the original occluded images. For example, QPM beats PGFA [14] by 8.2% and 4.8% in terms of Rank-1 and Rank-3 accuracy respectively. It also outperforms one of the most recent methods using partial data, i.e. HOReID [15], by 4.7% in terms of Rank-1 accuracy. For Partial-REID, QPM outperforms all other methods using the original occluded images: for example, QPM beats PVPM [16] by 3.4% in terms of Rank-1 accuracy. While its performance is slightly lower than HOReID [15] (which evaluates on partial data), it should be noted that unlike HOReID, our method does not require either the manual removal of occluded regions during testing or any additional tools. $\rm{ISP}^{*}$ [30] achieves 65.7% Rank-1 and 75.3% Rank-3 accuracy on Partial-REID, which is lower than those by QPM. This is because the image quality of Partial-REID is poor and the image resolution is relatively low. In this case, ISP is prone to errors in pixel classification. In comparison, our method is more robust.

Ablation study are conducted on two large-scale datasets, i.e. Occluded-Duke [14] and P-DukeMTMC [20]. The experimental results on the reported two benchmarks are shown in Table V. It should be noted that Table V lists the results on Occluded-Duke under supervised setting and the results on P-DukeMTMC under transfer setting. These results show the robustness and effectiveness of our method under different experimental settings.

Moreover, we compare the performance of ‘AGFE global’ with ‘SI global’ in Table VI. ‘SI global’ means that we obtain global features for each single image (SI) using Eq. 4, without considering the difference in occlusion locations for each image pair. To facilitate a fair comparison, ‘SI global’ adopts the same loss functions as ‘AGFE global’, i.e., the cross-entropy loss and triplet loss. It is shown that the performance of ‘SI global’ drops dramatically compared with ‘AGFE global’, which suggests that Eq. 4 alone cannot promote ReID performance. This experimental result indicates that it is vital to extract semantically consistent global features for each image pair.

Finally, the combination of the part branch and the global branch, which is denoted as QPM in Table V, achieves better performance than using either one branch alone. The above comparisons justify the effectiveness of each key component in QPM.

Moreover, we illustrate the heat maps for feature maps after the processing of different attention modules in Fig. 5. We have the following observations. First, heat maps for the baseline model have high responses on both occluded and non-occluded regions; second, existing popular spatial attention models [67, 68, 69, 70] handle occlusions between pedestrians poorly; third, with the identity-aware guidance, our ISA module can well differentiate discriminative body parts from the occluded ones by both objects and other pedestrians. The above comparisons further demonstrate the effectiveness of ISA.

Three recent powerful occluded ReID approaches are compared: PGFA [14], HOReID [15], and $\rm{PGFA}^{+}$ [59]. To facilitate fair comparison, all experimental settings are consistent with the paper description. Following [14, 15, 59], the input images are resized to 256 × 128 pixels for HOReID and 384 $\times$ 128 pixels for PGFA, $\rm{PGFA}^{+}$ and QPM. The batch size is set to 64 uniformly for all methods. Comparisons are conducted on a Titan V GPU, and results are summarized in Table VIII. The inference time cost in Table VIII includes the feature extraction of the query image and the matching time of all gallery images.

As is evident from the Table VIII, the model size of QPM is much smaller since it does not need an additional detection model. In addition, QPM has a faster training and testing speed. Specifically, our test time is only 60% and 24% of PGFA [14] and HOReID [15], respectively. This is because QPM does not require time-consuming human key point extraction. Although QPM uses less additional information during training and testing than PGFA [14] and HOReID [15], it still has significant performance advantages. Accordingly, the above comparisons demonstrate that the proposed QPM model is both compact and efficient.