Dynamic Prototype Mask for Occluded Person Re-Identification

Holistic person re-identification aims to address the problem of matching people over a distributed set of non-overlapping cameras. The prior works mainly focus on exploring the hand-craft descriptors (Ma et al., 2014; Yang et al., 2014; Liao et al., 2015) with a well-designed metric learning strategy (Zheng et al., 2012; Koestinger et al., 2012). With the resurge of deep learning, deep feature representation learning has dominated the vision tasks (He et al., 2017; Mou et al., 2020; Chen et al., 2021a; Peng et al., 2021; Zhang et al., 2021). Luo et al. (Luo et al., 2019) introduce the BN-Neck structure in the CNN-based ReID framework. The research provides a strong baseline for the holistic ReID. Chen et al. (Chen et al., 2019) introduced a high-order attention mechanism to capture and use high-order attention distributions. Zheng et al. (Zheng et al., 2019b) integrate discriminative and generative learning in a single unified network for person re-identification. Besides using the global feature representation (Luo et al., 2019; Zheng et al., 2019b; Ye et al., 2021b), employing part-level features for pedestrian image description to offer fine-grained information is also a mainstream strategy that has been verified as beneficial for person ReID. Methods like PCB (Sun et al., 2018), MGN (Wang et al., 2018), and Pyramid (Zheng et al., 2019a) horizontally divide the input images or feature maps into several parts to conduct a fine-grid representation. Most recently, we have witnessed the thriving of transformer structures from natural language processing to computer vision. TransReID (He et al., 2021) firstly takes the advantage of ViT structure and applies it to the ReID task. Although those methods reach a satisfying performance in the holistic ReID benchmarks, the widely existing occlusion condition is largely ignored. Most of the methods suffer significant performance degradation when being applied to the real-world scenarios which contain the occluded cases.

Occluded person re-identification points out the weakness of holistic ReID methods in such occluded cases. The main challenge of occluded ReID lies in the incomplete body information which can not provide high-quality feature representation. To tackle this issue, early works attempt to remove the influence of obstacles in an end-to-end framework and generate the global feature representation from the visible part. Zhuo et al. (Zhuo et al., 2018) introduce an extra occluded/non-occluded binary classification task to distinguish the occluded images from holistic ones. Chen et al. (Chen et al., 2021b) combine an occlusion augmentation scheme with an attention mechanism to precisely capture body parts regardless of the occlusion. Although this kind of work is easy to achieve and shows a good performance before, it always suffer the noise caused by the obstacles which limited the performance upper bound. Therefore, recent methods attempt to avoid such a condition with two typical strategies. The first one aims to aggregate the information from the whole image and handles the above issue by compensating for the invisible body regions by its visible near-neighbor. Wang et al. (Wang et al., 2020) utilize the high-order relation and human-topology information that is based on keypoint estimation to learn well and robustly aligned features. Hou et al. (Hou et al., 2021) propose a region feature completion module to exploit the long-range spatial contexts from non-occluded regions to predict the features of occluded regions. Though aggregating the information from the visible neighbor node can alleviate the occluded condition, this process is still facing a great challenge when lacking efficient evidence in the neighbor nodes. Meanwhile, using the key point estimate network which is pre-trained on other datasets also faces a challenge to provide reliable results when suffering a domain variation. Another strategy inherits the idea of fine-grid feature representation and aims to match the image between the visible parts. Miao et al. (Miao et al., 2019) introduce the Pose-Guided Feature Alignment (PGFA), exploiting pose landmarks to disentangle visible part information from occlusion noise. Gao et al. (Gao et al., 2020) introduce the Visible Part Matching (PVPM) model to learn discriminative part features via a pose-guided attention map. Li et al. (Li et al., 2021a) employ the prototypes to disentangle the fine-grid body part without the help of an extra network in order to achieve satisfying performance. However, most of the fine-grid methods still rely on the extra network to provide body clues and suffer the same domain variation problem. Furthermore, since the fine-grid methods demand strict part prediction to send the feature to its corresponding branch, those incorrect results will make those valuable nuances be ignored easily.

Differing from the above methods, the DPM not only escapes from the extra pre-trained networks but also simultaneously retains the global information from the whole image representation and achieves an automatic alignment.

The overview of our proposed DPM framework is illustrated in Figure  2. The DPM adopts a pre-trained ViT (Dosovitskiy et al., 2020) to extract the original feature representation from the input images. Herein, we denote the input image as $I$ with the resolution as $H\times W$, We first split the image into $D(h\times w)$ patches with the size after flatten as $x_{i}\in 1\times c$. Specifically, it can be described as:

where the $P$ and $s_{d}$ refers to the size of image patch and the step size of sliding window. After the linear projection $\mathcal{F}$, a learnable class token $x_{cls}$ is attached to aggregate the information from image patches. Before feeding into the transformer block, following the TransReID (He et al., 2021), a learnable position embedding $\mathcal{P}$ and camera embedding $\mathcal{C}$ is added to the patch embeddings to retain positional information and camera information respectively, which can be formulated as:

where the $z_{0}$ is the input of the transformer blocks. The hyper-parameter $\lambda$ is used to balance the weight of camera embedding. The query and key vectors of the last transformer block are fed into the head enrich module (HEM), which takes the advantage of the multi-attention structure to explore a diverse feature representation for the different heads. Meanwhile, the representation of class-token will be utilized to train a holistic prototype for each class. To well tackle the occluded case, the representation for image patches of the 2_st, 4_th, 10_th, and 12_th is concatenated and sent to the hierarchical mask generator (HMG) to provide the dynamic prototype mask for every single input image. Different from spatial attention methods (Chen et al., 2021b) which take effect on the feature map itself, the prototype mask is used to cut the holistic prototype to select the subspace of high-discriminative visible patterns.

Based on the two prior knowledge mentioned before, the main idea of DPM is to explore a spontaneous alignment and select a discriminative subspace for the holistic prototype to match every single image. Therefore, one of the most important parts for achieving the DPM is to generate an efficient prototype mask. In most cases, a pattern within an image is usually conducted by pixels that are spatially concentrated and form a connected component. Therefore, using a local region-based sliding window to weigh the importance of patterns is quite suitable in such an application. In order to provide a high-quality prototype mask, as shown in Figure 2, we apply a convolutional-based mask generator that can take the neighbor nodes into consideration for each patch.

In specific, after the $l_{th}$ block, HMG adopts the reshaped image representation $f_{l}\in\mathbb{R}^{h\times w\times c}$ by excluding the class-token in the feature representation $z_{l}$. Although directly using the image representation provided by the last block seems like the most intuitive strategy, as shown in Figure 3 (b), after calculating the cosine similarity between the most dissimilar image patches in every block, we observe that by passing information through the similarity among image patches, the feature representations will be smoothed and become more similar. Those representations with few discriminative make it hard to provide an efficient prototype mask. To this end, a hierarchical structure is utilized to aggregate those image representations from the shallow layers. Inspired by the success of Swin-Transformer (Liu et al., 2021) which merges the patches after the $2_{nd}$, $4_{th}$, $10_{th}$ transformer block, we also pick the image representations from these three blocks and combine them with the last block as the input for the HMG. The final prototype mask is generated as:

Herein, the $\sigma$ refers to the sigmoid function. The $\mathcal{G}$ denotes the convolutional layers of HMG and the $G\in\mathbb{R}^{1\times L}$ is a binary gate to choose the input for HMG.

Finally, the masked prototype $W_{mp}$ for $i_{th}$ input image is generated by Hadamard product of the row-extended prototype mask $M^{i}_{p}$ and the prototype weight matrix $W_{p}$ as:

Multi-head self-attention which aims to extract difference and discriminative feature representation is considered as the key component to conducting the transformer block. This kind of aggregation of different patterns suits the DPM both in training a complete holistic prototype matrix and selecting an aligned and discriminative subspace. However, as shown in Figure 3 (a), in the original transformer block, there is no explicit optimization to encourage multiple heads to aggregate more nuances in the whole image, which makes it possible for the different heads to have similar feature embedding. The similar head representations will in turn limit the DPM to select a well-aligned subspace.

On account of this condition, we introduce a head enrich module (HEM) to push multiple heads in the class token to obtain diverse patterns in the last transformer block. Generally, the multi-head attention adopts the query matrix Q, key matrix K, and value matrix V to pass the information from different patches. In the training phase, we only employ the class token as the global representation. Therefore, when ignoring the key vector of the class token itself, we can obtain the attention map of $A\in\mathbb{R}^{N\times D}$ between the class token and the image patches as:

where $N$ is the number of heads, $q^{cls}_{L}$ is the query vector of class token in the $L_{th}$ transformer block, $K^{img}_{L}$ is the query vector of image patches in the $L_{th}$ transformer block, and $N$ refers to the number of heads. To push the attention map of each head apart, an orthonormal constraint is impose as:

where the $\left\|\cdot\right\|^{2}_{F}$ is Frobenius norm, the $\mathbb{I}_{N}\in N\times N$ is the identity matrix, $\dot{A}$ is a normalized $A$ matrix with each row being L2 normalized. With $\mathcal{L}_{hem}$, the class token could provide richer representation, which not only benefit the learning for holistic prototype and the masked generator.

The DPM contains a two-branch learning framework as the original classification loss and the masked classification loss. Intuitionally, we should employ the softmax loss to optimize both two branches. However, this strategy will cause a condition that the mask can not be well learned. After adding the mask, the scale of the loss generated by the masked branch is much smaller than the softmax loss, which can not provide enough power to learn a high-quality prototype mask. Although increasing the weight of the masked branch may make sense, the softmax loss is not a strong constraint to clustering the samples. Limited constrains also limits the performance of DPM. Inspired by the progress of metric learning (Deng et al., 2019), we employ an extra angular margin in the original softmax loss to optimize the masked branch. This strategy not only balances the scale of the original branch and masked branch but also highlights the importance of learning a high-quality mask during the training phase. With the extra margin, for input $x_{c}ls^{L}$ with label $y^{i}$, the $L_{M-cls}$ can be given as:

where the $B$ and $C$ refers to the batch size and the number of class. The $m$ denotes the angular margin and $s$ is the hyper-parameter to adjust the scale.

To increase the intra-class similarity and decrease the inter-class similarity, triplet loss $\mathcal{L}_{tri}$ with online hard-mining (Schroff et al., 2015) are combined during the supervise training. Therefore, the overall loss function can be formulated as:

where the $\alpha$ and $\beta$ is the hyper-parameters to adjust the weight of $\mathcal{L}_{M-cls}$ and $\mathcal{L}_{hem}$ respectively.

Datasets. To evaluate the effectiveness of the proposed DPM, we conduct extensive experiments on four publicly available ReID benchmarks which include both occluded and holistic person re-identification datasets. The details are as follows.

Occluded-Duke (Miao et al., 2019) is a large-scale dataset collected from the DukeMTMC for occluded person re-identification. The training set consists of 15,618 images of 702 persons. The testing set contains 2,210 images of 519 persons as the query and 17,661 images of 1,110 persons as the gallery. Until now, Occluded-Duke is still the most challenging dataset for occluded ReID due to its scale.

Occluded-REID (Zhuo et al., 2018) is an occluded person dataset captured by mobile cameras. It consists of 2000 images from 200 persons, where each person has 5 whole-body images and 5 occluded person images. Following the evaluation protocol of previous works (Gao et al., 2020; Wang et al., 2020; Chen et al., 2021b), the Occluded-REID is only used as a testing set. The model used for experiments in this dataset is trained under the training set of Marker-1501 (Zheng et al., 2015).

Market-1501 (Zheng et al., 2015) is a widely-used holistic ReID dataset captured from 6 cameras. It includes 12,936 training images of 751 persons as the training set, 3,368 images of 750 persons as the query, and 19,732 images of 750 persons as the gallery.

DukeMTMC-reID (Zheng et al., 2017) contains 36,441 images of 1,812 persons captured by eight cameras, in which 16,522 images of 702 identities are used as the training set, 2,228 and 16,522 images of 702 persons that do not appear in the training set are used as the query and gallery, respectively.

Evaluation Protocol. To verify fair comparison with other methods, we adopt the widely used Cumulative Matching Characteristic (CMC) and mean Average Precision (mAP) as evaluation metrics and follow the evaluation settings provided by existing occluded methods (Wang et al., 2020; Gao et al., 2020).

Implementation details. We employ the ViT (Dosovitskiy et al., 2020) pre-trained on ImageNet (Deng et al., 2009) as the backbone network. Particularly, we resize all the input images to $256\times 128$ and adopt commonly used horizontal flipping, padding, random cropping, and random erasing (Zhong et al., 2020) as data augmentation. Following (Wang et al., 2020; Yang et al., 2021), we use extra color jitter augmentation to avoid domain variance when conduct testing in the Occluded-REID. Following the success of TransReID (He et al., 2021), we adopt a lower stride and set $\lambda$ to $3.0$. During the training stage, each mini-batch is conducted by 64 images from 4 identities. In order to strengthen the power of HMG, the training phase is divided into two-step in every iteration. In the first step, we froze the parameter of HMG to train the holistic prototype. In the second step, we froze the parameter except the HMG to train a high-quality prototype mask. During the testing phase, we apply the mask generated by the query image to the gallery images. Then the retrieval stage can still be computed in parallel for each query image. The SGD is utilized as the optimizer, in which the learning rate is initiated as $0.008$ with cosine learning rate decay. The hyper-parameters for $s$ and $m$ in Arcface loss are set to $30$ and $0.5$ respectively in training the Occluded-Duke. In training Occluded-REID, since the strong constraints will induce the overfitting easily when considering the domain variance of Market-1501 and Occluded-REID, we decrease the $m$ to $0.3$ and $\beta$ to $0.01$ in training. We implement our DPM with PyTorch and conduct all experiments on a single Nvidia Tesla A100.

Results on Occluded Datasets. To comprehensively demonstrate the performance of DPM, we evaluate DPM against the previously reported state-of-the-art methods on the Occluded-Duke and Occluded-REID in Table 1, The compared methods include holistic ReID methods (Sun et al., 2018; Suh et al., 2018; Ge et al., 2018; Zhu et al., 2020; He et al., 2021) and occluded ReID methods (He et al., 2018; Huang et al., 2018; He et al., 2019; Miao et al., 2019; Gao et al., 2020; Wang et al., 2020; Li et al., 2021a; Chen et al., 2021b; Yang et al., 2021). Obviously, the transformer-based structure (PAT, TransReID) has the advantage in solving occluded cases when compared to the convolutional neural network. The DPM also inherits this advantage and further outperforms other transformer-based methods. In the most challenging occluded ReID dataset Occluded-Duke, the DPM reaches an impressive performance, with $71.4\%$ in rank-1 and $61.8\%$ in mAP, which at least outperforms other occluded ReID methods with $6.9\%$ and $8.2\%$ in rank-1 and mAP respectively. Meanwhile, in the Occluded-REID, the proposed DPM consistently surpasses current state-of-the-art methods. Specifically, the DPM achieves $85.5\%$ in rank-1 accuracy $79.7\%$ in mAP, which improves the Rank-1 accuracy by $3.9\%$ and mAP by $7.6\%$ over the PAT.

Although not relying on the extra network to provide body clues, the DPM still achieves superior performance in the occluded ReID benchmarks.

Results on Holistic Datasets. Although occluded ReID methods mainly focused on solving the occluded ReID issue, they may suffer a performance decrease in the original holistic ReID task due to incorrect alignment or ignoring of valuable regions. Therefore, in this section, we also evaluate the proposed DPM on the holistic ReID dataset Market-1501 and DukeMTMC-ReID. For better comparison, we select five holistic ReID method (Sun et al., 2018; Wang et al., 2018; Zhu et al., 2020; Li et al., 2021b; He et al., 2021) and five occluded ReID methods (He et al., 2019; Miao et al., 2019; Wang et al., 2020; Chen et al., 2021b; Li et al., 2021a).

The results are shown in Figure 2. In the Market-1501 dataset, the DPM gets $95.5\%$ in rank-1 accuracy and $89.7\%$ in mAP. In the DukeMTMC-reID, the the DPM gets $91.0\%$ in rank-1 accuracy and $82.6\%$ in mAP. It is clear that DPM shows competitive results in both two holistic ReID datasets when compared to the state-of-the-art holistic ReID methods. When compared to the occluded ReID methods, the DPM outperforms the previous state-of-the-art methods in these two datasets. Overall, the above results show that DPM is a universal framework, which mainly aims to tackle occluded cases. But it will not destroy the performance on the general holistic ReID task.

To evaluate the influence of the proposed architectural components. We conduct a series experiments over the occluded-Duke with different settings and show the quantitative results in Table 3. The baseline uses the ViT as backbone and training with the original softmax loss $L_{cls}$ and triplet loss $L_{tri}$.

Compared to the baseline method, adding the DPM strategy highly improved the performance in both rank-1 accuracy and mAP as $1$ and $1$ respectively. After adding the hierarchical structure in the mask generator, the performance further increases from $1$ and $1$ to $1$ and $1$ in rank-1 and mAP. On the other hand, it also demonstrates that the image representation provided by the last transformer block which lacks sufficient diversity information will limit the performance of mask generator. Meanwhile, HEM also provides significant performance gains as $1$ and $1$ in rank-1 and mAP based on the above results. The experiments results indicate that all these components have made sense and satisfied their motivation in the DPM. All the components contribute to an effective framework consistently and finally result in an impressive performance.

The classification loss function for DPM. As the most important module, how to train a efficient mask generator is one of the main challenges to achieve the the DPM. In Section 3.4, we have mentioned that we utilize an extra angular margin to train the masked branch to avoid inefficient optimization for the masked branch. Therefore, in this part, we conduct a comparison to show the performance of different loss function combinations when training the class branch and masked branch. The results are shown in Table 4. Here, we use the $\mathcal{S}$ to denote the branch which is training with the original softmax loss, and the $\mathcal{A}$ to denote the branch which is training with an extra angular margin. The two baselines are training without the masked branch.

From Table 4, we can observe that adding an extra angular margin could improve the rank-1 accuracy since it can enforce the network to pay more attention to those outlier samples. However, we can also observe that this strategy can not bring such a significant increment in the mAP, which denotes that the whole distribution may not be ameliorated. Meanwhile, as we have mentioned before, adding an extra mask to select the subspace will further decrease the scale of the loss. It indicates that the mask generator just needs to output the same score for all the channels, the scale loss will still be satisfying. Therefore, as shown in Table 4, when using the same loss function in two branches, the optimization of the mask generator will be limited and the whole network will degrade to optimize the classification branch. After adding an extra angular margin in the masked branch and keeping the original classification branch, the mask generator has been encouraged to adopt a more radical optimization thus alleviating the above dilemma. As shown in the last line in Table 4, this kind of strategy obtain a significant improvement in both rank-1 accuracy and mAP accuracy.

Effect of head enrich module. The HEM aims to enrich the feature representation of multiple heads in the class token to overcome the tendency that more different head has similar representations. The HEM can help not only the training for a holistic prototype with more nuances but also the training for mask generator to select the potential subspace. In Figure 3 (a), we have visualized the cross-correlation matrix between multiple heads’ attention map to show the limitation of the original transformer block. Therefore, in this part, we visualized the cross-correlation matrix with the same input once again in Figure 4 after adding the HEM.

By adding the HEM that can explicitly encourage each head to have a different attention map, we could observe that the cross-correlation matrix between multiple heads’ attention maps has significantly decreased as shown in Figure 4. It indicates that the class token has received more diverse pattern information from the different image patches. Overall, the visualization well demonstrates the effectiveness of the HEM in such an application.

Impact of the hyper-parameters $\alpha$ and $\beta$. As indicated by the loss function in Eq. 8, we set two hyper-parameters $\alpha$ and $\beta$ to balance the weight of different components in the overall loss functions. Specifically, the $\alpha$ controls the trade-off between the holistic prototype matrix and the prototype mask, while the $\beta$ controls the correlation between different heads. Hence, in this part, we conduct empirical experiments to measure the performance of the model under different hyper-parameters settings. When discussing the $\alpha$, we select the DPM with HMG as the baseline to conduct the experiments. As we have mentioned before the performance is sensitive to the $\alpha$, a small $\alpha$ will limit the ability to learn a holistic prototype matrix while a large $\alpha$ can not provide enough power to learn a high-quality prototype mask. As shown in the Figure 5, we observe that the performance of rank-1 accuracy and mAP increase linearly when the $\alpha$ is less than $0.5$. The performance reaches the peak when the $\alpha$ is set to $0.5$ with $71.0\%$ and $61.0\%$ in rank-1 accuracy and mAP. After that, a larger $\alpha$ will decrease the performance.

Based on the model, we further discuss the influence of $\beta$. As shown in Figure 5, after adding the HEM into training, the rank-1 accuracy and mAP also get improved when $\beta$ is less than $0.15$. The best performance in rank-1 reaches when the $\beta$ is set to $0.05$ and the best mAP reaches when the $\beta$ is set to $0.10$. Considering the overall performance, we select the $\beta$ as $0.10$ in the further experiments.

Different types of DPM. In DPM, we apply the mask to the holistic prototype matrix, while attention-based strategies explore the spatial attention mask which takes effect on the input image itself to alleviate the noise caused by obstacles. Therefore, in this part, we also make discuss whether the mask should also be applied to the feature representation. Meanwhile, in this part, we also evaluate the influence of L2 normalization in the DPM. Herein, we select the complete DPM as the baseline in the comparison and give an empirical analysis on the Occluded-Duke. The experimental results are shown in Table 5.

Benefits from the great ability of transformer structure that provides an effective representation for the visible part, in both settings, applying the mask upon the feature representation can not provide an extra performance gain. On another side, under both settings, applying the mask on the prototype after the L2 normalization works better than applying the mask before the L2 normalization. Thus, we select the only prototype mask which is applied before the L2 normalization as the final model.