Dynamic Patch-aware Enrichment Transformer for Occluded Person Re-Identification

Person re-identification is a task that searches for or identifies a target person from multiple camera views. Existing methods can be roughly categorized into traditional [19, 20] and deep learning [21, 22] approaches. As a representative work, Yang et al. [19] propose a unique color descriptor and generate feature representation in the color space. With the emergence of large-scale datasets together with modern GPUs, deep learning techniques have been extensively adopted in the person re-ID field, among which part-based methods have shown competitive performance by leveraging fine-grained information of a human body. Sun et al. [2] presents a simple and efficient part-based baseline convolutional network that employs a uniform partition strategy to learn part-level features. Wang et al. [23] design a multi-branch deep network, consisting of one global branch and two local branches with varying numbers of segmentation parts. Lin et al. [24] introduce additional attribute information, such as gender, hair length, age, backpacks, and so on, to enforce model learning of local details. Attention mechanisms have been adopted in [25, 26, 27] to emphasize feature extraction on human body areas. [28, 29] fully exploit the knowledge from the source domain to address cross-domain re-ID. Tan et al. [30] incorporate spatial and temporal dual attention to refine pseudo-labels in unsupervised re-ID tasks. Xu et al. [31] introduce a novel recycling strategy for pseudo-labels, addressing both pre-clustering and post-clustering stages. Unfortunately, these above methods exhibit limited accuracy on retrieving individuals under occlusions, thereby hindering their applicability in crowded and complex scenarios.

The current mainstream treatment for such occluded person re-ID is to adopt external information, including human parsing and pose estimation. Huang et al. [12] are the pioneers in employing human parsing techniques for localizing body parts. He et al. [15] introduce a new network called Pose-Guided Feature Alignment (PGFA), which leverages pose landmarks to extract meaningful information while excluding occlusion noise. He et al. [32] introduce a new model called Foreground-aware Pyramid Reconstruction (FPR), which is dedicated to extracting features from foreground human body parts and computing matching scores between occluded pedestrians. Gao et al. [33] propose an integrated framework called Pose-guided Visible Part Matching (PVPM) to learn discriminative features using a part visibility predictor and a pose-guided attention module. Wang et al. [34] present an adaptive-direction graph convolutional (ADGC) model to learn semantic features, together with a cross-graph embedded alignment (CGEA) method for robust feature alignment. Yan et al. [35] propose a lightweight PRE to exploit local feature correlations and aggregate them without external detectors.

By leveraging transformer architecture, Jia et al. [36] propose DRL-Net through disentangled representation learning. Additionally, Wang et al. [37] propose a novel approach, called Feature Erasing and Diffusion Network (FED), which can effectively eliminate occlusion features in images. To summarize, the above methods represent significant advances in the field of occluded person re-ID, and demonstrate encouraging performance for this challenging task.

Unlike the aforementioned previous methods that rely heavily on external human detectors or predefined occlusion priors from training sets, the proposed DPEFormer does not require any additional knowledge and also enables end-to-end training. Through careful consideration and design for occlusion elimination, DPEFormer demonstrates good generalizability in handling different occlusion scenarios, and can also adapt to unseen occlusion cases effectively.

Recognizing that not all patch tokens from ViT contribute equally to the representation of a pedestrian, especially an occluded one, we propose a dynamic patch token selection module (DPSM) to exclude a certain amount of ineffective patches. In turn, DPSM aims to emphasize contributions of those advantageous ones based on specific rules to enhance model performance. Firstly, we introduce a proxy token $f_{proxy}$ that is carefully selected from all patch tokens using a similarity measure that compares with the global features, namely the class token. This proxy serves as a key link between the global and patch features and will be used for subsequent patch token selection to alleviate the impact of occlusion. The features obtained after ViT backbone can be denoted as $f=\{f_{g},f_{1},f_{2},...,f_{N}\}$, where $f_{g}$ is the class token and can be deemed as the global features for the re-ID task. Owing to that all the token features are normalized after ViT, we directly measure similarities by computing the inner product between $f_{g}$ and each patch token $f_{i}$, where $i=1,2,...,N$. Next, we identify the patch token having the greatest similarity with the class token $f_{g}$ as the proxy token $f_{proxy}$. Mathematically, the lower-index $proxy$ of $f_{proxy}$ can be formulated as:

Since $f_{proxy}$ is obtained from the strongest patch token representation and it is reasonable to assume it corresponds to the pedestrian body area, $f_{proxy}$ can be considered as a reliable feature representation. To further evaluate similarities between $f_{proxy}$ and the other tokens, we perform dot product again between $f_{proxy}$ and each patch embedding, obtaining a set of similarity scores as:

where we note that we do not yet exclude $f_{proxy}$ itself from the set, yielding $N$ scores in total, among which the maximum score equals 1.

As shown in Fig. 3, we then sort all similarity values in descending order via $\operatorname{Sort}(\cdot)$ operation as:

and calculate the first-order gradient of $\mathtt{S}^{o}$:

and let $\mathtt{Diff}$ be denoted as:

where $N$ means the total number of patch tokens. We assume that the pedestrian body and other information, such as background and occlusion, belong to distinct categories in the feature space. Our goal is to divide the patch tokens into two clusters, among which one corresponds to body information and the other is associated with background/occlusion information. To address this, we refer to $\mathtt{Diff}$ and consider the position with the maximal gradient magnitude, namely the maximal first-order difference, as the splitting point. The underlying assumption is that there should be a distinct feature transition when body features change to those tokens that are contaminated by occlusions or other interfering information. Therefore, this splitting point is chosen as:

This means those tokens corresponding to the first $k$ scores in $\mathtt{S}^{o}$ will be selected as body feature tokens to represent a pedestrian, and the rest tokens are deemed less effective and will be discarded subsequently.

Despite the above maximal gradient magnitude demonstrating certain efficacy (will be discussed in Section IV), we also observe some failure cases in practice due to the strong assumption of a clear boundary between body tokens and background/occlusion tokens. If a pedestrian presents a similar appearance to the occlusion object or background, this strategy may lead to the misclassification of similar parts as disturbance, resulting in the selection of too few patch tokens. Consequently, this reduction in the number of selected patch tokens can compromise the model’s ability to accurately capture comprehensive pedestrian information, ultimately affecting its overall performance.

To address this issue, we set up an initial minimum for $k$, denoted as $k_{min}$, to define the least number for patch selection. The value of $k_{min}$ is empirically determined (see Section IV-D) and acts as a relaxation condition. Finally, the dynamic number of the selected candidate tokens can be defined as $k\leftarrow max(k,k_{min})$. Lastly, we perform adaptive average pooling on the selected $k$ patch tokens, yielding $P$ part features denoted as $f_{{part}_{i}}$, where $i=1,\dots,P$, and together with $f_{g}$ as the outputs of DPSM.

Discussion. Actually, the proposed hard attention mechanism DPSM can be deemed as performing a two-class clustering process to divide the tokens into two classes, and the proposed algorithm is a simple tailored one to achieve such a goal. Given the clustering guidance by $f_{proxy}$, DPSM treats $f_{proxy}$ as a seed and compares all the samples with this seed. Tokens similar to $f_{proxy}$ are clustered into one class, while those far away are put into the other class. It is also worth noting that, in the view of clustering, one can also apply any clustering algorithm, like naive K-means, to bi-partition tokens $\{f_{1},f_{2},...,f_{N}\}$. However, such a way is much less efficient as we need to handle enormous training images during learning. The clustering results from K-means may also be less stable and reasonable. Therefore, we reject those clustering approaches and employ a proxy token-based sharpest gradient scheme to establish the splitting point.

To visualize the selected patch tokens, we map the selected ones back to image space, where raw image patches corresponding to the selected patch tokens are highlighted. The visualization results are shown in Fig. 1 and Fig. 4. One can observe that the majority of selected patches align with pedestrian bodies, effectively avoiding occlusions caused by other pedestrians or objects. However, DPSM sometimes fails to “perfectly” select patches. Instead, the selected patches could cover the background or some occlusion areas. One possible reason is that the global self-attention mechanism of Transformer has exchanged information across all patch tokens, making their feature representation complicated and not well aligned with the original image space, e.g., a patch token corresponds to the background of the original image may also convey pedestrian identity representation. Despite the above explanation, we observe notable performance gains after using DPSM.

To accurately identify pedestrians, it is imperative to consider both contextual information from the global features $f_{g}\in\mathbb{R}^{1\times 768}$ and detailed information from the part features $f_{{part}_{i}}\in\mathbb{R}^{1\times 768},i=1,\dots,P$. In recent work [40], a module was developed specifically for communicating information between global and part features in the multi-modal re-ID task. Inspired by [40], we propose a feature enrichment strategy for occluded re-ID. To this end, given these two types of features, we embed global context information into each local part feature to enrich representation power via FBM. Fig. 2 bottom right illustrates the structure of FBM, which we describe below using the $i$-th part features $f_{{part}_{i}}$ and the global features $f_{g}$ for simplicity.

The underlying idea of FBM is to leverage the well-known multi-head self-attention (MHSA) mechanism as an enrichment mean (Fig. 2), rather than simple point-to-point addition (i.e., $f_{{part}_{i}}+f_{g}$) or concatenation. We initially implement three $1\times 1$ convolution layers on global and part features to obtain $f_{g}^{{}^{\prime}}$, $f_{g}^{{}^{\prime\prime}}$ and $f_{{part}_{i}}^{{}^{\prime}}$. Then we split these feature vectors into 48 non-overlapping sub-parts, with each sub-part having a fixed size of 16. Such 48 sub-parts are then formulated as regular inputs to the attention mechanism. Learnable positional embeddings are added to these feature sub-parts. In contrast to the standard MHSA, we modify MHSA to derive query set $Q_{p}$ from the feature sub-parts of $f_{{part}_{i}}^{{}^{\prime}}$, and to derive key set $K_{g}$ and value set $V_{g}$ from the feature sub-parts of $f_{g}^{{}^{\prime}}$ and $f_{g}^{{}^{\prime\prime}}$. The graphic illustration is shown in Fig. 2 bottom right, and our modified MHSA enables comprehensive mutual interactions between the part and global features, instead of naive point-to-point integration performed by simple addition.

Given $Q_{p},K_{g},V_{g}$, the modified MHSA is formulated as:

where $W^{O}$ is the output transformation matrix for integrating multi-head outputs, and $head_{i}$ is computed as:

where $Q_{p}^{i},K_{g}^{i},V_{g}^{i}$ are the query, key, and value matrices of the $i$-th head, respectively. The general computation of Attention($\cdot$) is defined as:

where $\sqrt{d_{k}}$ denotes a scaling factor, and is equal to $4(\sqrt{16})$ in our case. So generally, the feature blending process in FBM is represented as:

where $\operatorname{RE}$ represents the reshape operation, which aims to maintain the same dimension as $f_{{part}_{i}}$. $\operatorname{PE}$ and $\operatorname{LN}(\cdot)$ denote position embedding and layer normalization, respectively. $\operatorname{FPR}(\cdot)$ represents feature partition and sub-part matrix formulation, namely transforming $\mathbb{R}^{1\times 768}$ into $\mathbb{R}^{48\times 16}$. Finally, the entire forward blending process of FBM is computed as:

where $\operatorname{FFN}(\cdot)$ denotes the feed-forward layer, and $f_{{fbm}_{i}}$ denotes the obtained blending features by embedding $f_{g}$ into $f_{{part}_{i}}$.

To ensure semantic diversity, we employ FBM for each part, namely, there will be $P$ FBM in total. Finally, all enriched part features are concatenated to form the final pedestrian descriptor:

where $P$ denotes the total number of parts. During the training process, $f_{g}$ is employed solely for classification and discarded in the final feature representation. The final representation $f_{final}\in\mathbb{R}^{1\times 3072}$ encompasses diverse detail-specific information from different local parts, and is also enriched by global knowledge through the proposed feature embedding via FBM.

A significant challenge faced by existing approaches in occluded person re-ID lies in the limited availability of occlusion data. In response to this challenge, researchers have explored various strategies. Zhong et al. [41] introduced a method known as Random Erasing (RE), which involves the random occlusion of a rectangular region with random pixel values. Chen et al. [42] adopted a random cut-and-paste approach, where a patch is cropped from a training image and pasted onto the input image. Following the idea of [42], Wang et al. [37] further cropped occlusion objects like bags, manually from the training set and pasted them. Such an approach can be deemed as incorporating additional prior information specifically related to pedestrian occlusion. The aforementioned approaches have made significant progress in addressing data scarcity. However, a common limitation shared by these methods is that the synthesized images appear to be coarse and not realistic. To overcome this limitation, we present Realistic Occlusion Augmentation (ROA) that significantly diminishes data redundancy and greatly enhances the realism of generated images, better simulating real-world occlusions.

The procedure of the proposed ROA is outlined in Algorithm 1, whose mask set is generated by the Segment Anything Model (SAM) [1] applied on natural images (Fig. 5). In Algorithm 1, variable $random\_area$ means the area of the bounding box of an occlusion object, and therefore, the actual occlusion area is usually smaller than this value. Following the steps in Algorithm 1, ROA is able to generate occluded images that closely resemble real-world scenarios, with occlusions manifesting in different directions and covering various regions of a pedestrian body. As shown in Fig. 5, the generated occlusion image by ROA is more natural and exhibits diverse contour details when compared to cut&paste [42].

For better training our DPEFormer model, we adopt the same memory bank strategy as in [39] to regularize feature learning. Since the learning process of DPSM may not be stable at the beginning (patch tokens reside in learning and have not yet converged), the patch selection process yields less reliable results. So we apply the memory bank to the patch tokens prior to DPSM and construct the associated contrastive loss and identity loss.

Occluded-DukeMTMC [15] represents one of the most challenging occluded person ReID datasets due to the diversity of scenes and distractions encountered. It comprises 15,618 training images of 702 persons, 2,210 query images of 519 persons, and 17,661 gallery images of 1,110 persons.

Occluded-REID [44] focuses on occluded individuals captured through mobile cameras, comprising 2,000 images spanning 200 unique identities. Within each identity, the dataset includes five unobstructed full-body images and five images featuring severe occlusions, showcasing diverse viewpoints and occlusion types.

Market-1501 [45] is one of the most well-known holistic person ReID datasets, with 12,936 training images of 751 persons, 19,732 gallery images, and 3,368 query images of 750 persons captured from six cameras. Few of images in this dataset are occluded.

DukeMTMC [46] consists of 16,522 training images of 702 persons, 2,228 queries of 702 persons, and 17,661 gallery images of 702 persons. The images are captured by eight different cameras, making it more challenging. As it contains more holistic images than occluded ones, this dataset can be treated as a holistic ReID dataset.

We compare our method with state-of-the-art methods on both occluded and holistic re-ID datasets in Table I and Table II, respectively. The backbones of compared methods are ResNet-50 [48] and Vision Transformer [18].

Comparisons on Occluded Re-ID Datasets. DPEFormer is evaluated with recent state-of-the-art occluded re-ID methods. The comparison results are presented in Table I. We can see that PAT [38] significantly improves accuracy by adopting a transformer encoder-decoder structure capable of leveraging diverse part-aware attention maps. These attention maps serve as specific part detectors and boost network performance. Our DPEFormer achieves 69.9% Rank-1 accuracy, 82.8% Rank-5 accuracy, 86.4% Rank-10 accuracy, and 58.9% mAP, which surpasses all other types of occluded methods by a substantial margin. Notably, our method does not rely on external cues. It is designed to mitigate interference of occlusion solely based on the available training data and the designed architecture.

Comparisons on Holistic Re-ID Datasets. We also evaluate our model on two holistic person Re-ID datasets and compare it with other state-of-the-art methods in Table II. While our proposed strategies and modules aim to address occlusion problems, they may not fully function well for holistic scenarios. Nonetheless, we still achieve an impressive Rank-1=95.4%/mAP=88.1% and Rank-1=90.0%/mAP=80.3% accuracy on Market-1501 and DukeMTMC, respectively, surpassing all other types of holistic re-ID methods. This comprehensive performance indicates that our proposed methods effectively exploit a robust feature representation not only for occlusion challenges but also for holistic ones.

In this section, we present a comprehensive series of ablation studies to thoroughly analyze our proposed framework.

Effectiveness of each component. In Table III, we present a series of ablation studies that investigate the individual contributions of various components: random erasing (RE) [41], realistic occlusion augmentation (ROA), the dynamic patch token selection module (DPSM), and the feature blending module (FBM). In detail, “index 1” corresponds to the baseline model, where different images of the same pedestrian are used as input for contrastive learning. Subsequently, we incrementally introduce augmentations and modules, denoted as indexes from 1 to 5: baseline + RE, baseline + ROA, baseline + ROA + DPSM, baseline + ROA + FBM, and DPEFormer, respectively. Comparing “index 0” (the baseline) to “index 1/2” (baseline + RE and baseline + ROA), we observe that the inclusion of data augmentation strategies, specifically ROA, significantly enhances performance. This improvement can be attributed to the generation of a more diverse and realistic set of augmented images, which in turn aids the model during training.

Comparing “index 3” to “index 4”, we observe that the DPSM leads to a further enhancement in representations, resulting in 1.6% improvement in Rank-1 accuracy. In Fig. 1 and 4, we provide a visual representation of the patches selected by DPSM, demonstrating its effectiveness in directing the model’s focus towards extracting human body information while filtering out tokens associated with occlusions or background elements. Furthermore, when comparing “index 3” to “index 5”, the FBM contributes to performance gains, showing improvements of 1.0% in Rank-1 accuracy and 0.5% in mAP. This underscores FBM’s ability to diversify local part features and enhance local representations by effectively fusing complementary global information. Ultimately, “index 6” (DPEFormer) achieves the highest accuracy, highlighting the efficacy of each component, both individually and in synergy.

Parameter analysis of DPSM. We investigate the impact of the patch token selection module on model performance. To this end, we introduce a fixed threshold selection strategy referred to as Fixed Threshold PSM (FPSM). In FPSM, we modify the token selection process in the Dynamic Patch Selection Module (DPSM) to a fixed number while keeping all other training strategies unchanged. Table IV presents our results. We observe an obvious trend as the number of selections increases: both mAP and Rank-1 accuracy improve. However, this trend changes after reaching a threshold of 48, where we begin to observe a gradual decrease in performance with further increases in the selection number. This observation suggests that an abundance of patch tokens, which incorporate occlusion or background information, can degrade the feature representation. Conversely, an insufficient number of patch tokens may fail to capture the necessary information required for accurate person characterization. Balancing the selection of patch tokens is a critical factor in achieving an optimal feature representation and robustness in person recognition. Therefore, it becomes essential to establish a minimum threshold for the number of patch token selections to ensure a more effective implementation of the dynamic selection strategy. Setting this threshold ensures that an adequate number of candidate patch tokens are retained, striking the right balance in the process.

We conducted experiments to determine the optimal hyperparameter, denoted as $k_{min}$, by varying its values and assessing the impact on the model performance, as presented in Table V. Notably, when there were no constraints on the minimum selection number ($k_{min}=0$), meaning tokens were selected solely based on their first gradient value, the model achieved competitive performance. As we increased $k_{min}$ from 12, we observed an improvement in Rank-1/mAP performance by 1.6%/1.3% (reaching $k_{min}=48$), indicating that this parameter became beneficial for learning discriminative features. However, further increases in $k_{min}$ led to performance degradation, as more noisy tokens, including occlusions and background information, were included. Additionally, the dynamic selection strategy outperformed the fixed number strategy presented in Table IV. This improvement can be attributed to the enhanced flexibility of the dynamic selection module, which adapts better to the varying number of characteristics in each image. From another perspective, it is worth noting that the model achieved its best performance when the hyperparameter was set to 48, as evidenced in both Table IV and Table V.

Furthermore, Table VI presents a comparison of recognition accuracy achieved by using the feature representations before and after the DPSM for inference, without the need for retraining. “Before” signifies the utilization of a pooling operation on all patch tokens to form the feature representation, while “After” denotes the use of the patch tokens selected by DPSM for pooling. It is evident that utilizing the feature representation after the DPSM results in higher accuracy, underscoring the module’s effectiveness.

Necessity for the proxy token in DPSM. We refrain from using global features as proxies due to their inclusion of information from all body parts. The results, as shown in Table VII, clearly demonstrate that employing a selected patch token as the proxy yields superior performance compared to using global features (cls token). This highlights that the selected patch token provides more representative information for token selection. The effectiveness of using patch tokens as proxies can be attributed to the elimination of discrepancies caused by redundant information present in global features.

FBM vs. REM. The efficacy of our FBM is confirmed through a comparative analysis with an alternative fusion strategy known as REM, introduced by Wang et al. [40]. All other experimental settings remain constant. The results, as presented in Table VIII, reveal that REM exhibits a decrease in Rank-1 accuracy by 1.2% and a decline in mAP by 0.5% when compared to our proposed FBM. This outcome underscores the superior capability of our FBM in effectively integrating and fusing global and local features within the context of DPEFormer.

On Realistic Occlusion Augmentation. We assess the efficacy of our Realistic Occlusion Augmentation (ROA) approach through a comparative analysis with two established methods: Random Erasing (RE) [41] and the Cut&Paste method [42]. Specifically, in the Cut&Paste configuration, we employ the same object instances as ROA to ensure a fair comparison. In this scenario, we directly extract rectangular regions from scene images instead of employing our occlusion generation method. The results of this evaluation are presented in Table X. We observed that the Cut&Paste method outperforms Random Erasing, possibly due to its ability to provide rich occlusion information. However, ROA significantly outperforms both of these augmentation methods. This substantial improvement validates the effectiveness of ROA, which leverages the Segment Anything Model (SAM) [1] to generate realistic occluded images. ROA proves to be a superior approach for augmenting occluded person images.

To ensure a fairer comparison, we replaced the occlusion augmentation in FED and the ROA in DPEFormer with the widely recognized occlusion augmentation RE. The results are also presented in Table X. We can see that our approach consistently outperforms FED, underscoring the effectiveness of our meticulously designed modules (DPSM and FBM). It is worth mentioning that FED achieves competitive performance, especially when leveraging its NPO augmentation strategy, which incorporates prior manual occlusion information from the training set.

Furthermore, to evaluate the influence of the generated occlusion mask set size on model performance, we employed varying quantities of original images choosen from SA-1B datasets. The findings, as illustrated in Table IX, reveal a performance uptick with a candidate image count ranging from 50 to 100. Optimal performance is attained at approximately 100 candidates. Beyond this threshold, augmenting the number of images yields minimal impact. At this juncture, the model is exposed to a sufficiently diverse array of occlusion shapes, and texture saturation has reached an optimal value. Hence, our investigation establishes a candidate set size of 100 as the optimal enhancement size.

About the contrastive loss. We introduced an additional contrastive loss after the Feature Blending Module (FBM). As outlined in Table XI, the results demonstrate that the inclusion of extra contrastive loss led to a decrease of 1.5% in mAP and a 3.1% reduction in Rank-1 accuracy. Our analysis aligns with the observation that body region information differs between occluded and non-occluded images. Furthermore, this discrepancy is accentuated by the DPSM. Introducing contrastive loss as a constraint in this context may hinder the learning of crucial discriminative features, potentially impacting DPSM’s performance. However, the contrastive loss applied to the memory bank serves as guidance for the Vision Transformer (ViT) backbone to extract features devoid of occlusions. This preparation enhances the subsequent processing by DPSM.

We present qualitative experimental results to demonstrate the efficacy of DPEFormer. In Fig. 4, we provide visualizations of the patch tokens selected by DPSM, mapping them back to their corresponding locations in the image space. We have highlighted the raw image patches that correspond to these selected patch tokens. These images depict instances where pedestrians and objects occlude the scene. Notably, a significant proportion of the selected patches align precisely with the pedestrians’ body regions, effectively mitigating occlusions caused by other pedestrians or objects. This visualization underscores the robustness and effectiveness of DPEFormer in handling occlusions within complex real-world scenarios.

Fig. 6 illustrates how DPEFormer overcomes occlusions by presenting some retrieval results. Each set comprises an occluded query person image displayed on the left, along with two rows of images on the right. These right-hand images represent the top-10 matches generated by the baseline model and our proposed DPEFormer. Upon observation, it is evident that DPEFormer excels at overcoming occlusions, accurately identifying images of the same pedestrian. In contrast, the baseline network exhibits heightened sensitivity to occlusions, resulting in a substantial number of false-positive matches.