Erasing, Transforming, and Noising Defense Network for Occluded Person Re-Identification

Most existing re-ID studies assume that the pedestrian body is holistic and can be used to learn a complete representation. Hand-crafted feature-based methods [23, 24, 25] adopt pre-designed descriptors to capture pedestrian characteristics. For example, Farenzena et al. [23] computed maximally stable color regions and recurrent high-structured patches to extract color information and texture details, respectively. Liao et al. [24] proposed a local maximal occurrence algorithm that models salient representations to handle viewpoint and illumination changes. Moreover, Li et al.[25] combined Gaussian of Gaussian (GOG) features with dictionary learning to reduce domain discrepancies across datasets. However, with the expansion of data, the applicability of such methods becomes unsatisfactory since they fail to exploit sample distribution adequately. To overcome these limitations, deep learning-based methods [26, 27, 28, 29, 30] have emerged. For example, Yi et al. [26] first introduced deep neural networks (DNNs) for person re-ID and built a siamese model to capture pedestrian similarities. Considering the importance of fine-grained information in re-ID, Sun et al. [27] developed a part-aware feature extraction method with an adaptive partitioning strategy to align body parts. Li et al.[31] exploited features from multiple camera views to enable multi-view imaginative reasoning. Recently, additional techniques such as BNNeck [4] and GeM [5] have been developed, further advancing person re-ID tasks. Nevertheless, in crowded scenes where pedestrians are often occluded by obstacles or other persons, these methods struggle to achieve satisfactory performance. To address this, our work focuses on solving the occlusion problem in person re-ID while achieving excellent recognition rates in holistic scenarios.

Compared with holistic re-ID, occluded person re-ID is more practical and challenging, where the probe samples are occluded and the gallery database contains both holistic and occluded images. To encourage the model to focus on human body parts, Zhuo et al. [32] simulated occluded samples of holistic data using an occlusion simulator and introduced a co-saliency branch to assist the feature extractor in capturing identity-related information. Moreover, Kiran et al. [33] adopted a teacher-student manner to match the distributions of intra- and inter-class distances of occluded and holistic data, thereby separating visible regions from occluded images. To extract discriminative features of visible regions, He et al. [34] inserted a foreground-aware pyramid reconstruction module that adaptively assigns weights to body and occlusion parts. Despite their effectiveness, learning occlusion-robust representations remains a significant challenge for these methods. Our proposed ETNDNet falls into the external model-free category, however, it endows the re-ID model with the intrinsic ability to defend against occlusion perturbations.

Recent studies have focused on guiding the re-ID model to extract features from the visible region by utilizing external tools. For example, Miao et al.[35] integrated a pre-trained pose estimator into the re-ID model, generating pose-aware heatmaps that indicate if specific body parts are occluded. Similarly, Gao et al. [10] developed a pose-guided attention mechanism and a visibility predictor to end-to-end match identities. To capture high-order relation information of human parts, Wang et al. [11] treated local features learned by keypoints as graph nodes to model the topology structure of the human body. Leveraging the intrinsic connection between re-ID and human parsing, Zhang et al. [36] introduced a semantic branch to extract global and local features simultaneously. Moreover, Zheng et al. [37] proposed a pose-guided feature learning approach with knowledge distillation to obtain semantics-aligned representation. Hou et al. [12] performed person foreground segmentation and devised a spatial region feature completion module to reason about occluded areas. Nevertheless, the performance of these methods largely relies on the accuracy of pose landmarks and body parsing. Additionally, the use of the auxiliary model increases computational costs and may compromise the accuracy of visible cues. In contrast, our method overcomes these limitations and achieves state-of-the-art performance.

The application of DNNs has led to significant advancements in various computer vision tasks [38, 39, 40, 41]. However, Szegedy et al. [42] discovered that even highly performing models may produce incorrect inferences when imperceptible perturbations are applied to images. Building on this observation, Goodfellow et al. [15] introduced adversarial examples by adding noise to original inputs using the fast gradient sign method. Kurakin et al. [43] proposed a multi-step attack method that iteratively classifies attacked samples into categories with the lowest classification probabilities. For person re-ID tasks, recent studies [44, 45] have demonstrated that the re-ID system is also vulnerable to adversarial attacks, such as images of pedestrians wearing different clothes or captured by different cameras, which can lead to misidentification of individuals. Specifically, Wang et al. [44] designed the advPattern algorithm to explore the impact of clothing changes on recognition performance. To maintain visual quality in adversarial examples, Wang et al. [45] employed differentiable multi-shot sampling to control the number of malicious pixels and proposed a novel perception loss to ensure inconspicuous attacks.

To mitigate the interference of noise to DNNs, the concept of adversarial defense has been introduced. Specifically, Zheng et al. [16] proposed joint fine-tuning of pre-trained models using clean samples and their corresponding distorted copies to improve the accuracy of predicting adversarial examples. Lyu et al. [17] developed a family of gradient regularization techniques to constrain the optimization of DNNs and defend against perturbations. Gao et al. [46] enhanced model robustness by removing irrelevant features through a mask layer positioned between the feature extractor and classifier. Building on these studies, Wang et al. [47] proposed a multi-expert adversarial attack detection approach to differentiate perturbed examples for person re-ID tasks. Treating occlusion as a form of noise perturbation, Huang et al. [18] and Zhao et al. [19] drove models to extract robust features using brute-force adversarial defense that requires perturbed examples. In contrast, our method generates reasonable adversarial representations to simulate various occlusion perturbations and trains the feature extractor and identity classifier using a novel GAN-based adversarial defense paradigm, effectively defending the re-ID system against occlusion interference.

We adopt the standard re-ID model as our baseline framework, which is commonly employed for the occluded person re-ID task. This model contains a feature extractor with ResNet-50 [48] pre-trained on ImageNet [49] and an identity classifier composed of a pooling layer, a normalization layer, and a fully-connected layer. As illustrated in Figure 2, given a pedestrian image $x$, the feature extractor is responsible for learning representation, and the classifier is utilized to predict which identity the person belongs to. Let $\boldsymbol{E}$ and $\boldsymbol{W}$ respectively represent the feature extractor and identity classifier, we deploy the cross-entropy loss to optimize them:

where $N$ is the number of pedestrians, $p_{i}$ denotes the prediction logits of class $i$, $q_{i}$ is associated with the ground truth $y$:

where $\epsilon$ is a constant and set to be 0.1. The association is derived from the label smoothing (LS) [50] technique, which effectively avoids the risk of overfitting.

The above baseline branch ensures the basic discriminability of learned features. However, the model is vulnerable to occlusion interference. As mentioned before, it is feasible to train an occlusion-robust network from the perspective of adversarial defense. In this process, generating reasonable adversarial examples to simulate occlusion issues is crucial. The adversarial samples generated in [18, 19] can achieve this purpose. However, they are less aggressive and can only simulate the issue of missing information. To this end, we apply perturbation on pedestrian feature maps by erasing, transforming, and noising to generate multiple adversarial representations to simulate various occlusion issues.

Erasing. We randomly erase the feature map to acquire an adversarial representation with incomplete information, which can simulate the issue of pedestrian identity information loss caused by occlusion. Suppose the feature map is $F\in\mathbb{R}^{H\times W\times C}$, where $H$, $W$, and $C$ denote the height, width, and the number of channels, respectively. Similar to [51], we first compute the spatial size $S=H\times W$ of the feature map. Next, we randomly initialize a proportion $\alpha_{e}$ to determine the size $S_{e}=\alpha_{e}\times S$ of the erasing rectangle region $R_{e}$, where $\alpha_{e}\in[0.02,0.4]$. And then, we arbitrarily take an aspect ration $r_{e}$ and accordingly compute the height $H_{e}=\sqrt{S_{e}\times r_{e}}$ and width $W_{e}=\sqrt{S_{e}/r_{e}}$ of $R_{e}$, where $r_{e}\in[0.3,1/0.3]$. Finally, we randomly select a point $(x_{e},y_{e})$ in $F$. If $x_{e}+W_{e}\leq W$ and $y_{e}+H_{e}\leq H$, $R_{e}=(x_{e}:x_{e}+W_{e},y_{e}:y_{e}+H_{e})$ can be determined. Otherwise, the above process is repeated until an appropriate $R_{e}$ is selected. Let each pixel belonging to $R_{e}$ be 0, and the adversarial representation $F_{e}$ can be formulated as:

Transforming. Occlusion may cause the pedestrian information at a certain position on the non-occluded image to appear at another position on the occluded image. Therefore, we randomly transform the feature map to simulate such a position misalignment issue, which is an ability that the adversarial samples do not possess. Firstly, we randomly initialize a proportion $\alpha_{t}$ and an aspect ratio $r_{t}$ to determine the height $H_{t}=\sqrt{\alpha_{t}HW\times r_{t}}$ and width $W_{t}=\sqrt{\alpha_{t}HW/r_{t}}$ of areas to be transformed, where $\alpha_{t}\in[0.02,0.4]$ and $r_{t}\in[0.3,1/0.3]$. Next, arbitrarily choosing two points $(x_{t_{1}},y_{t_{1}})$, and $(x_{t_{2}},y_{t_{2}})$, if $x_{t_{1}}+W_{t}\leq W$, $y_{t_{1}}+H_{t}\leq H$, $x_{t_{2}}+W_{t}\leq W$, and $y_{t_{2}}+H_{t}\leq H$, two areas $R_{t_{1}}=(x_{t_{1}}:x_{t_{1}}+W_{t},y_{t_{1}}:y_{t_{1}}+H_{t})$ and $R_{t_{2}}=(x_{t_{2}}:x_{t_{2}}+W_{t},y_{t_{2}}:y_{t_{2}}+H_{t})$ can be located. Accordingly, we transform the pixels of $R_{t_{1}}$ to $R_{t_{2}}$ for generating an adversarial representation with misaligned information $F_{t}$.

Noising. The feature map extracted from an image occluded by obstacles and non-target pedestrians often contains some noise irrelevant to the target pedestrian, which is detrimental to the re-ID system for identification. To this end, we further perturb the feature map by noising to generate an adversarial representation with noisy information. Specifically, with the initialized $\alpha_{n}$, $r_{n}$, and $(x_{n},y_{n})$, we take a noising region $R_{n}=(x_{n}:x_{n}+W_{n},y_{n}:y_{n}+H_{n})$ and replace its pixels with random values that follow a uniform distribution. The value range of $\alpha_{n}$ and $r_{n}$, and the calculation way of $W_{n}$ and $H_{n}$ are the same as the above erasing process. Let $F_{n}$ denote the adversarial representation, it satisfies the following formula:

Note that the erased, transformed, and noised regions are the same for all images in a mini-batch, which prevents the model from being hard to converge [19].

Existing adversarial defense studies on person re-ID mainly focus on training the model jointly with adversarial samples and original inputs, which is time-consuming and poorly defensive. Inspired by the adversarial training in GAN [21], which optimizes a generator and a discriminator in an adversarial way to encourage the former to generate fake samples that make the latter can not distinguish them from the real ones. For person re-ID, we expect the feature extractor to learn a sufficiently robust representation whose perturbed version can be correctly identified by the classifier as well. To this end, with the above adversarial representations, we develop three modules with a novel GAN-based adversarial defense strategy to overcome the issues of missing information, position misalignment, and noisy information caused by occlusion. No additional networks are required, each module employs adversarial training between the feature extractor and identity classifier, simple yet effective.

Erasing Defense. Compared to the clean feature map $F$ that can be correctly identified (Baseline Branch), the adversarial representation $F_{e}$ loses some identity-related cues due to the information of $R_{e}$ being erased, leading the classifier into misidentification. Therefore, we maximize the cross-entropy loss to optimize identity classifier $\boldsymbol{W}$:

where $p_{i}^{e}$ is the probability that $F_{e}$ is recognized as category $y$.

For the feature extractor $\boldsymbol{E}$, we expect the learned feature map $F$ from it can well depict the pedestrian characteristics, even if part of the information is lost, it can be correctly identified. To achieve this goal, we utilize $F_{e}$ to update the parameters of $\boldsymbol{E}$ by minimizing the cross-entropy loss:

Transforming Defense. Given the adversarial representation $F_{t}$, since its information at $R_{t_{2}}$ is inconsistent with the information at $R_{t_{2}}$ on the feature map $F$, the classifier fails to accurately identify the target pedestrian, which can be formulated as:

where $p_{i}^{t}$ is the prediction result of $F_{t}$ belonging to the class $i$.

Conversely, if the feature map $F$ enables the classifier to correctly identify its attacked version $F_{t}$, it means that $F$ is not affected by the misaligned information. To this end, we constrain $\boldsymbol{E}$ to satisfy the following formula:

Noising Defense. In order to further endow $F$ with the ability to resist the interference of noisy information, similar to the above processes, we utilize the perturbed representation $F_{n}$ to maximize the cross-entropy loss for optimizing $\boldsymbol{W}$, and in turn, minimizing the cross-entropy loss for optimizing $\boldsymbol{E}$. Suppose that the probability of $F_{n}$ being correctly classified is $p^{n}_{i}$, the above optimization strategy can be expressed as follows:

Note that in all the above adversarial learning processes, the feature extractor is fixed when training the identity classifier and vice versa. Through continuous gaming and optimization between them, the re-ID system is able to defend against the perturbation of various occlusion issues. To gain a better understanding of the above adversarial game process, we take the erasing defense module as an example to further illustrate. As shown in Figure 3, assuming that the erased area on the feature map is the part marked by the red box. Initially, we fix the feature extractor and optimize the classifier by maximizing the identity loss. This enhances the discriminative capability of the classifier’s class prototype and improves its correct classification threshold. Next, we fix the classifier and optimize the feature extractor by minimizing the identity loss, which ensures that even if the information is erased, the feature map extracted by the feature extractor can be correctly identified by the classifier. Achieving this requires ensuring that the model can focus on non-erased areas with highly discriminative information (such as the part marked by the blue box). Through continuous adversarial gaming between the two, we can mine as much discriminative information as possible, thus protecting the re-ID model from the disruption caused by missing information due to occlusion. Similarly, our transforming defense and noising defense modules can also resist the interference of position misalignment and noisy information.

During the training stage, we utilize the baseline learning module to extract discriminative features and three defense modules to enhance the occlusion robustness of the features. Correspondingly, the total loss can be formalized as:

where $\mathcal{L}_{adv}^{e}={\operatorname{argmax}}\mathcal{L}_{ce}^{e}(\boldsymbol{W})+{\operatorname{argmin}}\mathcal{L}_{ce}^{e}(\boldsymbol{E})$, and $\mathcal{L}_{adv}^{t}$ and $\mathcal{L}_{adv}^{n}$ are similar. $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are hyper-parameters utilized to control the relative importance of the three adversarial losses. The training process is carried out in an end-to-end manner, which is summarized in Algorithm 1.

In the test phase, we extract the feature map $F$ and process it with pooling and normalization. After that, the obtained feature vector is used for inference.

Occluded-DukeMTMC [35] is a classical occluded person database derived from DukeMTMC-reID [52]. It contains 15,618 training images of 702 pedestrians and 19,871 testing images of 1,110 pedestrians. Due to various occlusions in both query and gallery images, this dataset is quite challenging.

Occluded-REID [53] contains 2,000 images of 200 identities, and each identity has five occluded images and five full-body images. Following [53, 19], we take occluded images as the queries and non-occluded images as the galleries, and randomly select half of them for training and the rest for testing.

P-DukeMTMC-reID [53] is another variant of DukeMTMC-reID that targets occluded person re-ID task. It consists of 12,927 training images of 665 identities and 11,216 testing images of 634 identities (2,163 occluded images in the query set and 9,053 holistic images in the gallery set).

Market-1501 [54] consists of 12,936 training images, 3,368 query images, and 19,732 gallery images. All images are captured from 6 non-overlapping cameras deployed at the campus. This dataset primarily targets the holistic person re-ID task due to few images being occluded.

DukeMTMC-reID is widely used to evaluate the performance of holistic person re-ID, which contains 36,411 images of 1,812 pedestrians collected from 8 cameras (408 distractor identities only appear in one camera view). All images are divided into 16,522 training samples and 19,889 test samples (This dataset is for research use only in this work).

MSMT17 [55] includes 126,441 images of 4,101 pedestrians, in which 32,621 images of 1,041 identities are used for training and the remaining are used for testing. This dataset is currently the most challenging holistic person database due to the long time span of sample collection.

More details about each dataset can be found in Table I.

Implementation Details: We conduct all experiments on one GTX3090 with the Pytorch platform. All images are uniformly resized to 256$\times$128 and subjected to random flipping, padding, cropping, and erasing to achieve data augmentation. The batch size is set to 64, each batch contains 8 pedestrians and each pedestrian has 8 images. We set the initial learning rate to $3\times 10^{-4}$ and decrease it by a factor of 0.1 at the 40th epoch and 70th epoch. The size of the feature map is $H=16$, $W=8$, and $C=2048$. The hyper-parameters are set to $\lambda_{1}=0.1$, $\lambda_{2}=0.15$, and $\lambda_{3}=0.1$. We utilize the Adam optimizer [69] to update the model parameters and train 120 epochs in total.

Evaluation Protocol: We follow the public evaluation indicators to verify the performance of the proposed method, namely Cumulative Matching Characteristics (CMC) and mean Average Precision (mAP). Note that all experiments are performed in the single query setting.

Experiments on occluded datasets: We first evaluate the superiority of the proposed method on the three public occluded datasets, namely Occluded-DukeMTMC, Occluded-REID, and P-DukeMTMC-reID. The comparison results are shown in Table II and Table III, where we report the accuracy of Rank-1, Rank-5, Rank-10, and mAP.

For Occluded-DukeMTMC, methods that rely on external models have achieved satisfactory results. For example, RFCNet [12] achieves 63.9% Rank-1 and 54.5% mAP by utilizing pose and semantic information. However, this superior accuracy is dependent on the reliability of external models and requires significant computational resources. As shown in the table, in the absence of auxiliary information, RFCNet only achieves 52.4% (-11.5%) Rank-1 and 44.8% (-9.7%) mAP. These results demonstrate the heavy reliance of such methods on external visual cues. In contrast, our method achieves comparable performance to RFCNet without utilizing external models and with minimal computational resources. Recently, model-free methods without external models have gained attention, and our proposed method falls into this category while having notable advantages in terms of performance and model simplicity. Specifically, the optimal ISP [61] predicts human body parts using clustering, avoiding the need for a pre-trained semantic parsing network, but it suffers from significant computational complexity and inferior performance compared to our method (52.3% vs. 54.7% mAP). Furthermore, Adver Occluded [18] and IGOAS [19], which utilize adversarial defense by generating aggressive samples, achieve only 44.5% and 60.1% Rank-1 accuracy, and 32.2% and 49.4% mAP, respectively, while our ETNDNet achieves 63.6% Rank-1 accuracy and 54.7% mAP. These results highlight the stronger occlusion defense ability of the model trained by our method. Additionally, the attention mechanism is commonly employed by state-of-the-art methods [64, 66] to extract feature maps containing crucial pedestrian information. In our proposed ETNDNet, adversarial representations formed by perturbing such feature maps are more aggressive, and leveraging them for adversarial defense enables the re-ID model to learn more robust features against occlusions. To achieve this, we incorporate Non-Local [71] into our backbone. As shown in Table II, our method (Ours*) achieves superior performance, surpassing the aforementioned methods (e.g., +1.2% mAP compared to FED [64]), further demonstrating the effectiveness and superiority of our approach.

For Occluded-REID, we repeat the experiment 10 times as in [19] and take the mean as the final result. As shown in Table II, the proposed method achieves satisfactory results. For example, compared with the optimal pose-guided based method PGFL-KD [37], the proposed ETNDNet improves the Rank-1 accuracy from 80.7% to 89.3%, and mAP from 70.3% to 81.5%. Compared with the optimal external model-free method FED [64], Rank-1 and mAP of ours are improved by 3.0% and 2.2%. Moreover, IGOAS [19], which is based on adversarial learning, the proposed ETNDNet surpasses it by 8.2% in Rank-1 accuracy. Finally, for P-DukeMTMC-reID, as shown in Table III, our method also outperforms state-of-the-art algorithms and achieves significant improvements in mAP, such as 6.8% over the optimal ISP [61].

According to the above results, the proposed method has significant advantages for occluded person re-ID, which is mainly attributed to the fact that our adversarial representations fully simulate perturbation caused by various occlusion issues (missing information, position misalignment, and noisy information), and the model trained by the developed GAN-based adversarial defense has the ability to protect the re-ID system from such disturbance, effectively overcoming the adverse effect of occlusion issues on recognition performance.

Experiments on holistic datasets: To further evaluate the superiority and practicality of our method, we compare with the state-of-the-art methods on holistic re-ID datasets, i.e., Market-1501, DukeMTMC-reID, and MSMT17, and report the comparison results in Table IV.

Although these three datasets are released for holistic person re-ID, due to the changeable postures and complex backgrounds, they still exist problems of information loss, information misalignment, and noisy information. Therefore, our method is also applicable and can further improve the recognition rate. Specifically, for Market-1501, our ETNDNet outperforms the state-of-the-art GPS [79] in terms of Rank-1 accuracy. Additionally, the integration of an attention mechanism into our framework led to a further improvement in recognition performance, with Rank-1 being 0.5% higher than GPS and mAP being 0.7% higher than PLIP [77]. As for DukeMTMC-reID, our ETNDNet also surpasses most state-of-the-art methods. Moreover, MSMT17 is currently the largest dataset, and it is relatively difficult due to the large difference between positive and negative pairs. Existing methods increase training samples (PTGAN [55]) or introduce attention mechanisms (IANet [73], OSNet [75], and AGW [5]) to reduce the intra-class distance. Even so, our method with less training cost achieves superior performance with 80.9% Rank-1 and 56.6% mAP, which is ahead of the state-of-the-art methods.

Ablation Studies. We conduct ablation experiments to demonstrate the contribution of each module in the proposed ETNDNet, including erasing defense (ED), transforming defense (TD), and noising defense (ND). All experiments were performed on Occluded-DukeMTMC, which can better reflect the effectiveness of modules. The Rank-1 accuracy and mAP (%) are reported in Table V.

Effectiveness of ED. The developed ED is designed to protect the re-ID system from missing information. As shown in Table V, when deploying it to the baseline re-ID system, the Rank-1 accuracy and mAP are improved from 52.5% to 58.7% and from 44.6% to 49.7%, respectively. Moreover, compared with NDNet, the Rank-1 and mAP obtained by ENDNet have also been significantly improved. These results verify that our ED is indeed able to promote the re-ID model to learn features that are not disturbed by information loss, playing a positive role in identifying occluded target pedestrians and can cooperate with other modules to further improve the recognition performance.

Effectiveness of TD. The proposed TD strives to defend the re-ID model for extracting features unaffected by misaligned information. As shown in Table V, our TDNet promotes the Rank-1 accuracy and mAP of the Baseline by a large margin. And, when it is equipped on the above EDNet, the Rank-1 accuracy and mAP are improved from 58.7% to 62.1% and from 49.7% to 52.8%. These results demonstrate that our TD is also beneficial to occluded person re-ID, effectively alleviating the adverse impact of position misalignment caused by occlusion on recognition performance.

Effectiveness of ND. The proposed ND is dedicated to solving the problem of occlusion noise introduced by obstacles and non-target pedestrians, which is ignored in existing studies. As shown in Table V, it also contributes to the improvement of recognition performance, and when it is combined with other modules, the formed framework can achieve higher accuracy. Specifically, our NDNet increases the Rank-1 accuracy of the re-ID baseline model by 5.5% and mAP by 4.2%. And, the performance of ENDNet and TNDNet is also higher than that of EDNet and TDNet. These results confirm our ND is indeed robust against the interference of noisy information.

Finally, when all modules are equipped, the developed ETNDNet can achieve the best recognition rates of 63.6% Rank-1 and 54.7% mAP. It follows that our three adversarial defense modules are able to collaborate with each other, effectively overcoming the negative effects of perturbation of various occlusion issues on identification.

Parameter Analysis. Hyper-parameters $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are used to balance the relative importance of adversarial losses in the above three adversarial defense modules. Here, we study the influence of their different values on recognition performance. Each parameter was tested with 11 different values and the experimental results are illustrated in Figure 4. Note that when one of the parameters is analyzed, the remaining two are fixed at the optimal values.

The effect of $\lambda_{1}$. From Figure 4, we can see that when $\lambda_{1}$ increases from 0 to 0.1, both Rank-1 accuracy and mAP are improved, and when $\lambda_{1}=0.1$, our method achieves the best recognition performance. This indicates that $\lambda_{1}=0.1$ is an optimal value for the proposed method. Moreover, we observe that the recognition performance decreases as $\lambda_{1}$ continues to increase, which may be due to over-guaranteeing the robustness of the learned features while destroying their discriminativeness.

The effect of $\lambda_{2}$. The hyper-parameter $\lambda_{2}$ is used to control the relative importance of the adversarial loss in the proposed transforming defense module. As shown in Figure 4, the performance of the developed ETNDNet remains relatively stable when $\lambda_{2}$ increases from 0 to 0.5. And, when $\lambda_{2}=0.15$, the recognition performance slightly exceeds the corresponding performance achieved at other values of $\lambda_{2}$. So, $\lambda_{2}$ is set to 0.15 in all the experiments.

The effect of $\lambda_{3}$. In order to further improve the robustness of the learned features, the ND is designed and the parameter $\lambda_{3}$ is used to control its importance. It can be seen that Rank-1 accuracy and mAP reach the peak when $\lambda_{3}=0.1$, which further confirms the validity of ND and the rationality of $\lambda_{3}=0.1$.

In this section, we further evaluate the proposed method. All experiments were performed on Occluded-Duke with the same hardware (1 GTX3090 GPU, 8 CPU, and 32GB memory).

Erasing Strategy. In our erasing module, we perturb feature maps with a random erasing strategy to form adversarial representations with incomplete information. The strategy proposed in [80] can also serve this purpose. However, since it fixes the erased area’s size proportion and aspect ratio while different obstacles have various sizes and shapes in real scenes, perturbing feature maps by such strategy hinders the aggressiveness of adversarial representations and ultimately limits recognition performance improvement. As shown in Figure 5(a), when we follow DropBlock [80] to fix the erased area’s size proportion and aspect ratio to 0.3, the Rank-1 and mAP of our EDNet dropped by 1.3% and 3.0%. Obviously, it is more appropriate to adopt our random erasing strategy to perturb feature maps.

Defense Paradigm. In this paper, we design a novel GAN-based adversarial defense paradigm to protect the re-ID system from various occlusion disturbances, which is significantly different from the adversarial representation learning of IGOAS [19]. Specifically, IGOAS employs an adversarial suppression branch that implements a mean square loss between the feature map and its masked version to make the model focus on pedestrian information in non-erased areas. In contrast, our GAN-based adversarial defense paradigm continuously plays an adversarial game between the identity classifier and feature extractor, encouraging the model to focus on highly discriminative information in non-disturbed areas. Particularly, in the developed defense paradigm, optimizing the feature extractor alone without the adversarial game process is comparable to IGOAS’s learning approach (that is, treating adversarial representations as supervision signals to optimize the feature extractor, making the model focus on pedestrian information in non-disturbed areas). As demonstrated in Figure 5(b), removing the adversarial game process reduces the mAP and Rank-1 accuracy by 4.8% and 5.9%, respectively, indicating the superiority of the learning way in our GAN-based adversarial defense paradigm.

Model Complexity. To further demonstrate the practicality of the proposed method, we conduct an analysis of the model complexity and compare ours with four state-of-the-art methods. The results are illustrated in Table VI, where we report the parameter number of models (Param), the number of floating-point operations for an input image (FLOPs), and the inference time of retrieving all queries (IT). It can be observed that existing state-of-the-art methods have a relatively large model complexity. In comparison, our Param, FLOPs, and IT are only 24.9M, 4.1, and 25s, thus the proposed ETNDNet has a significant advantage in model simplicity. Moreover, our method does not introduce any external modules with learnable parameters. Therefore, it has the same model complexity as the baseline, that is to say, our method improves the recognition performance without increasing the computational burden, with strong practicability. Additionally, incorporating Non-Local into our backbone, ETNDNet* increases the number of parameters by only 0.2M, but leads to significant performance improvements, surpassing state-of-the-art methods.

Model Generalization Ability. We also evaluate the model generalization ability of our algorithms, with the results presented in Table VII. Importantly, it should be noted that our ETNDNet was not primarily tailored for domain generalization, and its performance falls short of the state-of-the-art method [81]. Nonetheless, in comparison to other occluded person re-ID algorithms, our model demonstrates superior generalization ability. To illustrate, for Market $\rightarrow$ Duke, our ETNDNet outperforms HOReID by 1.6% and IGOAS by 0.2% in terms of mAP. Furthermore, compared with the Baseline, our algorithm yields a substantial enhancement of 6.0% in Rank-1 accuracy and 4.1% in mAP. This observation underscores the efficacy of the proposed algorithm in enhancing model generalization.

Adversarial Representation. Aggressiveness is an important basis for judging the quality of adversarial examples. As discussed earlier, our adversarial representations formed by perturbing feature maps are more aggressive than adversarial samples formed in [18, 19]. To demonstrate this, we test the performance of the model trained by the baseline framework on adversarial samples and various adversarial representations. The experimental results are shown in Figure 6 (a). Firstly, compared with the baseline, the adversarial samples reduce the mAP and Rank-1 by 9.3% and 8.0%, while our adversarial representations with incomplete information reduce the mAP and Rank-1 by 20.8% and 19.1%, which suggests that applying perturbation on the feature map is indeed more aggressive. Secondly, the adversarial representations generated by random transforming simulate the issue of position misalignment caused by occlusion, which is a capability that adversarial samples do not possess. And, as can be seen from Figure 6 (a), they lead to the reduction of the mAP and Rank-1 from 44.6% to 36.0% and from 52.5% to 46.1%. Thirdly, our adversarial representations with noisy information also reduce the recognition rate by a large margin (-16.3% mAP and -16.5% Rank-1) and are more effective in attacking the re-ID model than adversarial samples.

Model Defensive Capability. As we discussed above, the model trained by our ETNDNet has a stronger model defensive capability than IGOAS [19]. In order to further prove this, during testing, we randomly erase the query and gallery samples to further cheat the re-ID system and compare the recognition performance obtained by our method and IGOAS. As illustrated in Figure 6 (b), the performance that IGOAS achieved was far from the results of its baseline model. Different from it, the proposed ETNDNet gains a recognition rate comparable to the baseline. This is mainly because our adversarial defense strategy enables the re-ID model to pay close attention to as many pedestrian body parts as possible. Even if the information of a certain part is lost, the target pedestrian can still be correctly identified (as illustrated in Figure 7). The above qualitative and quantitative analysis fully demonstrate that our method with the developed novel GAN-based adversarial defense is valuable in object re-identification and deserves further exploration.

Limitations. In the proposed ETNDNet, we perturb feature maps by random erasing, transforming, and noising to form adversarial representations that can simulate various issues caused by occlusions. However, the erased, transformed, and noised areas possibly overlap with occlusions. Additionally, the shapes of obstacles and human bodies are often irregular, while our erased, transformed, and noised areas are regular rectangles, hindering the aggressiveness of adversarial representations and ultimately limiting performance improvement. These limitations prompt us to conduct further investigations in the future.