Rethinking of Pedestrian Attribute Recognition: Realistic Datasets and A Strong Baseline

Most recent efforts resort to learning discriminate attribute-specific feature representation by enhancing attribute localization. Li et al. [16] firstly considered pedestrian attribute recognition as a multi-label classification task and proposed the weighted sigmoid cross entropy loss. Considering better attribute localization can reduce the impact of irrelevant areas, Liu et al. [25] proposed HydraPlus-Net with multi-directional attention modules to locate fine-grained attributes. Liu et al. [23] proposed a Localization Guided Network based on Class Activation Map(CAM) [33] and EdgeBox [35] to extract attribute-specific local features. Considering corresponding area scales of attributes are various, multi-scale feature maps were reused from different backbone layers [32, 26], i.e. pyramidal feature hierarchy, instead of single feature map of last layer. Guo et al. [10] utilized the assumption that visual attention regions are consistent between different spatial transforms of same image and proposed an attention consistency loss to get a robust attribute localization. Inspired by Feature Pyramid Network(FPN) [20], Tang et al. [28] constructed Attribute Localization Module with Squeeze-and-Excitation(SE) block [13] and Spatial Transformer Network (STN) [15] to enhance attribute localization. From the viewpoint of capturing attribute semantic dependencies, some methods focused on modeling attribute relationship. Wang et al. [30] transformed pedestrian attribute recognition to sequence prediction by Long-Shot-Term-Memory (LSTM) [12] to explore attribute context and correlation.

Compared to previous work, our work rethinks recent progress made on pedestrian attribute recognition from the perspective of datasets and methods. First, the dataset problem we first noticed leads to overestimated performance and misleads the evaluation of recent methods. Thus, we propose two reasonable and realistic datasets. Second, based on a fully utilized baseline network, we find localizing specific-attribute area adopted by recent SOTA methods [26, 10, 28] is not beneficial for performance improvement.

As far as we know, APiS [34] is the first pedestrian attribute recognition dataset followed by PETA [7], RAPv1 [19], PA100k [25], RAPv2 [18] which promote the development of pedestrian attribute recognition. However, no clear, concrete and unified setting is proposed for pedestrian attribute dataset construction.

For PETA, RAPv1, RAPv2 datasets, randomly splitting is adopted as default setting to construct train and test set and the protocol are used by almost all methods [19, 25, 30, 18, 26, 10, 28]. This results in a large number of identical pedestrian identities in train and test set. Considering images of identical pedestrian identities are often collected by the same surveillance camera in very short frame intervals, the appearance of these images is extremely similar with negligible background and pose variation as detailed in Fig. 1(a) and Fig. 1(b). As a result, there are plenty of extremely similar images in train set and test set as shown in Fig. 1(c).

To further illuminate the difference between existing datasets and our proposed datasets, the pedestrian identity distribution of PETA is given in Fig. 3. In existing dataset PETA, there are 1106 identical identities in train and test set, accounting for 26.91% identities and 57.70% images of the test set. It means that more than half of test set images have their similar counterparts in train set, which is similar to ’data leakage’. In our proposed PETA_$zs$, there is no overlap between pedestrian identities of train and test set. As for RAPv1 (or RAPv2), all images (or part of images) have no pedestrian identity annotation, so we can not get the accurate identity distribution.

Given the problem of existing datasets, the reasons why it is crucial for industry application and academic research are given as follows. Whether used as primary task in video surveillance or auxiliary task in person retrieval, pedestrian identities of test set barely overlap with identities of train set. So, existing datasets setting is inconsistent with real world application. More importantly, the performance of SOTA methods is overestimated on existing datasets and the model evaluation is misled. We reimplement recent methods MsVAA [26], VAC [10], ALM [28] and report their performance in Fig. 2. Compared to performance on the whole test set, consistent performance degradation of three methods on unique-identity images of test set confirms the overestimated model performance.

To solve the problem of existing pedestrian attribute datasets, we propose a realistic setting: there is no identical pedestrian identities in train and test set, i.e. pedestrian identities follow the zero-shot setting. Concretely, we propose the following criteria for dataset construction, to serve as reference for future dataset.

Criteria of proposed datasets:

where $\mathbb{I}$ denote the pedestrian identity set, $N$, $R$, $T$ denote the number of images, the ratio of positive samples of attributes and pre-defined threshold separately. $T_{id}=0.01$, $T_{img}=300$, $T_{attr}=0.03$ are used in our experiments by default. Subscript ${train}$, ${valid}$, ${test}$ denotes train set, validation set and test set separately. $\absolutevalue{\ \cdot\ }$ is the set cardinality.

To solve the problem of identical identities between train set and test set, Criterion 1,2,3 are proposed to repartition train set and test set of PETA and RAPv2 datasets based on pedestrian identities to make sure that pedestrian identities follow zero-shot setting. To better evaluate the model performance and control the attribute distribution difference between train set and test set, Criterion 4, 5 are proposed.

Based on criteria mentioned above, considering pedestrian identity label is only provided in PETA and RAPv2 datasets, two realistic datasets PETA_$zs$ and RAPv2_$zs$ are proposed, where subscript $zs$ denotes pedestrian identities of dataset following zero-shot setting. Details of the two proposed dataset are given in Table 1 and supplementary material.

Given a dataset $\mathbb{D}=\{(\bm{X}_{i},\bm{y}_{i})\}$, $\bm{y}_{i}\in\{0,1\}^{M}$, $i=1,2,...,N$, where $\bm{y}_{i}$ indicates ground truth vector and $N$, $M$ denotes the number of train images and attributes respectively. $\bm{X}_{i}$ denotes $i$-$th$ pedestrian image. Pedestrian attribute recognition is a multi-label task to learn to predict attributes $\bm{\hat{y}}_{i}\in\{0,1\}^{M}$, given the pedestrian image $X_{i}$. The element values of zeros and ones in the label vector $\bm{y}$ and $\bm{\hat{y}}$ denote the absence and presence of the corresponding attributes in the pedestrian image.

Pedestrian attribute model generally adopts multiple binary classifiers with sigmoid function [10, 28] instead of a multi-class classifier with softmax function [29, 6]. So the loss can be computed as

where $\sigma(z)=1/(1+e^{-z})$ and $logits_{ij}$ is the output of classifier layer and $\omega_{j}$ adopted here is proposed in [17] to alleviate the distribution imbalance between attributes.

where $r_{j}$ is the positive sample ratio of $j$-$th$ attribute in train set.

Denote the feature representation of the $i$-$th$ example as $\mathbf{x}_{i}\in R^{d}$ , then the conditional probability output by a deep neural network can be predicted by a linear classifier followed by a sigmoid function in Eq. (3)

where $[\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{M}]\in R^{d\times M}$ is weight of linear classifier and $p_{ij}$ is the $j$-$th$ attribute probability of the $i$-$th$ image. We denote this method as baseline in the following sections.

We conduct experiments in four existing pedestrian attribute datasets, PETA [7], RAPv1 [19], RAPv2 [18], PA100k [25] and two proposed realistic datasets PETA_$zs$, RAPv2_$zs$. Details of each dataset are given in Table 1.

Two types of metrics, i.e. a label based metric and four instance based metrics, are adopted to evaluate attribute recognition performance [18]. For label based metric, we compute the mean value of classification accuracy of positive samples and negative samples as the metric of specific attribute, then we take an average over all attributes as mean Accuracy (mA). For instance based metrics, Accuracy, Precision, Recall and F1-score are used.

The proposed method is implemented with PyTorch and trained end-to-end. ResNet50 [11] is adopted as backbone to extract pedestrian image feature. Pedestrian images are resized to $256\times 192$ with random horizontal mirroring as inputs. SGD is employed for training, with momentum of 0.9, and weight decay of 0.0005. The initial learning rate equals to 0.01 and batch size is set to 64. Plateau learning rate scheduler is used with reduction factor 0.1. Total epoch number of training is 30

To make a fair comparison with previous SOTA methods, we reimplement MsVAA, VAC, ALM methods and report their performance on PETA, PA100k, RAPv1 as well as their corresponding baseline performance, as shown in Table. 4. It is worth noting that our baseline achieve a much better performance than baseline of previous methods, even if they adopt a powerful backbone ResNet101 [11]. And compared to previous SOTA methods reimplemented with same backbone, our baseline achieve a comparable even better performance. We argue that the effectiveness of a method can not be fully verified when compared with a worse baseline.

The reason why our baseline can achieve a comparable even better performance than previous methods is that a strong baseline itself can implicitly learn the location of attribute-specific area. We utilize GradCAM [27] to locate discriminative visual cues of our baseline model. As show in Fig. 4, even if without explicitly modeling the localization of attribute-specific area, our baseline can localize attribute-specific area to learn discriminative representation. The important thing is not to locate the area of specific attribute, but to distinguish the fine-grained attributes in the same are, such as distinguish sandals from sneakers. Compared to original performance of SOTA methods, there is little difference in performance of reimplemented methods except for ALM. The reason is that attention area of ALM is hard bounding-box, which is coarse-grained and introduces environmental noise.