POAR: Towards Open Vocabulary Pedestrian Attribute Recognition

Pedestrian attribute recognition (PAR) has received much interest in person recognition (Cao et al., 2018; Hand and Chellappa, 2017; Jia et al., 2021; Liu et al., 2016; Ding et al., 2021; Wang et al., 2022b) and scene understanding (Jia et al., 2021; Liu et al., 2016; Ding et al., 2021). The mainstream methods address this problem by building a multi-label classifier based on CNN. To improve the recognition accuracy, global methods (Li et al., 2015; Liu et al., 2017), local methods (Sarafianos et al., 2018), global-local methods (Sarfraz et al., 2017), attention-based methods (Guo et al., 2022; Liu et al., 2022; Lu et al., 2023), sequential prediction methods (He et al., 2017; Yang et al., 2020), curriculum learning methods (Zhao et al., 2019), Graphic model methods (Tan et al., 2020; Fan et al., 2022), and group based methods (Zhao et al., 2018; Jia et al., 2022) have been proposed. Nikolaos et al. (Sarafianos et al., 2018) proposed an effective method to extract and aggregate visual attention masks across different scales. Tang et al. (Tang et al., 2019) proposed a flexible attribute localization module to learn attribute-specific regional features. These methods focused on domain-specific model designing. To use additional domain-specific guidance, M. Kalayeh et al. (Kalayeh et al., 2017) used semantic segmentation methods to learn attention maps for accurate attribute prediction. Liu et al. (Liu et al., 2016) learned clothing attributes with additional landmark labels. Yang et al. (Yang et al., 2021) proposed a cascaded split-and-aggregate learning to capture both the individuality and commonality for all attributes. Li et al. (Li et al., 2022) proposed an image-conditioned masked language model to learn complex sample-level attribute correlations from the perspective of language modeling. Tang et al. (Tang and Huang, 2022) and Weng et al. (Weng et al., 2023) employed ViT as a feature extractor for its nature of modeling long-range relations of regions. Cheng et al. (Cheng et al., 2022) proposed an additional textual modality to explore the textual semantic correlations from attribute annotations. These methods are trained on a predefined attribute set and used to recognize the same attributes, which limits the attribute capacity of these models. In our work, we build a TEMS model based on the CLIP (Radford et al., 2021) model to recognize pedestrian open-attributes.

In classification, open-world recognition (OWR) is first proposed by Scheirer (Scheirer et al., 2013), which aims to discriminate known from unknown samples as well as classify known ones. Later, prototype-based method (Saranrittichai et al., 2022; Vaze et al., 2022), knowledge distillation method (Gu et al., 2021), and out of distribution detection method (Esmaeilpour et al., 2022) become popular in image classification and object detection. Definitions and solutions for open-world recognition also differ based on the specific applications. Esmaeilpour et al. (Esmaeilpour et al., 2022) used an extended model to generate candidate unknown class names for each test sample and compute a confidence score based on both the known class names and candidate unknown class names for zero-shot out-of-distribution detection. Oza et al. (Oza and Patel, 2019) proposed the class conditional auto-encoder to tackle open-set recognition, which includes closed-set training and open-set training stages. Gu et al. (Gu et al., 2021) adopted an image-text pre-trained model as a teacher model to supervise student detectors. Zhao et al. (Zhao et al., 2020) unified the label space from the training of multiple datasets to improve the generalization ability of a model. In addition, some methods (Li et al., 2019; Rahman et al., 2020) aligned region features and the pre-trained text embeddings in base categories to realize zero-shot detection. Dan et al. (Dan et al., 2022) enhanced class understanding via prompt-tuning for zero-shot text classification. Du et al. (Du et al., 2022) proposed a detection prompt, to learn continuous prompt representations for open-vocabulary object detection. The CLIP model was proposed by Radford et al. (Radford et al., 2021), which performs task-agnostic training via natural language prompting. CLIP can realize zero-shot image recognition. However, CLIP is usually used to recognize general objects, such as airplane, bird, ocean, and so on. For fine-grained attributes recognition, CLIP will fall short in these situations. Our work addresses the pedestrian open-attribute recognition task by identifying both classes, seen and unseen classes, simultaneously.

Pedestrian attribute recognition aims to recognize the fine-grained attributes of a pedestrian (e.g., hairstyle, age, gender, etc.) from the given image. The conventional PAR usually predetermines a set of pedestrian attributes and follows the close-set assumption during the training and test phases. Suppose there are $M$ predetermined pedestrian attributes $\mathcal{A}=\{A_{1},A_{2},\dots,A_{M}\}$, e.g., $A_{1}=$“long hair”, $A_{2}=$“short hair”, etc. Then, given a labeled pedestrian dataset $\mathcal{D}=\{(I_{i},\mathcal{A}_{i})\}_{i=1}^{N}$, where each image $I_{i}\in\mathbb{R}^{H\times W\times 3}$ is annotated with a subset $\mathcal{A}_{i}\subset\mathcal{A}$ of the existing pedestrian attributes. The main objective of PAR is to learn a model to answer which pedestrian attributes from $\mathcal{A}$ appear in a given image $I$.

Existing approaches (Tang et al., 2019; Guo et al., 2022; Cheng et al., 2022) usually convert this to a multi-label classification problem. However, the pedestrian attributes in the real world are potentially unlimited. It is difficult to exhaust all the attributes in a predetermined set and collect the corresponding pedestrian images. Unseen attributes are highly possible to exist in real world applications, while existing classification-based methods (Cheng et al., 2022; Li et al., 2022) are inherently incapable of handling such cases. To address this limitation, we formulate a pedestrian open-attribute recognition problem in this work. Let $\mathcal{A}_{u}=\{A_{M+1},A_{M+2},\dots,A_{M+M_{u}}\}$ denote a set of extra attributes that are also of interest in the test phase but not included in the predetermined attribute set $\mathcal{A}$. In POAR, we expect that the model can recognize not only the seen attributes from $\mathcal{A}$ but also the unseen attributes from $\mathcal{A}_{u}$.

Overview. As illustrated in Figure 2, the proposed Transformer-based encoder with a masking strategy (TEMS) framework consists of an image encoder $\mathbf{\Phi}_{I}$ and a text encoder $\mathbf{\Phi}_{T}$. The image encoder processes the input images and derives the visual representation of various pedestrian attributes. Besides, we construct a set of text descriptions (e.g., “this person has long hair.”, “this person is carrying backpack.”, etc, and utilize the text encoder of CLIP to encode these attribute descriptions into text embeddings. Then, we compute the vision-text similarity to find the best-matched pedestrian attributes for the input images. Different from the original CLIP, we assume one pedestrian may have more than one associated attribute, as shown in Figure 3. To train our model, we propose a many-to-many contrastive (MTMC) loss with masked tokens to handle the many-to-many relationships between the images and text descriptions. The details of each part are described below.

Image Encoding. The image encoding process is organized based on token prediction with Transformer. First, the input image $I$ is split into a sequence of non-overlapping small patches {$P_{0}$, $P_{1}$, $\cdots$, $P_{S-1}$}, where $P_{i}\in\mathbb{R}^{{r}\times{r}\times 3}$ and $S=\frac{H}{r}\times\frac{W}{r}$. Then, each patch $P_{i}$ ([PAT]) is projected into the embedding space as a vector $\mathbf{x}_{i}\in\mathbb{R}^{D}$, where $D$ denotes the feature dimension. Thus, we can obtain $\mathbf{X}=[\mathbf{x}_{0},\mathbf{x}_{1},\dots,\mathbf{x}_{S-1}]\in\mathbb{R}^{D\times S}$ with $S$ patch embeddings. Inspired by (Dosovitskiy et al., 2020; Radford et al., 2021), we introduce $K$ learnable attribute tokens $\mathbf{Z}=[\mathbf{z}_{0},\mathbf{z}_{1},\dots,\mathbf{z}_{K-1}]\in\mathbb{R}^{D\times K}$, where each attribute token $\mathbf{z}_{k}\in\mathbb{R}^{D}$ shares the same dimension as $\mathbf{x}_{i}$. For each patch embedding and each class token, we generate a corresponding position encoding $\mathbf{e}_{j}\in\mathbb{R}^{D}$, where $j\in\{0,1,\cdots,S+K-1\}$. The input of the image encoder $\mathbf{\Phi}_{I}$ is the concatenation of class tokens and patch embeddings combined with their positional embeddings $\mathbf{E}=[\mathbf{e}_{0},\mathbf{e}_{1},\cdots,\mathbf{e}_{S+K-1}]\in\mathbb{R}^{D\times(S+K)}$, which is denoted as $\mathbf{V}=[\mathbf{Z},\mathbf{X}]+⃝\mathbf{E}$, where $\mathbf{V}\in\mathbb{R}^{D\times(S+K)}$, $[,]$ is the concatenate operation, and $+⃝$ denotes the element-wise summation notation. The output of the image encoder $\mathbf{\Phi}_{I}$ is the learned embeddings of class tokens, denoted as $\hat{\mathbf{Z}}\in\mathbb{R}^{D\times K}$. The above process can be summarized as follows:

Text Encoding. The text encoding is performed by using the text encoder $\mathbf{\Phi}_{T}$ transferred from the CLIP model. The input of the text encoder is the prompts of pedestrian attributes which are organized as follows. First, the attributes are divided into $K$ groups based on the characteristics of pedestrians, as shown in Table 1. Then, we form a prompt template for the attributes of each group. Finally, each prompt sentence in Table 1 is tokenized into an embedding using the Byte-Pair-Encoding method¹¹1https://huggingface.co/transformers/v4.8.0/model_doc/clip.html. Suppose that the maximum number of words in any of these sentences is $L$. Then, each sentence is tokenized into a vector $\mathbf{y}_{i}\in\mathbb{R}^{L}$. For the $k$-th group, we can obtain $m$ sentence vectors corresponding to the $m$ prompts in this group, denoted as $\mathbf{Y}^{k}=[\mathbf{y}_{1}^{k},\mathbf{y}_{2}^{k},\cdots,\mathbf{y}_{m}^{k}]\in\mathbb{R}^{L\times m}$. The output of the text encoder $\mathbf{\Phi}_{T}$ is the learned text embeddings $\hat{\textbf{Y}}^{k}$ for the prompts of the $k$-th group. The above process can be defined as:

where $\hat{\textbf{Y}}^{k}\in\mathbb{R}^{D\times m}$.

Many-to-Many Contrastive Loss.

For a mini-batch images $\{I_{1},I_{2},\cdots,I_{B}\}$, we first extract their token embeddings $\hat{\mathbf{Z}}_{b}$ and text embeddings $\hat{\mathbf{Y}}_{b}$ using (1) and (2), respectively. $\hat{\mathbf{Z}}_{b}$ and $\hat{\mathbf{Y}}_{b}$ are sets of token embeddings and text embeddings in all attribute groups related to the given mini-batch images, respectively. Different from the CLIP model, which has a one-to-one relationship between image and text. In the TEMS model, the image and text are involved in a many-to-many relationship, as shown in Figure 3. To effectively tackle this scenario, a loss function combined with visual-to-text and text-to-visual contrastive learning is proposed. The visual-to-text contrastive learning term is defined as follows:

where $\tau$ is a temperature parameter, $t_{i}$ is the number of positive text embeddings that have the same attribute label with $\hat{\mathbf{z}}_{i}$. $v$ and $t$ are the total numbers of token embeddings and text embeddings, respectively. $\hat{\mathbf{z}}_{i}\in\hat{\mathbf{Z}}_{b}$, $\hat{\mathbf{y}}_{k}\in\hat{\mathbf{Y}}_{b}$, $\hat{\mathbf{y}}_{j}^{+}\in\hat{\mathbf{Y}}_{b}$. $\hat{\mathbf{z}}_{i}$ and $\hat{\mathbf{y}}_{j}^{+}$ are a positive pair share the same attribute label. Similarly, the text-to-visual contrastive learning term is defined as follows:

where $v_{j}$ is the number of positive token embeddings that have the same attribute label with $\hat{\mathbf{y}}_{j}$. The final loss function is the combination of (3) and (4), defined as:

Image Encoder. The image encoder $\mathbf{\Phi}_{I}$ is a stack of multiple Transformer blocks. Each Transformer block is composed of layer norm (LN) layers (Dosovitskiy et al., 2020), a multi-head self-attention (MSA) layer (Dosovitskiy et al., 2020), and a multi-layer perceptron (MLP) network. In the multi-head self-attention layer, the attention weights of each attribute token are automatically calculated among all input tokens. Specifically, the attention weights between the $k$-th attribute token and the others can be computed as

We observe that the first term $\mathbf{z}_{k}^{\top}\mathbf{Z}$ often dominates the attention, making the module hardly find the true regions of interest from the input image. This shortcut learning often leads to overfitting and inferior results in our experiments. To address this issue, we mask out the self-attentions between the attribute tokens. This allows useful visual information to be extracted from the input image rather than simply relying on the information that has been extracted by the others. Specifically, we calculate the attention weight of the $k$-th attribute token as

In our experiments, we observe that this technique leads to significant performance gain (Table 7).

In the PAR task, one key challenge is that a pedestrian can have multiple attributes, and there is no corresponding location and scale information in the ground truth label set. We propose to mask the irrelevant patches (MIP) to tackle this challenge. Specifically, we divide the whole attribute classes into multiple groups, and each group ([ATT] token) corresponds to one visual region. Then we mask out regions ([PAT] token) that do not need to pay attention to. For example, the “hair” class token pays more attention to the head of the pedestrian, and we block out regions on the lower part of the head region; similarly, the “upper body” class token pays more attention to the top part of the image, and we block out the bottom part of the image, etc. Thus, the final attention can thus be calculated as follows

where $\mathbf{\varpi}\in\mathbb{R}^{1\times S}$ is a mask vector with value $-\infty$ for the blocked image patches and $0$ otherwise. Taking the “hair” class token, for example, the region of the head will be $0$, and the remaining regions will be $-\infty$. The output of the self-attention unit is as follows:

where $l$ is the layer index of the self-attention, $\mathbf{Z}^{0}=\mathbf{Z}$, $\mathbf{V}^{0}=\mathbf{V}$. The whole process of a Transformer block can be formulated as:

Contrastive Learning with Masked Tokens. It should be noted that our contrastive loss is computed based on all the token embeddings and attribute embeddings. The embeddings of those masked tokens are also used to compute the loss. This is because the contrastive loss computed with the masked embeddings will lead to more robust embedding learning.

The knowledge distillation allows for the transfer of knowledge from a complex teacher model to a simpler student model, albeit a slight decrease in performance. The previous paper underscored CLIP’s impressive generalization capabilities. Nevertheless, its accuracy in certain tasks, such as pedestrian attribute recognition, leaves room for improvement. To address this, we propose the grouped knowledge distillation (GKD) method, as shown in Figure 4, which leverages the CLIP model as a teacher model and distills its extensive knowledge into our pedestrian attribute model during training. The objective function is defined as:

The Kullback-Leibler (KL) divergence is computed by each visual token embedding $\hat{z}_{i}^{l}$ of the student network and the visual embedding $\hat{w}^{l}$ of the teacher network. These loss functions introduce a non-linear transformation to the input data, which helps to capture more complex and abstract features of the data. The CLIP model has strong generalization performance and can be used as a teacher network to guide the learning of embedding space, reducing semantic disparities between the image embedding features and the unseen text features, and thereby improving the performance of the TEMS model.

Datasets and Evaluation Metrics. The proposed method is evaluated on three benchmark datasets, i.e., PETA (Deng et al., 2014), RAPv1 (Li et al., 2015), and PA100K (Liu et al., 2017) with an open-attribute setting, meaning the model is trained on one dataset and evaluated on the other two datasets. The classes included in different datasets may not be identical. Recall@K and mA (mean Accuracy) are used to evaluate the performance of our open-attribute recognition model. For the PETA dataset, the “upper body” and “lower body” groups may include two attribute classes. We select R@2 texts to show for attribute groups in “upper body” and “lower body”.

Implementation Details. Our experiments are conducted on the ViT-B/16 backbone networks, which are stacked with 12 Transformer blocks. The input image size is set to 224$\times$224. The value of $r$ is 16, and $K$ is set to be 8, 8, and 11 in PETA, PA100K, and RAPv1, respectively. The learning rate is $5e^{-2}$ with a weight decay of 0.2. The temperature $\tau$ in contrastive loss is set as 5. Data augmentation with random horizontal flip and random erasing are used during training.

We compare the performance of our method with the CLIP (Radford et al., 2021) and VTB (Cheng et al., 2022) methods based on the image-to-text $K$-nearest neighbor retrieval idea, the top-1 and top-2 recall rates are shown in Table 2. In Table 3, SN and UN represent seen and unseen attributes in the different test sets, respectively.

From Table 2, we can see that the Recall@1 (R@1) scores of our method are 1.7% and 8.6% higher than the CLIP method when the model is trained on the PETA dataset and evaluated on the PA100K and RAPv1 datasets, respectively. When the model is trained on the PA100K and evaluated on the PETA dataset, the R@1 score of our method is 7.9% lower than the CLIP model. When the model is trained on the RAPv1 and evaluated on the PETA, the R@1 scores of our method are 1.4% lower than the CLIP model. This is likely due to the fact that the CLIP model is trained on 400 million image-text pairs which have a high probability of containing “age” and “hair” and other attributes; thus, those unseen attributes defined in our experiments can be considered seen in the CLIP model. However, when we fuse the features of our model and the CLIP model, a higher performance for the POAR task can be obtained. When we train the proposed TEMS model with GKD using the CLIP model as the teacher model, the model’s transferred performance to other datasets improves significantly. However, the TEMS model’s performance on its training dataset decreases as compared to the results without distillation. For instance, when the model is trained on the PETA dataset, its performance on the PETA test set decreases by including distillation. Nevertheless, the model exhibits significant improvements on other datasets.

To analyze the details of the POAR performance, we show the performance comparison of each group of the CLIP model and the proposed TEMS model in Figure 5. The results on different datasets and attributes vary between the TEMS method and the CLIP method. However, the model trained through GKD exhibits a significant improvement in the accuracy of various attribute tokens, as evidenced by the comparison between the orange (without GKD) and yellow (with GKD) graphs in Figure 5.

Furthermore, we compute the recognition performance of seen and unseen attributes in the PA100K dataset, respectively. Results are shown in Figure 6, the proposed TEMS method has the capability to recognize both seen and unseen categories. For example, when the method is trained on PETA dataset and tested on the PA100K dataset, Figure 6 shows the accuracy of the “Carry” group for seen (blue) and unseen (gray) classes. Moreover, after implementing GKD training, there is a remarkable enhancement in the model’s ability to recognize unseen classes, as evidenced by the comparison of R@1 results for unseen classes in Figure 6 (a) (b) (gray and yellow). This further confirms the effectiveness of the proposed method in identifying unseen classes.

To examine the generalization ability of our method, we also evaluate the text-to-image retrieval results. The results are reported in Table 4. Compared to the image-to-text retrieve performance of Table 2, we can see that the text-to-image recognition task is more challenging than the image-to-text recognition task. This is mainly due to the fact that the text embedding space is more sparse than the image embedding space.

Our ablation study is conducted on the PETA dataset using the close-set and open-set evaluation mechanisms. Table 7 shows the image-to-text retrieval performance of different components in our proposed method, SAT represents a single attribute token. MAT represents multiple attribute tokens. MAM represents multiple attribute tokens with the masking. Table 6 shows the image-to-text retrieval performance for different loss functions, OTOC represents one-to-one contrastive loss. MTMC represents many-to-many contrastive loss. The one-to-one loss function is performed by defining all attributes of one image in a paragraph description as text input. From Table 7, we can see that each component of the proposed method can contribute positively to the final performance gain. From Table 6, we can see that the many-to-many loss function significantly improves the final performance.

To illustrate the effectiveness of the proposed masking method, we show the attention map of each attribute token on the PETA dataset in Figure 7. In our method, we proposed a masking strategy to block out distract regions and mask the corresponding attribute tokens. From the attention map, we can see that each attribute token is responsible for a specific part of the body, which shows the effectiveness of the proposed masking method.

In Figure 8, we visualize some image-to-text retrieval examples based on embeddings obtained using the CLIP model and our proposed TEMS model. The TEMS model was trained on the PETA dataset. Figure 8 (a) and Figure 8 (b) show examples tested on the PA100K and RAPv1 datasets, respectively. We can see that there are a lot of unseen attributes in the test set, and our model can effectively recognize the unseen attributes. From Figure 8, the CLIP model exhibits a higher rate of errors in recognizing pedestrian attribute categories. In contrast, as illustrated in Figure  8 (a) (i), our proposed model achieves perfect recognition of the pedestrian attributes, while the CLIP model only correctly identifies three attributes for the same pedestrian. In the RAPv1 dataset, there are 39 unseen classes, the recognition accuracy for these classes is much higher by the TEMS model than that of the CLIP model.