Dual-path CNN with Max Gated block for Text-Based Person Re-identification

For natural language processing (NLP) [20, 21, 22], word embedding has become an essential component for various tasks. Word2vec [23] can learn the context information of words to obtain word representations. Nevertheless, it only concerns the local information of the word and ignores the relevancy with the words outside the local window. Glove [24] utilizes local and global information simultaneously. Different from word2vec and glove, EMLO [25] dynamically adjusts word embedding and addresses the problem of polysemy with bi-LSTM. GPT [26] replaces bi-LSTM with Transformer [27], which consists of the Multi-Head attention mechanism, to better capture long-distance language semantics. Moreover, BERT [18] leverages the bidirectional training of the Transformer to learn the contextual relations between the words in a textual description. It also introduces the word masking mechanism to learn the robust word embedding.

In this paper, our method combines CNN with BERT to learn better word embeddings for text-based person Re-id and achieves better performance than other word embeddings.

Person re-identification (Re-id) has gained notable attention in all of its aspects. The early approaches focus on improving hand-crafted features and state-of-the-art approaches are mostly dominated by deep CNNs. The combination of global feature and partial feature is an essential solution to improve the performances in person Re-id task. Wang et.al [28] proposed a multi-granularity network (MGN), which integrates discriminative information with various granularities. To utilize the spatial-temporal information, Wang et.al [29] constructed a two-stream spatial-temporal person Re-id framework to mine both visual semantic information and spatial-temporal information. Considering the pixel-level accuracy and capability of modeling arbitrary contours of human semantic parsing, Kalayeh et.al [30] integrated human semantic parsing in person Re-id. To obtain unique pose information, Sarfraz et.al [31] incorporated both the fine and coarse pose information of the person to learn a discriminative embedding. Recently, the development of the generative adversarial network (GAN) [32] also promotes the field of person Re-id, such as SPGAN [33]. AANet [34] is an attribute-based person Re-id method, which integrates person attributes and attribute attention maps into a classification framework.

Unlike the traditional person Re-id approaches that exploit additional uni-modal clues, such as pose and mask, etc, text-based person Re-id is a multi-modal problem where the major challenge is the modality heterogeneity.

In the initial work, the first large scale person Re-id data (CUHK-PEDES), which contains person images with detailed natural language annotations, was introduced by Li et al [35]. They also proposed the Recurrent Neural Network with Gated Neural Attention mechanism model (GNA-RNN) to learn the word by word affinity of a sentence with image features. To effectively utilize identity-level annotations, Li et al [36] further used a two-stage CNN-LSTM network for textual-visual matching. To alleviate the cross-modal fine-grained problem, Niu et al. [37] achieved better cross-modal similarity evaluation by a Multi-granularity Image-text Alignments (MIA) model. Recently, Nikolaos [10] combined BERT with bi-LSTM to learn better word embedding and introduced a Text-Image Modality Adversarial Matching (TIMAM) approach, which learns modality-invariant feature representations with adversarial and cross-modal matching objectives. Different from the above methods which are all CNN-RNN architectures, Zheng et al [38], proposed a dual CNN architecture to extract image-text features and used each image-sentence pair as a single class to train the whole network with instance loss and ranking loss.

Beyond the aforementioned methods, we are probably the first to combine BERT with CNN to extract word embeddings for text-based person Re-id. We further utilize GMP and gated block to make visual-textual association concern more on remarkable features of both modalities for more reasonable similarity measurement.

Text processing:To extract better word embeddings, we apply a recently proposed language representation model named BERT. Before we input sentence to BERT, some minimal processing is needed to turn them into the format BERT requires, shown in Figure 3. Our first step is to tokenize the sentences, which breaks them up into words and subwords in the format BERT is comfortable with. Specifically, we add the special words to the sentence (add [CLS] at the bedding of the sentence and [SEP] at the end). We then replace the tokens with corresponding numbers in the embedded table, which can be obtained from the pre-training model. After tokenization, each sentence is represented as a list of tokens. To process the examples all at once, we need to pad all lists to the same size. For the integrity of the description, the fixed-length of the sentence is set to 120 words, which is the max length of samples. Our experiments show the effectiveness of long sentences. The sentences shorter than the fixed-length are zero-padded.

After that, each tokenized sentence is inputted to the BERT to extract a 768-D word embedding for each word. Thus, each sentence can be converted to word embeddings of size $120\times 768$. We further reshape the word embeddings into the $1\times 120\times 768$ format, which can be considered as height, width, and channel of images, respectively.

Image Processing: Since pedestrians are usually rectangular, all images are resized to 384x128. For data augmentation, we pad images on all sides with 10 values and random crops of the original dimensions are extracted.To make the model more robust, we horizontally flip the images randomly with 50% probability and shuffle the images. Finally, the images are normalized with mean and standard deviation.

Deep text CNN:In Figure 2, the upper branch corresponds to deep text CNN, which is a 50 layered deep ResNet modified to deal with word embeddings. The filter size of the first convolutional layer is $1\times 1\times 300$ and the max-pooling layer after the first convolutional layer is removed. The rest of the network is similar to the ResNet-50 except that the convolution filters are of size $1\times 3$ instead of $3\times 3$, which produce responses by looking at the three neighboring words. The $1\times 30\times 2048$ feature map ${{T}_{txt}}$ can be extracted after the word embeddings are inputted to the text CNN.

Deep vision CNN: In Figure 2, the lower branch is vision CNN which is based on ResNet-50 pre-trained on ImageNet. The structure before the original global average-pooling layer (GAP) is maintained the same as the original architecture. The GAP layer and what follows are removed. Given an input image of size $384\times 128$, a forward pass of the network produces a $12\times 4\times 2048$ feature map ${{T}_{img}}$ of activation.

The visual-textual feature maps ${{T}_{txt}}$ and ${{T}_{img}}$ are obtained by the vision CNN and the text CNN, respectively. To make the visual-textual features focus more on discriminative characteristics, i.e. salient part, we use the global Max-pooling (GMP) layer to gather important clues about distinctive object features. Then, we can obtain two 2048-dimensional feature vectors, ${{f}_{img}}$ and ${{f}_{txt}}$.

Gated block (GB): The GMP activates on more local saliency, but only a few features of high-dimensional features are effective and max pooling will introduce a lot of noise. Thus, GB is proposed to produce an attention map to selectively enforce informative features and suppress the noise in the maxed-pooled features, as shown in Figure 4. The GB is composed of a multi-layer perception with two hidden layers (MLP), which is defined as:

where $\delta$ denotes the ReLU function, ${{W}_{1}}\in{{\mathbb{R}}^{\frac{2048}{r}\times 2048}}$ and ${{W}_{2}}\in{{\mathbb{R}}^{2048\times\frac{2048}{r}}}$. To limit model complexity and add generalization, the first hidden layer is a dimensionality-reduction layer with parameters ${{W}_{1}}$ and reduction radio $r$. Then feature vectors, ${{f}_{img}}$ and ${{f}_{txt}}$, are rescaled with the attention map:

The GB introduces dynamics conditioned on the features, helping to boost feature discriminability. The final image descriptor, $v$, and text descriptor, $t$, are obtained after a fully-connected layer (input dim:$2048$, output dim:$2048$). We also add a shared-weight ${{W}_{share}}$ classification layer (input dim:$2048$, output dim: $11003$).

We employ two kinds of loss functions, i,e., identification loss and matching loss that bring the features originated from the same identity close and push away features originated from different identities.

Identification loss: We regard different IDs in the training set as the number of categories and classify images and descriptions to the corresponding ID category separately. The identification loss is CMPC loss where the cross-modal projection is integrated into the norm-softmax loss to further enhance the compactness of the matched embeddings.

Given a mini-batch with $N$ image-text pairs, image descriptors, $V=\left\{{{v}_{i}}\right\}_{i=1}^{N}$, and text descriptors, $T=\left\{{{t}_{i}}\right\}_{i=1}^{N}$ can be extracted. The label set $Y=\left\{{{y}_{i}}\right\}_{i=1}^{N}$ is from $M$ classes, then the CMPC loss for classifying image can be described as follows:

where ${{\bar{t}}_{i}}$ denotes the normalized text feature, ${{\hat{v}}_{i}}$ represents the vector projection of image feature ${{v}_{i}}$ onto the normalized text feature ${{\bar{t}}_{i}}$, $N$ is the batch size, ${{W}_{j}}$ and ${{W}_{{{y}_{i}}}}$ are the ${{y}_{i}}-th$ and $j-th$ column of the weight matrix $W$ in the classification layer. The text classification loss function is computed in a similar manner and the final CMPC loss can be calculated with:

Matching loss: To refrain from using loss that sample triplets or quadruplets within each batch, which introduce a computational overhead during training to obtain hard negatives, we use CMPM loss which incorporates the cross-modal projection into KL divergence to associate the representations across different modalities.

Given $n$ image and text samples in a batch, for each image feature, ${{v}_{i}}$, the image-text feature pairs are constructed as $\left\{\left({{v}_{i}},{{t}_{j}}\right),{{m}_{i,j}}\right\}_{j=1}^{n}$, where ${{m}_{i,j}}=1$ means that ${{v}_{i}}$ and ${{t}_{j}}$ are the same ID category, while ${{m}_{i,j}}=0$ indicates the unmatched ones. The probability of matching ${{v}_{i}}$ to ${{t}_{j}}$ is defined as:

where ${{p}_{i,j}}$ denotes the proportion of scalar projection of $\left({{v}_{i}},{{t}_{j}}\right)$ among all pairs $\left\{\left({{v}_{i}},{{t}_{j}}\right)\right\}_{j=1}^{n}$ in a batch. Thus, the more similar the image feature to text feature, the larger the scalar projection is from former to the latter. Consider the fact that in each batch there might be more than one matched text samples for image sample ${{v}_{i}}$, the true matching probability is normalized as:

The CMPM loss of associating ${{v}_{i}}$ with correctly matched text samples for each bach is defined as:

where $\varepsilon$ is a small number for preventing numerical problems. The same procedure is followed to perform the opposite matching (i.e., from text to image) to compute loss $L_{cmpm}^{tpi}$ and the bi-directional CMPM loss is computed by:

The overall objective function is formulated as:

The CUHK-PEDES is the only large scale available dataset for the text-based person Re-id [35]. It contains 40206 pedestrian images and 80412 textual descriptions by combining several Re-id datasets, for 13003 identities. Each image is annotated by two textual descriptions. On average, each person has 3.1 images and each textual description contains more than 23 words. The dataset contains 9408 different words. The data split method proposed by Li et al [35] is utilized in our experiments. The training set has 34054 images, 11003 persons, and 68126 textual descriptions. The validation set has 3078 images, 1000 persons, and 6158 textual descriptions. The test set has 3074 images, 1000 persons, and 6156 textual descriptions.

Implementation details: We use PyTorch as our deep learning framework and a single NVIDIA GeForce GTX 1080ti GPU. The model is trained for 160 epochs using stochastic gradient descent (SGD) algorithm with momentum fixed to 0.9. The size of the image feature $v$ and text feature $t$ is equal to 2048. The reduction ratio $r$ is 16. The batch size is 64 and the initial learning rate is set to 0.1. The warm-up learning rate as descripted in equal 13 is adopted to make the model stable and converge fast.

The learning rate $lr\left(e\right)$ in epoch $e$ is computed as:

We split the model training into two steps. We fix the weights of BERT in the whole training process. In step-1, we fix the parameters of pre-trained visual CNN and only train the other model parameters for 80 epochs. In stage-2, we release the parts of the visual CNN and train the visual CNN and text CNN for 80 epochs.

Evaluation metrics: At testing time, features $f$ (i.e. image descriptor $v$ and text descriptor $t$) are extracted through the network. The distance between the query (text) and the gallery (image) is computed using the cosine similarity:

The distances are then sorted. We adopt rank-1, rank-5, and rank-10 accuracies to evaluate the performance. Given a textual description, we rank all test images by their similarities with the query text. If top-K images contain any corresponding person, the search is successful.

Table 1 shows the results of our proposed DCMG and the nine best-performance methods on the CUHK-PEDES. The nine methods that have been evaluated on this dataset include (1) GNA-RNN [35] which applies unit-level attentions and word-level gates to learn the word by word affinity of a sentence with image features. (2) IATVM [36] which is a two-stage CNN-LSTM network with two attention modules working on two modalities at different levels. (3) PWM-ATH [15] which utilizes a patch-word matching and learns an adaptive threshold for each word to enhance or suppress its effect on the final text-image affinity. (4) Dual-path [38] which has similar dual-path CNN architecture to extract image-text features optimized with ranking loss and instance loss. (5) CMPM+CMPC [14] which proposes CMPM loss and CMPC loss for learning deep discriminative image-text embeddings. (6) MCCL [17] which propose a novel mutually connected classification loss to fully exploit the identity-level information. (7) MIA [37] which combines three different granularities hierarchically, i.e., global-global, global-local, and local-local, to alleviate the cross-modal fine-grained problem. (8) PMA [16] which exploits the multilevel corresponding visual contents and learns latent semantic alignment between the visual body part and textual noun phrase. (9) TIMAM [10] which combines BERT with bi-LSTM to learn better word embedding, and use adversarial and cross-modal matching objectives to learn modality-invariant feature representations. We observe that DCMG obtains the rank-1 score of 55.81% and outperforms all compared methods in terms of rank-1, which demonstrate the effectiveness of our methods on the dataset. Although TIMAM obtains the highest accuracy in terms of rank-5, our approach outperforms TIMAM by 1.30% in terms of rank-1 and 0.09% in terms of rank-10, which are more important.

We conduct the qualitative evaluation on CUHK-PEDES for our proposed DCMG. Figure 5 presents three visual retrieval results with natural language descriptions. The top 2 rows show successful cases where corresponding images within the top-10 retrieval results. For the successful cases, we can observe that the top results have multiple regions that fit the noun phrase in the text. Some non-corresponding images also show correlations to the query descriptions (e.g., a blue and white stripe tank top and a pair of green pants). Some interesting observations can be made from the failure cases. While all retrieved images in the bottom case have a different ID with the ground truth of the language description, the retrieved persons are similar to the textual descriptions, who all wear a black and white purse. It indicates that DCMG can successfully learn cloth and accessory-related things and retrieve images that match the textual input.

To understand how DCMG learn discriminative visual-textual feature embeddings, we visualize the activations of the last convolutional feature maps to investigate where the network focuses on to extract features. Following [39], the activation maps are computed as the sum of absolute-valued feature maps along the channel dimension followed by a spatial l2 normalization. Figure 6 compares the activation maps of DCMG and the Pre-trained ResNet-50. It demonstrates that DCMG can capture the local discriminative regions (e.g., white shoes and a white backpack), which match the corresponding person image to a textual description. In contrast, the Pre-trained ResNet-50 pays more attention to the uncorrelated regions. Therefore, this qualitative result shows that DCMG makes the visual-textual features focus more on discriminative characteristics, i.e. salient part, and gathers important clues about distinctive object features.

To investigate the several components in DCMG, we perform a set of ablation studies and assess how each component of DCMG contributes to the final matching performance on the CUHK-PEDES dataset.

Impact of proposed components: In our first ablation study, the influence of GAP, GMP, and GB on the CUHK-PEDES is investigated in Table 2. We first test only applying pooling operation in the network. As shown by the results, the concatenating average-pooled and max-pooled features have the best results in terms of rank-1, which both contain the information of max-pooled and average-pooled feature maps. However, in terms of rank-5 and rank-10, GMP shows the best performance.

In the following experiments, We combine GB with different pooling layers. The results show that GB can improve the performance of all pooling operations, which demonstrates that GB can selectively enforce informative features and suppress the noise in the features. Compared with others, GMP+GB is the best match, which can better utilize the relationship between the image area and the noun phrase in the text than Average Pooling.

Loss functions: In the second ablation study, Table 3 illustrates the impact of the CMPM loss and CMPC loss in image-text embedding learning on the CUHK-PEDES dataset. We notice that the CMPM loss alone achieves competitive results. While the performance of CMPC loss is common, the rank-1 rate of CMPM loss is increased from 54.20% to 55.71%, via the CMPC loss. The results verify that CMPC loss further improves the performance of CMPM loss and two losses can work together to improve the final retrieval results. On the one hand, CMPM loss increases the variance between unmatched samples and the association between the matched ones. On the other hand, CMPC loss does help to regularize the model.

Impact of description length: We then investigate the influence of different description lengths. As shown in Table 4, DCMG yields a steady improvement throughout the increasing of description length and DCMG with fixed-length of 120 achieves the best results. We argue that the long description can ensure the integrity of the description and avoid the risking of losing critical clues, because 120 is the max length of samples. What is more, CNN overcomes the limitation of parallelization at longer sequence lengths, thus, the additional computational overhead is acceptable.

Discussion of word embedding: In this section, we compare several word embeddings with BERT, i,e., glove50, glove100, glove200, and glove300, which extract a 50-D, 100-D, 200-D, and 300-D word embedding for each word, respectively. Note that we remove the words, which have not appeared in the training set as well as dictionary. For CUHK-PEDES, the dictionary size is 7012. When testing, the missing words in the dictionary will also be removed in advance. Glove initializes the embedding layer and then generates word embedding. Meanwhile, the embedding layer is fixed during the whole training step.

As shown in Table 5, different types of glove word embeddings have similar performance in terms of rank-1. The rank-1 rate of BETR outperforms the glove300 by 2.94%, which the best result of glove word embeddings. The results indicate that BERT can successfully be combined with text CNN and significantly improve the performance of the model, which originates from that BERT learns the contextual relations between the words in a textual description and introduces a word masking mechanism to learn robust word embedding.

Discussion of Training Strategies: To avoid compromising one modality with another and gain better insight into the joint training of a multi-modal system, the retrieval performance of four training strategies defined by various initialization of the weights and learning schemes has been analyzed. For text CNN, we initialize it with Xavier initialization [40]. For vision CNN, we initialize it with a pre-trained ImageNet model, ResNet-50, except from strategy 1, which initializes vision CNN with Xavier initialization. Specifically, strategy 1 and strategy 3 train the two CNNs simultaneously. Strategy 2 fixes the vision CNN and updates the parameters of the text CNN throughout the training process. Strategy 4 fixes the parameters of pre-trained visual CNN and only trains the other model parameters on stage 1. Then, the whole network is trained until it converges on stage 2.

Table 6 compare the performance of different training strategies. Compared with strategy 2, strategy 3 with ImageNet initialization increases the rate of rank-1 from 53.27% to 54.10%. It indicates that ImageNet initialization offers a better starting point and a wider learning surface as compared to the visual CNN training from scratch. Thus, the choice of initial weights is greatly affecting the results. For strategy 2, freezing the parameters of visual CNN and only tuning the parameters of the text CNN throughout the training phase will greatly reduce the accuracy of the retrieval results. On the contrary, strategy 4 achieves the best results, which fixes the vision CNN in the first stage and tunes the vision and text CNN simultaneously in the second stage, due to the fact that it may be hard to train the two CNNs simultaneously at the beginning. The results suggest that our two stage strategy is most suitable to be adopted in the experiments.