Adaptive Deep Metric Embeddings for Person Re-Identification under Occlusions

The overall framework of the proposed method is illustrated in Fig. 1. For each input pedestrian image $I$, we firstly use a base convolutional neural network (in this paper, ResNet [22] is used due to its superiority), to extract mid-level convolutional feature maps. For notational simplicity, we refer to the output of the last convolutional layer of ResNet as $g(I)$ for each input image. Specifically, the last convolutional layer is denoted as $g\in\mathbb{R}^{C\times H\times W}$ ($C=512$, $H=8$, $W=4$ in this paper), where $H$ and $W$ denote the spatial size (i.e., height and width) of the last convolutional layer and $C$ is the number of feature channels. Secondly, the global features and spatial encoded local features (SELF) are respectively extracted in the global and local branches. In the global branch, we extract the global features by applying global average pooling (GAP) to the mid-level feature maps to capture high-level semantics. Meanwhile, in the local branch, the local features are extracted by applying average pooling (AP) to each row of the mid-level feature maps, and then using a $1\times 1$ convolutional layer to reduce the number of feature channels from $C$ to $c$ ($c=128$ in this paper). These local features are fed into the LSTM layer to learn the spatial dependencies between different local regions of the input pedestrian image (see Section 2.2). The output of LSTM layer is SELF, which is represented as $L\in\mathbb{R}^{c}$. Thirdly, the global features and SELF are concatenated to represent the pedestrian image, which contains complementary information from the different levels of semantics, following a fully-connected (FC) layer to get a compact representation. Finally, the deep neural network is jointly optimized by the softmax loss and the proposed adaptive nearest neighbor loss (see Section 2.3).

In this section, SELF is extracted to effectively capture the dependencies of spatial structure in the pedestrian image. Specifically, the pedestrian image is decomposed into a sequence of local regions from head to foot, where each local region has a relatively fixed position due to the prior knowledge about the human body structure. Based on the local regions, different local features are extracted accordingly. In this manner, all local features are treated as a spatial sequence. Following that, the sequence data consisting of all the local features are represented as $S_{t}\in\mathbb{R}^{c}$ ($t=1,\cdots,H$), where $S_{t}$ is the local features in each row and $H$ denotes the sequence length (i.e., the number of local regions). The LSTM layer sequentially accepts the input local features and the hidden state $h_{t}\in\mathbb{R}^{e}$ at each step $t$ is obtained using the following equations ($e$ is the number of hidden units of the LSTM layer. In this paper, we empirically set $e$ to be equal to $c$).

where $i_{t}\in\mathbb{R}^{e}$, $f_{t}\in\mathbb{R}^{e}$, $o_{t}\in\mathbb{R}^{e}$, $g_{t}\in\mathbb{R}^{e}$ and $d_{t}\in\mathbb{R}^{e}$ are the input gate, forget gate, output gate, cell state candidate and cell state, respectively. $sigm\in\mathbb{R}^{e}$ and $tanh\in\mathbb{R}^{e}$ denote the non-linear activation functions (i.e., the sigmoid function and the tanh function), which are applied in element-wise. $W_{L}\in\mathbb{R}^{4e\times(c+e)}$ denotes the weight matrix of the LSTM layer. $\odot$ denotes the element-wise multiplication.

From Eq. (1), $i_{t}$, $f_{t}$ and $o_{t}$ decide which information will be updated, thrown away and outputted, respectively, according to the previous hidden state $h_{t-1}$ and the current input $S_{t}$. A $tanh$ activation layer creates the cell candidate state $g_{t}$. In Eq. (2), the LSTM layer updates the old cell state $d_{t-1}$ by firstly multiplying $f_{t}$ (throw away the old information), and adding the cell candidate state scaled by $i_{t}$ (update the information), to obtain the cell state $d_{t}$. In Eq. (3), the LSTM layer passes the cell state $d_{t}$ through $tanh$ (constrain the output values between $-1$ and $1$) and multiplies it by the output gate $o_{t}$ (output the critical information and reduce the noises), to obtain the hidden state $h_{t}$.

The final hidden state of the LSTM layer is SELF (i.e., $L=h_{H}$, where $L\in\mathbb{R}^{c}$), which effectively captures the spatial relationship between different local regions. Since partial occlusions only affect some local regions, we exploit the intrinsic relationship between different local regions (recall that the occluded regions are considered as noises that can be filtered by LSTM) to alleviate the problem of occlusions. Therefore, the SELF extracted by the LSTM layer is highly robust to occlusions.

As we discuss previously, hard triplet mining is critical for the triplet loss (note that most triplets are uninformative). Although some methods [23, 24] have been developed for hard triplet mining, the selected triplets only exploit pairwise distance information, which may result in the local optima.

Motivated by the above issues, we propose a novel loss function, termed the adaptive nearest neighbor (ANN) loss, which effectively takes advantage of the neighborhood information to enlarge the inter-class dispersion while preserving the intra-class compactness. To be specific, the ANN loss is defined as follows,

where $[\cdot]_{+}$ denotes the hinge loss. $B$ is the number of training samples. $f(\cdot)$ is the function that maps the raw image to the metric embedding representation. $I_{a}$, $I_{pk}$ and $I_{nk}$ represent the anchor sample, positive sample, and negative sample, respectively. $D_{ap}$ and $D_{an}$ respectively denote the average distance between the anchor sample $I_{a}$ and the $K_{a}$ hardest positive samples (i.e, the $K_{a}$ farthest positive samples), and the average distance between $I_{a}$ and the $K_{a}$ hardest negative samples (i.e., the $K_{a}$ closest negative samples). $m$ is a margin to keep the separation between positive and negative pairs. $\left\|\cdot\right\|_{2}$ denotes the Euclidean distance. Note that $K_{a}$ denotes the number of positive/negative samples in the neighborhood of the anchor sample $I_{a}$.

In this paper, instead of fixing $K_{a}$ to be a constant value, we adaptively set $K_{a}$ based on the classification uncertainty. That is, $K_{a}$ is formulated as follows:

where $H_{a}=-\sum_{j=1}^{N}p_{a}^{j}log(p_{a}^{j})$ denotes the classification uncertainty of the anchor sample $I_{a}$. Here, $p_{a}^{j}$ is the probability that the sample $I_{a}$ belongs to the $j$-th class according to a softmax layer and $N$ is the number of classes.$\left\lfloor\cdot\right\rfloor$ denotes the ceil operation. $K_{0}$ is a constant, denoting the minimum number of nearest neighbors (we set $K_{0}$ to be 1 in this paper).

The classification uncertainty $H_{a}$ measures the confidence of classification based on the softmax classifier, which intrinsically characterizes the global data distribution. When the value of $H_{a}$ is high, the anchor sample is considered as the hard-classified sample (the probability of classifying the sample using the softmax classifier is around $1/N$ for each class). In this case, the number of neighbors $K_{a}$ should be increased and vice versa. Therefore, ANN effectively integrates both global and local information of the training data into metric embedding learning, which can successfully overcome the problems of slow convergence and poor optima in the triplet loss. Besides, different from the triplet loss (which only exploits the pairwise distance), the ANN loss considers the average distance between the anchor and the neighborhood of positive/negative samples, which makes the training process quickly converge.

Finally, to learn both separable and discriminative features, we combine the softmax loss (denoted as $L_{s}$) with the ANN loss to jointly optimize the deep neural network, that is,

where $\lambda$ is the tradeoff parameter used to balance the two loss functions.

To verify the effectiveness of the proposed method, we perform extensive experiments on four challenging person ReID datasets, including Market1501 [25], DukeMTMC-reID [26], CUHK03 [27], and Partial REID [28]. The Market1501 dataset contains 1,501 identities captured by six camera views, where the dataset is split into 12,936 training images with 750 identities and 19,732 gallery images with 750 identities. The DukeMTMC-reID dataset contains 1,404 identities collected from eight cameras. The dataset is divided into 16,522 training images with 702 identities and 17,661 gallery images with 702 identities. The CUHK03 dataset contains 13,164 images with 1,360 identities

captured by six cameras. Each identity is observed by two disjoint camera views, yielding an average of 4.8 images in each view. The Patial REID dataset contains 600 images of 60 identities, with 5 full-body images and 5 partial occluded images for each identity.

We use the standard metrics, including the mean Average Precision (mAP) and the Cumulative Matching Characteristic (CMC) curve at rank-1, to evaluate the performance of person ReID.

To observe the influence of the parameters on the proposed method, we evaluate two critical parameters in the proposed method, i.e., the neighbor size (i.e., $K_{a}$ in Eq. (7)) and the tradeoff parameter to combine two losses (i,e., $\lambda$ in Eq. (8)). The rank-1 accuracy with different values of $K_{a}$ and $\lambda$ on the Market1501 dataset is given in Fig. 2.

From Fig. 2, we can see that the proposed method with the adaptive value of $K_{a}$ can achieve much better results than that with the fixed value of $K_{a}$, which demonstrates the importance of adaptive neighborhood. The value of $\lambda$ also significantly affects the final performance. In summary, when $\lambda$ is set to 1 and $K_{a}$ is adaptively set based on the classification uncertainty $H_{a}$, the proposed method achieves the best performance. In the following, the value of $\lambda$ is fixed to 1.

In this section, we evaluate several variants of the proposed method to verify the effectiveness of the key components in the proposed method for person ReID under occlusions. We conduct the experiments on the Market1501 dataset, where the query images are set with the different levels of occlusions. More specifically, we randomly occlude a region with random values in an image with the aspect ratio $s$. The aspect ratio of occluded area is set within the range of [0.0, 0.6] (please refer to [19] for more details). Moreover, we also conduct the experiments on the real and challenging occlusion dataset, the Partial REID dataset, which contains different types of severe occlusions. The partial occluded images are used as the query images and the full-body images are used as the gallery images.

The proposed method contains two key components: LSTM, which exploits the spatial dependencies between the different local regions to enable the model to be robust to occlusions; and the ANN loss, which learns discriminative metric embeddings.

Therefore, nine different variants of the proposed method are evaluated. That is, (1) The baseline method (denoted as $RN_{S}$) that only uses the global branch and a ResNet model [22] based on the softmax loss. (2) The method (denoted as $RN_{A}$) that uses the same network as $RN_{S}$, where both the softmax loss and the ANN loss are employed to jointly optimize the network. (3)-(6) The methods (respectively denoted as $RNCONV_{A}$, $RNFC_{A}$, $RNRNN_{A}$, and $RNLSTM_{A}$) that employ a ResNet model, and combine the global branch and the local branch (here the convolutional layer, fully-connected layer, RNN layer and LSTM layer are respectively used as the local branch), where the softmax loss and the ANN loss are used. Note that $RNLSTM_{A}$ is the proposed method in this paper. (7) The method (denoted as $RNLSTM_{S}$) that uses the proposed network, where only the softmax loss is used. (8)-(9) The methods (respectively denoted as $RNLSTM_{C}$, $RNLSTM_{T}$) that uses the proposed network, where the softmax loss is combined with the contrastive loss [29] and triplet loss [23], respectively. The details of the nine variants are summarized in Table 1.

The rank-1 accuracy obtained by the nine different variants on the Market1501 and Partial REID datasets is shown in Fig. 3, where Fig. 3(a) shows the robustness of different variants against the different levels of occlusions.

From Fig. 3(a), we have the following conclusions: (1) In general, the recognition performance obtained by all the variants on Market1501 drops when the aspect ratio of occluded area increases. This further demonstrates the challenging task of person ReID under occlusions. (2) By comparing the rank-1 accuracy obtained by $RN_{S}$ and $RN_{A}$, the ResNet (only with global branch) jointly optimized with the softmax and ANN losses obtains better performance than that only using the softmax loss. This is mainly because that the joint loss enhances the discrimination ability of the model. However, the improvements are not significant, due to the fact that ResNet only considers the high-level semantic information and ignores the local information, which is critical for classification under occlusions. (3) By comparing the variants ($RNCONV_{A}$, $RNFC_{A}$, $RNRNN_{A}$ and $RNLSTM_{A}$) with different local branches, we can see that LSTM plays a critical role in the final performance. Specifically, compared with the variants without LSTM ($RNCONV_{A}$, $RNFC_{A}$, $RNRNN_{A}$), the variant with LSTM (the proposed $RNLSTM_{A}$) can effectively improve the rank-1 accuracy about 3% under different levels of occlusions. This is due to the fact that that LSTM not only memorizes the spatial correlations between the different local regions, but also reduces the noises caused by partial occlusions. (4) The proposed method ($RNLSTM_{A}$) outperforms the softmax loss based method ($RNLSM_{S}$), and the joint loss using contrastive loss ($RNLSTM_{C}$) and triplet loss ($RNLSTM_{T}$) based methods in a reasonable margin (about 3%~6% on Market1501). From this comparison, we can see that the combination of the classification loss and metric loss (i.e., contrastive loss or triplet loss) can improve the performance of the model. Furthermore, $RNLSTM_{A}$ obtains the superiority performance by exploiting the neighborhood information to enlarge the inter-class dispersion while increasing the intra-class compactness. Among all the competing variants, $RNLSTM_{A}$ consistently achieves the best results on the Market1501 dataset. This indicates that the deep model (based on LSTM and CNN) jointly optimized by the softmax loss and the ANN loss, can effectively enhance the robustness to occlusions.

From Fig. 3(b), we can observe similar conclusions on the Partial REID dataset. Note that $RNLSTM_{A}$ performs slightly inferior than $RNRNN_{A}$. This is mainly because that the LSTM layer have more parameters than the RNN layer. In other words, to ensure the effectiveness of the proposed method, a large number of training set are preferred to learn the parameters. However, the training set of Partial REID dataset is small (only 300 images are used for training).

In this section, we compare the proposed method (i.e., $RNLSTM_{A}$) with several representative methods, including the traditional metric learning methods (NPD [30], XQDA[9]) and the recently-proposed deep learning methods (IDE [31], TriNet [23], PAN [32], SVDNet [33] and DFFL [34]).

The rank-1 accuracy and mAP obtained by all the competing methods are shown in Table 2. Compared with the traditional person ReID methods (NPD and XQDA), the deep learning methods achieve significant performance improvements, which show the superiority of deep learning. The proposed method obtains much better results than the softmax-loss based IDE [31] method and the triplet loss based TriNet [23] method, which demonstrates the effectiveness of the proposed ANN loss. Moreover, the proposed method outperforms the part-based method PAN [32] under occlusions, since we exploit the spatial dependencies based on LSTM to learn discriminative representations. Compared with SVDNet [33], the proposed method achieves higher rank-1 accuracy and mAP under small occlusions. However, the proposed method obtains slightly inferior results under large occlusions ($s=0.6$) on the CUHK03 dataset. Although SVD in the FC layer of SVDNet can effectively extract the discriminative information for person ReID, the training complexity of SVDNet is high. DPFL [27] that trains a multi-channel network for multi-scale images achieves slightly better performance than the proposed method that only exploits the single scale image on DUKEMTMC-reID. However, the proposed method obtains better results than DPFL on the challenging CUHK03 database, where each pedestrian contains a relatively small number of training images.