Gait Recognition via Effective Global-Local Feature Representation and Local Temporal Aggregation

The overview of the proposed method is shown in Fig. 1, which aims to extract more comprehensive feature representation for gait recognition, including three key components. First, we use a convolution to extract the shallow features from the original input sequence. Next, the Local Temporal Aggregation (LTA) operation is designed to aggregate the temporal information and preserve more spatial information for trade off. After that, the Global and Local Feature Extractor (GLFE) is implemented to extract the combined feature ensembling both global and local information. Then, we leverage temporal pooling and GeM pooling layer to implement feature mapping. Finally, we choose the triplet loss [3, 5] and cross entropy loss to train the proposed model.

Previous works [3, 5] use a specific pattern “CL-SP-CL-SP-CL” to extract features, where CL means convolutional layer and SP denotes the spatial pooling layer. However, the spatial information may be lost due to the SP downsampling operation. Considering that temporal information in a gait sequence is periodic, we present the LTA operation to replace the first spatial pooling layer, which can integrate temporal information of local clips and maintain more spatial information.

Assume that $X_{in}\in\mathbb{R}^{C_{1}\times T_{1}\times H_{1}\times W_{1}}$ is the input of local temporal aggregation, where $C_{1}$ is the number of channels, $T_{1}$ is the length of the gait sequence and ($H_{1}$,$W_{1}$) is the image size of each frame. The process can be formulated as follows

$$X_{LTA}=f_{a\times a\times a}^{b\times 1\times 1}(X_{in}),$$

(1)

where $f_{a\times a\times a}^{b\times 1\times 1}(\cdot)$ denotes the 3D convolution operation with kernel size $a$ and temporal stride $b$. $X_{LTA}\in\mathbb{R}^{C_{2}\times T_{2}\times H_{1}\times W_{1}}$ is the output of LTA operation.

Except the global gait features [3, 18], some recent researchers propose different gait recognition frameworks which utilize local gait information, as shown in Fig.2(a)(b) [27, 5]. For example, Zhang et al.[27] propose the ACL framework to extract local gait features by using multiple separate 2D CNNs. Fan et al.[5] develop a focal convolutional layer to extract the local features and then combine them. Although it contains more detailed information than the global gait features, the local gait features do not pay attention to the relations among local regions. Hereby, we propose a novel GLFE module to extract features, which can take advantage of both global and local information. The GLFE module is implemented by the GLConv layer, which contains global and local feature extractors. The global feature extractor can extract the whole gait information, while the local feature extractor is used to extract more details from local feature maps. The GLConv has two different structures due to different combinations, e.g. GLConvA and GLConvB. The GLFE module includes four layers, “GLConvA-SP-GLConvA-GLConvB” as shown in Fig.1.

The GLConv layer is shown in Fig.2(c). Assume that its input is $X_{global}\in\mathbb{R}^{c_{1}\times t\times h\times w}$, where $c_{1}$ is the number of channels, $t$ is the length of feature maps and ($h$,$w$) is the image size of each frame. We first partition the global feature map into n-parts as local feature maps $X_{local}=\left\{X_{local}^{i}|i=1,...,n\right\}$, where $n$ is the number of partitions and $X_{local}^{i}\in\mathbb{R}^{c_{1}\times t\times\frac{h}{n}\times w}$ corresponds to the i-th local gait part. Then, we use 3D convolutions to extract global and local gait features, respectively. Note that all local feature maps share the same convolutional weights. There are two ways to combine the global and local feature maps, i.e. by element-wise addition (GLconvA) or by concatenation (GLconvB). The GLconvA and GLconvB layers can be representated as

$$Y_{GLConvA}=Y_{global}+Y_{local}\in\mathbb{R}^{c_{2}\times t\times h\times w}$$

(2)

and

$$Y_{GLConvB}=cat\begin{Bmatrix}Y_{global}\\ Y_{local}\end{Bmatrix}\in\mathbb{R}^{c_{2}\times t\times 2h\times w},$$

(3)

where $cat$ means concatenating operation. $Y_{global}$ and $Y_{local}$ can be represented as

$$Y_{global}=f_{3\times 3\times 3}(X_{global})\in\mathbb{R}^{c_{2}\times t\times h\times w}$$

(4)

and

$$Y_{local}=cat\begin{Bmatrix}f_{3\times 3\times 3}^{{}^{\prime}}(Y_{local}^{1})\\ f_{3\times 3\times 3}^{{}^{\prime}}(Y_{local}^{2})\\ ...\\ f_{3\times 3\times 3}^{{}^{\prime}}(Y_{local}^{n})\end{Bmatrix}\in\mathbb{R}^{c_{2}\times t\times h\times w},$$

(5)

where $f_{3\times 3\times 3}(\cdot)$ and $f_{3\times 3\times 3}^{{}^{\prime}}(\cdot)$ denote 3D convolutions with kernel size 3.

Based on the above two forms of GLGonv layer, the GLFE module can be built to extract gait features after LTA operation. Experimentally, GLConvA is used to implement the first few GLGonv blocks and GLConvB is ulitized to realize the last one in the GLFE module.

Since the length of the input gait sequences may be different, we introduce temporal pooling to aggregate the temporal information of the whole sequence [3, 5]. Assume that $X_{GLFE}\in\mathbb{R}^{C_{3}\times T_{2}\times H_{2}\times W_{2}}$ is the output of GLFE module, where $C_{3}$ is the number of input channels, $T_{2}$ is the length of feature maps and ($H_{2}$,$W_{2}$) is the spatial size of the feature in each frame. Because of the spatial pooling layer in the GLFE module, the spatial size becomes ($H_{2}$,$W_{2}$), while the length of feature maps remains unchanged. The temporal pooling $TP(\cdot)$ can be realized by

$$Y_{TP}=F_{Max}^{T_{2}\times 1\times 1}(X_{GLFE}),$$

(6)

where $F_{Max}^{T_{2}\times 1\times 1}(\cdot)$ means Max-pooling layer. $Y_{TP}\in\mathbb{R}^{C_{3}\times 1\times H_{2}\times W_{2}}$ is the output of temporal pooling.

To improve feature representation ability, researchers develop the spatial feature mapping operation with weighted sum [3, 5]. After temporal pooling, gait feature maps are split into strips and two statistical functions, max and average, are used to aggregate each strip’s information. The spatial feature mapping can be represented as

$$Y_{MA}=\alpha F_{Max}^{1\times 1\times W_{2}}(Y_{TP})+\beta F_{Avg}^{1\times 1\times W_{2}}(Y_{TP}),$$

(7)

where $Y_{MA}\in\mathbb{R}^{C_{3}\times 1\times H_{2}\times 1}$ is the output of spatial feature mapping. However, the weighted sum strategy is inflexible because trade-off parameters are predefined manully.

Hereby, we introduce the Generalized-Mean pooling (GeM) to integrate the spatial information adaptively. The GeM pooling layer $GeM(\cdot)$ can be represented as

$$Y_{GeM}=(F_{Avg}^{1\times 1\times W_{2}}((Y_{TP})^{p}))^{\frac{1}{p}},$$

(8)

where $Y_{GeM}\in\mathbb{R}^{C_{3}\times 1\times H_{2}\times 1}$ is the output of GeM operation. $p$ is the parameter which can be learned by network training. Specifically, if $p=1$, $Y_{GeM}$ is equal to $Avg^{1\times 1\times W_{2}}(Y_{TP})$, and if $p\rightarrow\infty$, $Y_{GeM}$ is equal to $Max^{1\times 1\times W_{2}}(Y_{TP})$. Then, we use multiple separate fully connected layers to further aggregate the information from channels of $Y_{GeM}$. The feature mapping can be defined as

$$Y_{out}={F}_{sfc}(Y_{GeM})\in\mathbb{R}^{C_{4}\times 1\times H_{2}\times 1},$$

(9)

$Y_{out}$ is the output of feature mapping with $H_{2}$ horizontal features, each of which has $C_{4}$ channels. $F_{sfc}(\cdot)$ denotes the multiple separate fully connected layers.

To effectively train the proposed gait recognition model, we simultaneously utilize the triplet loss [8] and cross entropy loss .

The triplet loss can improve the inter-class distance and reduce the intra-class distance, which can help the cross entropy loss to identify Human IDs. In the training stage, each horizontal feature of $Y_{out}$ is fed into the combined loss function to calculate the loss independently. The combined loss function $L_{combined}$ can be defined as

$$L_{combined}=L_{tri}+L_{cse},$$

(10)

where $L_{tri}$ and $L_{cse}$ mean the triplet loss and cross entropy loss, respectively. $L_{tri}$ can be defined as

$$L_{tri}=[D(F(i),F(k))-D(F(i),F(j))+m]_{+},$$

(11)

where $i$ and $j$ are the samples from the same class A, while $k$ represents the sample from class B. $F(\cdot)$ denotes the feature extraction and mapping operation of the proposed method. $D(d_{1},d_{2})$ is the Euclidean distance between $d_{1}$ and $d_{2}$. $m$ is the margin of the triplet loss. The operation $[\gamma]_{+}$ is equal to $max(\gamma,0)$.

We evaluate the performance of the proposed method on two commonly used datasets, i.e. CASIA-B [24] and OUMVLP [19] datasets.

We adopt the same preprocessing approach as [3] to obtain gait silhouettes for CASIA-B and OUMVLP datasets. The image of each frame is normalized to the size $64\times 44$. The network parameters are shown in Table.1. $m$ in Equ.11 is set to 0.2. $p$ in Equ.8 is initialized to 6.5. The batch size parameters P and K are both set to 8 in CASIA-B dataset. Since OUMVLP dataset is much larger than CASIA-B, the batch size P$\times$ K is set to 32$\times$8 = 256. During the training stage, the length of input gait sequences is set to 30. During the test stage, the whole gait sequences are put into the proposed model to extract gait features. In the setting of ST, MT and LT, the epoch number is set to 60K, 80K and 80K, receptively. All experiments take Adam as the optimizer. The weight decay is set to 5e-4 for CASIA-B dataset. For the setting of ST, the learning rate is set to 1e-4. For the settings of MT and LT, the learning rate is first set to 1e-4, and reset to 1e-5 after 70K iterations. For the OUMVLP dataset, the epoch number is set to 210K. The learning rate is first set to 1e-4, and reset to 1e-5 and 5e-6 after 150K and 200K iterations, respectively. In particular, since OUMVLP has 100 times more IDs than the CASIA dataset, we add the label smooth operation into the cross entropy loss [9]. The weight decay is first set to 0 and reset to 5e-4 after 200K for OUMVLP dataset.

We first explore the effect of different conditions (NM, BG, and CL). From Table. 2 we can observe that the accuracy has significantly decreased when the external environment changes. For example, the recognition accuracy of GaitPart in three conditions NM, BG, and CL is 96.2%, 91.5%, and 78.7% with the setting of LT. For the proposed method, the recognition accuracy in these conditions is 97.4%, 94.5%, and 83.6%, which outperforms GaitPart by 1.2%, 3.0%, and 4.9%, respectively. The experimental results show that the proposed method has significant advantages in the BG and CL conditions, indicating that the proposed model can extract more robust gait features. Even in the ST and MT setting, we can see the similar recognition results that the proposed method also obtains the best performance. On the other hand, human gait can be collected from arbitrary view angles and conditions in a real scenario. Hereby, we focus on the robustness of gait recognition under various external environment factors. Based on Table. 2, we further calculate the average accuracy of three conditions. Table.3 indicates that the comparison results of the proposed method with state-of-the-art gait recognition methods, including GaitSet [3] and GaitPart [5]. It can be observed that the average accuracy of the proposed method is 91.8%, which outperforms GaitSet and GaitPart by 7.6% and 3.0%, respectively.

We also discuss the comparison results with different setting of dataset scales. Chao et al. [3] provide an evaluation protocol, in which the performance of the method is evaluated on three different scales, i.e. ST, MT, and LT on CASIA-B. In this paper, we present the complete experimental results with these three settings. The experimental results are shown in Table. 2. We can observe that the recognition accuracy of the GaitSet [3] is 79.5%, 92.0%, and 95.0% for three settings under the condition of NM, respectively. For the proposed method, the corresponding recognition accuracy is 86.0%, 95.9%, and 97.4%, which increases by 6.5%, 3.9%, and 2.4%, respectively. Specifically, the proposed method can achieve larger improvement in the setting of small-scale dataset.

In this paper, the proposed recognition framework includes several key modules, e.g. Local Temporal Aggregation, Global and Local Feature Extractor and Generalized-Mean pooling layer. Hereby, we design different ablation studies to analyze the contribution of each key module.