OpenGait: Revisiting Gait Recognition Toward Better Practicality

As shown in Figure 2, our developed OpenGait covers the following highlight features.

Technically, we follow the design of most PyTorch deep learning projects and divide OpenGait into three modules, data, modeling, and evaluation, as shown in Figure 2.

Data module contains data loader, data sampler and data transform, which are responsible for loading, sampling, and pre-processing the input data flow respectively.

Modeling module is built on top of a base class (BaseModel) that pre-defines many behaviors of the deep model during training and testing phases, including optimization and inference. The four essential components of current gait recognition algorithms, namely backbone, neck, head, and loss, can be customized in this class.

Evaluation module is used to evaluate the obtained model. It is well known that different datasets have various evaluation protocols, and we unify them into OpenGait to free researchers from these tedious details.

We notice that most previous works only verify the effectiveness on the indoor gait datasets, i.e., CASIA-B[41] and OU-MVLP[34], and the further ablation study is only conducted based on the CASIA-B. In this section, excluding reproducing the officially stated performance, we re-conduct some ablation studies based on the newly-built outdoor dataset, i.e., Gait3D, to experimentally check the algorithm’s robustness for practical gait data.

In our view, there are three possible reasons causing the failure of the SMPL branch: a). Though the SMPL model is usually visualized as a dense mesh, its feature vector only possesses tens of dimensions that present the relatively sparse characterization of body shape and posture, making it challenging to enhance the fine-grained description of gait patterns. b). Since the SMPL model is not recognition-oriented, purposefully fine-tuning it may be more optimal than directly utilizing it to depict the subtle individual characteristics [16]. c). In the wild, estimating an accurate SMPL model that finely captures body shape and posture from a single RGB camera is still challenging. In a nutshell, introducing 3D geometrical information from the SMPL model to benefit gait representation learning is well worth further exploration.

Building on the findings and insights gained through our analyses, we believe that the model structures of previous gait recognition methods may not be robust enough. This is partly due to the fact that earlier research was more lab-oriented and relied on limited datasets that did not capture the complexity of outdoor environments. We provide a more detailed explanation below.

Recently, new large-scale outdoor gait datasets, such as GREW [48] and Gait3D [45], have emerged to promote gait recognition from in-the-lab setting to in-the-wild scenario. However, despite these developments, most lately-published works [44, 42, 39] still rely excessively on indoor gait datasets. In order to advance the development and applications of gait recognition systems, we suggest the community pay more attention to outdoor datasets.

Totally, conducting a comprehensive ablation study on various large-scale datasets can provide a more robust hyper-parameter configuration and model structure.

We employ several easy and widely-accepted modules to form GaitBase. As shown in Figure 4, GaitBase follows the popular set-based and part-based paradigm and takes a ResNet-like [10] network as the backbone. Specifically, the ResNet-like backbone will transform each input silhouette frame into a 3D feature map with the height, width, and channel dimensions. And then, a Temporal Pooling module will aggregate the obtained sequence of feature maps by performing the maximization along the temporal dimension, outputting a set-level understanding of the input gait sequence, i.e., a 3D feature map. Next, the obtained feature map will be horizontally divided into several parts, and each part will be pooled into a feature vector. Accordingly, we get several feature vectors and further use a separate fully connected layer to map them into the metric space. Finally, we employ the widely-used BNNeck [22] to adjust feature space, and the separate triplet and cross-entropy losses are utilized to drive the training process. The overall loss function is formulated by $L=L_{triplet}+L_{ce}$.

ResNet [10] is one of the most successful deep models and has been broadly used as the backbone for many vision tasks. As for gait recognition, most of the current works still take a shallow stack of several standard or customized convolution layers as the backbone. In this paper, we develop a ResNet-like network to serve as the backbone of GaitBase. As shown in Table 5, the network comprises initial convolution layer followed by a stack of four basic residual blocks. All the layers are equipped with BN [14] and ReLU activation layers which are skipped for conciseness.

In order to match the practical usage, we explore the data augmentation (DA) strategy specific to gait silhouettes that previous works have seldom considered. Our experiments show that several spatial augmentation operations, such as horizontal flipping, rotation, perspective, and affine transformations, can improve recognition accuracy. Additionally, we discover that allowing for an unfixed length of the input gait sequence during the training phase can further enhance the final performance.

The driving philosophy behind the development of GaitBase is the belief that simplicity is beautiful. Accordingly, we adopt several succinct, widespread, and validated techniques to make GaitBase robust and efficient. These include treating a gait sequence as a set, using part-based models for human body description, utilizing the widely used ResNet-like network as a backbone, and leveraging the general data augmentation strategy to fit the practical usage. These settings do not require any unique designs but still reach comparable or even better performance on the existing gait datasets, regardless of indoor or outdoor ones.

Four public gait datasets are utilized, involving the most widely-used indoor datasets, CASIA-B [41] and OU-MVLP [34], and the largest gait dataset in the wild, GREW [48] and Gait3D [45]. Table 6 displays the statistics on the number of identities and sequences that are covered in each of these datasets. The following section provides a detailed overview of the collection process for each of these datasets, emphasizing the significant differences that exist between indoor and outdoor datasets.

CASIA-B. Each subject is asked to follow a particular course with three walking conditions, i.e., normal walking (NM), walking with bags (BG), and walking with coats (CL). The obtained videos were captured by an all-side camera array with 11 filming viewpoints. The gait silhouettes were generated through the old background subtraction algorithm.

OU-MVLP is one of the largest public gait datasets. However, similar to CASIA-B, each subject in OU-MVLP was required to walk in a fixed course with fixed cameras and produce the gait silhouette by the outdated background subtraction algorithm. OU-MVLP contains only one walking status for each subject and thus lacks the clothing and carrying changes, making the recognition task relatively easy.

GREW is the largest gait dataset in the wild up to date, to our knowledge. Its raw videos are collected from 882 cameras in a large public area, containing nearly 3,500 hours of 1,080$\times$1,920 streams. Besides the tens of thousands of identities, many other human attributes have been annotated, e.g., 2 gender, 14 age groups, 5 carrying conditions, and 6 dressing styles. Therefore, GREW is believed to include adequate and diverse practical variations.

Gait3D is a large-scale gait dataset in the wild. It was collected in a supermarket and contains 1,090 hours of videos with 1,920$\times$1,080 resolution and 25 FPS.

Our implementation follows official protocols, including the training/testing and gallery/probe set partition strategies. Rank-1 accuracy is used as the primary evaluation metric.

Table 7 displays the main hyper-parameters of our experiments. All the source codes are based on OpenGait, which is proposed in this work. The input resolution for all datasets is set to $64\times 44$. We set the triplet loss margin as 0.2, the initial learning rate and the weight decay of SGD optimizer [28] as 0.1 and 0.0005, respectively. The input frame number is unfixed. Furthermore, we employ Horizontal Flipping, Random Erasing [47], Rotation, Perspective Transformation, and Affine Transformation as spatial data augmentation techniques. Their details are presented in the supplementary.

We compare our GaitBase with other published state-of-the-art methods, e.g., GaitSet [2], GaitPart [6], GaitGL [19], CSTL [12], 3DLocal [13], and SMPLGait [45], on the various popular datasets, i.e., CASIA-B [41], CASIA-B* [17], OU-MVLP [34], GREW [48] and Gait3D [45]. Among these datasets, CASIA-B* [17] is a re-segmented version of CASIA-B.

As shown in Table 8, excluding CASIA-B, GaitBase achieves a totally competitive or even significantly better performance than other state-of-the-art methods. More importantly, GaitBase exceeds other methods by a more considerable margin on the datasets captured from practical scenarios, i.e., +9.5% on GREW and +18.3% on Gait3D, richly exhibiting both the effectiveness and robustness of our strong baseline at the time of dealing the troublesome real-world noisy factors.

As for CASIA-B, our GaitBase achieves a less competitive accuracy. It illustrates that previous methods are finely designed for the CASIA-B dataset to achieve high accuracy while ignoring its bias from wild data. Additionally, our method still achieves relatively high performance on re-segmented CASIA-B*.

To study the effectiveness of BNNeck and DA, we conduct additional experiments on all the mentioned datasets, as shown in Table 8.

As shown in Fig. 5, we perform various spatial augmentation techniques to enlarge the data space and avoid overfitting of the model. We conduct an ablation study on two commonly used indoor and outdoor datasets, i.e., CASIA-B*²²2The conclusions obtained from the experiments on CASIA-B and CASIA-B* are consistent. Here we only present the results on CASIA-B* for brevity. and Gait3D, to evaluate the efficacy of these approaches experimentally. The results are shown in Table 9.

Horizontal Flip can largely simulate a mirror transformation of the filming viewpoint. In (a), we observe that although it fails to improve the performance of CASIA-B*, it significantly improves accuracy on Gait3D. This can be explained by the fact that CASIA-B* is recorded in a laboratory environment using an all-sided camera array, while Gait3D is captured in real-world conditions with comparatively fewer viewpoint changes per subject.

From the experiment (b) in Table 9, Rotation technique slightly benefits both CASIA-B* and Gait3D.

Perspective Transformation aims to simulate the effects of different camera heights. As shown in (c), our experiments indicate that this technique only has a significant impact on Gait3D dataset. The cause should be that there are no camera height changes in CASIA-B*, whereas there are such changes in Gait3D.

From the experiment (d) in Table 9, it appears that Affine Transformation is not able to effectively simulate the noisy factors present in both indoor CASIA-B* and outdoor Gait3D datasets, thereby failing to bring any performance gain on these datasets.

The main goal of Random Erasing [46] is to simulate the occlusion conditions and avoid the over-fitting problem in the spatial dimension. From (e), we note that the erasing operation is challenging to simulate the practical occlusion cases and thus makes almost no difference on Gait3D.

Building upon the above experimental results, we apply the Rotation and Random Erasing for the indoor datasets such as CASIA-B*, CASIA-B and OU-MVLP. On the other hand, for outdoor datasets like Gait3D and GREW, we employ the combination of Horizontal Flip, Rotation, and Perspective Transformation as the augmentation strategy. As evident from (f) and (g), it can be observed that the data augmentation approach brings accuracy improvements of 1.9% and 7.7% on CASIA-B* and Gait3D, respectively.

Based on the above analysis, we again expose the sizable gap between the experimental and practical gait data. Therefore, we propose that the research community should focus more attention on outdoor gait datasets for better practicality.

In this subsection, we investigate the effect of random training input length on the final recognition performance. As shown in Fig. 6, we observe that the fixed length input works relatively optimal for the indoor dataset such as CASIA-B, CASIA-B* and OU-MVLP. On the other hand, the use of random length input yields superior performance for outdoor datasets, such as GREW and Gait3D. Theoretically, similar to the popular Dropout [33] technique, the usage of random sequence length can maintain consistency in features, regardless of the input length, thus easing the over-fitting problem in the temporal dimension. In laboratory-acquired datasets, frame dropping and walking speed fluctuations are minimal, thereby resulting in a uniform number of frames in the gait cycle. As a result, the random training input length has little impact on indoor datasets such as CASIA-B, CASIA-B*, and OU-MVLP, which may be attributed to this consistent nature.

This paper presents a comprehensive benchmark study towards gait recognition applications, which includes a flexible codebase, a series of experimental reviews, and a robust baseline. In addition, here we highlight some subsequent works that are worth further exploration.