A Survey on 3D Skeleton Based Person Re-Identification:
Approaches, Designs, Challenges, and Future Directions

In the SRID task, when given a skeleton sequence containing the person-of-the-interest from the probe database, the target of the model is to query the identity of this person from the gallery database. Formally, a 3D skeleton sequence can be represented by $\boldsymbol{X}\!=\!(\boldsymbol{x}_{1},\cdots,\boldsymbol{x}_{f})\in\mathbb{R}^{f\times J\times 3}$, where $\boldsymbol{x}_{t}\in\mathbb{R}^{J\times 3}$ denotes the $t^{th}$ skeleton with 3D coordinates of $J$ body joints. Each skeleton sequence $\boldsymbol{X}$ corresponds to a person identity y, where $\text{y}\in\{1,\cdots,\text{C}\}$ and C is the number of different classes ($i.e.$, identities). We use $\Phi_{T}=\left\{\boldsymbol{X}^{T}_{i}\right\}_{i=1}^{N_{1}}$, $\Phi_{P}=\left\{\boldsymbol{X}^{P}_{i}\right\}_{i=1}^{N_{2}}$, and $\Phi_{G}=\left\{\boldsymbol{X}^{G}_{i}\right\}_{i=1}^{N_{3}}$ to denote the Training set, Probe set, and Gallery set that contain $N_{1}$, $N_{2}$, and $N_{3}$ skeleton sequences of different persons collected from different scenes or views. The SRID target is to learn a model that maps 3D skeleton sequences into effective representations, so that we can query the correct identity of an encoded skeleton sequence representation in the probe set via matching it with the sequence representations in the gallery set.

We show the taxonomy of SRID in Fig. 1, which classifies the SRID research from three perspectives: (1) Approaches (Sec. 3); (2) Model designs (Sec. 4); (3) Challenges and future directions (Sec. 5). For SRID approaches, we divide them into three categories, including hand-crafted methods using manually-extracted features ($e.g.,$ skeleton descriptors) (Sec. 3.2) , sequence learning methods that perform sequential representation learning of 3D skeletons (Sec. 3.3), and graph learning methods that model 3D skeletons as graphs (Sec. 3.4), and further subcategorize them according to different learning focuses such as pose or graph dynamics. Corresponding to different levels of skeleton modeling, three key parts of model designs are elaborated, including intra-skeleton modeling (Sec. 4.1) that focuses on body structure and relations, inter-skeleton modeling (Sec. 4.2) that captures skeleton correlations and importance, and skeletal sequential modeling (Sec. 4.3) that learns sequential pose dynamics and motion semantics. In challenges and future directions, we elucidate three main challenges and their potential solutions from skeleton representations (Sec. 5.1), SRID data (Sec. 5.2), and model performance (Sec. 5.3). Different open directions of SRID such as its security and future real-world applications are also discussed (Sec. 5.4).

Benchmark Datasets. Table 2 provides the statistics of commonly-used datasets for evaluating SRID methods, which can be categorized to two types: (1) Sensor-based datasets, in which 3D skeleton data are captured from depth sensors such as Kinect, and (2) RGB-estimated datasets, including CASIA-B-3D and PoseTrackReID-2D, in which skeleton data are estimated from RGB videos using pose estimation models Cao et al. (2019); Chen and Ramanan (2017). Existing SRID datasets typically contain skeleton data collected from varying scenarios such as multiple views (KS20 Nambiar et al. (2017), KinectREID Pala et al. (2015)), appearance and clothe changes (BIWI RGBD-ID Munaro et al. (2014b) IAS-Lab RGBD-ID Munaro et al. (2014c), or/and different illumination conditions (KGBD Andersson and Araujo (2015)), which enables a comprehensive evaluation of both short-term and long-term SRID performance. SRID as an emerging task is currently with a limited number and scale of datasets, which is further discussed in Sec. 5.2.

Models and Algorithms. As summarized in Table 1, Support Vector Machine (SVM), K-Nearest Neighbor (KNN), Multi-Layer Perceptron (MLP), and different metric algorithms (including Euclidean, Hamming, Jaccard distance metrics) are mainstream algorithms used in hand-crafted methods, while LSTM, CNN, MLP and their combination are extensively utilized in sequence learning models to capture sequential dynamics of skeletons. The representative sequence methods contain Bi-LSTM Wei et al. (2020), attention-based LSTM encoder-decoder Rao et al. (2020, 2021b), temporal-spatial CNN Liao et al. (2020), etc. The recent advancements of graph learning methods are primarily focused on the development of graph attention network (GAT), Transformer, and their variants, including graph relation networks (GRN) Rao et al. (2021a) and skeleton graph transformer Rao and Miao (2023).

Early research endeavors Barbosa et al. (2012); Munaro et al. (2014b); Andersson and Araujo (2015); Khamsemanan et al. (2017); Nambiar et al. (2017) design skeleton or body-joint descriptors in terms of anthropometric ($e.g.,$ bone lengths) and gait attributes ($e.g.,$ speed, joint angles) for person re-ID. For example, Euclidean distances between different joint pairs are computed as descriptors in Barbosa et al. (2012), which are followed and extended by different studies Munaro et al. (2014b); Pala et al. (2019) using domain knowledge such as human anatomy. These hand-crafted features are learned by different classifiers ($e.g.$, KNN, SVM) to perform person re-ID. They are also combined with different metric algorithms Pala et al. (2015); Rao et al. (2022) or other modalities such as 3D point clouds and face descriptors Gharghabi et al. (2015); Bondi et al. (2016); Pala et al. (2019); Munaro et al. (2014a) to further boost the person re-ID accuracy.

Hand-crafted methods possess relatively high explainability and efficiency: (1) They manually extract features based on well-established skeleton knowledge, which can be explained by domain experts Yoo et al. (2002). (2) Most of these methods utilize classic machine learning models ($e.g.,$ SVM) that are supported by solid theoretical foundations. Their model size and computational complexity are usually lower than deep neural networks, leading to higher efficiency under the same performance. However, they often lack the ability to fully represent advanced motion-related concepts ($e.g.,$ motion consistency) or latent high-level features beyond human cognition, which might limit their performance on capturing complex class-related motion patterns. Another limitation is their inferior performance compared to data-driven deep learning techniques, especially on large-scale datasets Zhou et al. (2017).

Sequence learning methods typically leverage deep neural networks to learn pose dynamics or latent motion semantics from consecutive 3D skeletons: (1) Pose Dynamics Models: Wei et al. (2020); Rashmi and Guddeti (2022) exploit LSTM Hochreiter and Schmidhuber (1997) and its variants to capture temporal dynamics of pose features such as body part lengths, joint distances, relative joint positions, joint angles within a gait cycle for SRID, while Huynh-The et al. (2020); Liao et al. (2020) further model poses as spatial geometric features and temporal statistic features, which are encoded by coarse-to-fine level convolutional architectures; (2) Latent Motion Semantics Models: AGE Rao et al. (2020) and SGELA Rao et al. (2021b) devise self-supervised high-level semantics tasks including skeleton sequence reconstruction, sorting, and prediction to encourage the LSTM-based encoder-decoder model to learn sequential dynamics and useful motion/gait semantics such as motion continuity and local skeleton/joint correlations (defined as locality in Rao et al. (2021b)) for person re-ID. SimMC Rao and Miao (2022a) proposes an MLP-based skeleton prototype learning framework with intra-sequence similarity learning and masked contrastive learning, which respectively facilitate learning local motion continuity and high-level class semantics for SRID. Hi-MPC Rao et al. (2023) further designs hierarchical prototype learning with a hard skeleton mining approach to learn multi-level motion semantics of key informative skeletons in an unsupervised manner without using skeleton labels.

Sequence learning methods can automatically model sequential dynamics of skeletons to learn latent effective representations without necessitating external domain knowledge. They typically do not require complicated pre-modeling such as graph modeling and can directly use raw or standardized skeleton sequences for learning, which could possess lower model complexity and less model parameters than graph learning models Rao and Miao (2022a). However, as these methods do not explicitly model intrinsic structural relations and motion correlations of human body, they might largely ignore some valuable skeleton patterns for SRID. On the other hand, they usually require devising effective learning tasks such as skeleton sequence prediction to enhance skeleton or motion semantics learning.

Recently, 3D skeleton graph models have been widely employed to capture body structural and actional features, which can be roughly divided into two categories: (1) Graph Dynamics Models: MG-SCR Rao et al. (2021c) constructs multi-level skeleton graphs based on physical connections of body joints or parts, and utilize GAT Velickovic et al. (2018) and LSTM to model joint relations and graph dynamics for person re-ID. In SM-SGE Rao et al. (2021a), a self-supervised multi-scale skeleton reconstruction and prediction mechanism is further integrated to enhance the discriminative graph dynamics learning for SRID; (2) Graph Prototype Models: Combining structural-collaborative body relation modeling based on GAT, SPC-MGR Rao and Miao (2022b) proposes an unsupervised clustering-contrasting paradigm to contrast and learn the most representative features, $i.e.$, feature cluster centroids (termed prototypes), from different-level skeleton graph representations for person re-ID. In TranSG Rao and Miao (2023), a skeleton graph transformer is devised to simultaneously learn both structural and actional relations of body joints, while the model utilizes ground-truth skeleton labels to generate more reliable graph prototypes to capture richer class-specific semantics for SRID.

Graph learning methods explicitly model human body structure and relations based on the constructed skeleton graphs, which can mine richer structural features and motion patterns than non-graph methods. In contrast to sequence learning methods that learn latent motion semantics, graph-based models can offer more intuitive explanations such as visualization of joint nodes and relations Rao and Miao (2023), thereby aiding in selecting key features and further improving the model. Despite several advantages, these methods typically require pre-defining skeletal topology ($i.e.,$ mapping body joints to specific parts) to construct graphs of different levels, and necessitate combining multi-level or/and multi-stage relation modeling Rao et al. (2021c) for effective intra-skeleton and inter-skeleton feature learning. Recent efforts such as TranSG Rao and Miao (2023) unify different relation modeling into a transformer-based framework, while they demand multiple learning tasks ($e.g.,$ graph structure prediction) to enhance the capture of valuable graph semantics.

Most existing studies learn skeleton features from a single level ($e.g.,$ body joint level Rao et al. (2020, 2021b)), while they rarely explore a comprehensive multi-level skeleton learning from structural features, body relations, and motion dynamics. This could cause an overlook of valuable hierarchical skeleton information ($e.g.,$ global-local patterns Rao et al. (2023)) and limit the SRID performance.

Solutions and Directions: First, as human body can be naturally modeled with several key functional regions at different levels ($e.g.,$ joints, limbs) Winter (2009), we can exploit different groups of joints to construct higher-level body representations ($e.g.,$ limbs) to characterize anthropometric or kinetic features within body structure from coarse to fine. Second, it is feasible to generalize the joint-level relation modeling to multi-level body representations. We can exploit the multi-grained relations between different-level body parts in terms of physical or kinematic attributes to mine richer body and motion features. Last, by devising and employing motion semantics learning tasks on different level skeleton sequences, the model can not only capture dynamics of local parts such as joints but also learn more high-level semantics ($e.g.,$ identity-related patterns) from global body components such as limbs.

Existing SRID data, $i.e.,$ 3D skeletons, are mainly collected from prevailing depth sensors such as Kinect Shotton et al. (2011), while diverse skeleton collection settings ($e.g.,$ different devices in uncontrollable environments) have not be thoroughly explored. In contrast to existing large-scale RGB-based person re-ID data ($e.g.,$ MSMT17 Wei et al. (2018)), the available data for SRID is relatively scarce and imbalanced. As shown in Table 2, existing Kinect-based SRID datasets contain 4.8 thousand to 205.8 thousand skeletons with less than 200 identities, while the numbers of skeletons belonging to each identity often differ greatly ($i.e.,$ imbalanced class distribution). This might negatively influence the learning and generalization ability of current data-driven SRID models using deep neural networks, with a risk of model over-fitting. On the other hand, either Kinect-based or RGB-estimated skeleton data unavoidably contain noise, which are possibly affected by factors such as device’s tracking distance, illumination changes (which may affect structured light used in Kinect V1), and pose estimation algorithm precision. Such inherent noise puts high demand on the robustness of model against random perturbations.

Solutions and Directions: To address these challenges, more high-quality 3D skeleton data should be collected or generated. First, the number of different identities and the skeleton data size of each identity should be controlled within a reasonable range, $e.g.,$ enlarging the identity size to hundreds and keeping the total data size of each identity as similar as possible. Such balanced data could benefit the model to learn higher inter-identity difference with better robustness against intra-identity diversity Johnson and Khoshgoftaar (2019). To make up for the data scarcity and imbalance in this area and facilitate related research, we will collect and open a large-scale SRID dataset in the future. Second, to reduce skeleton noise and sample more balanced data, it is feasible to devise skeleton denoising models and augmentation strategies for skeleton generation. For example, the GAN-based pose generator Yan et al. (2017) and newly-emerging diffusion models Ho et al. (2020) could be transferred to generate or/and denoise 3D skeleton data, while skeleton-level, sequence-level, and class-level data augmentations can be further explored.

Existing SRID models Rao and Miao (2022a, 2023); Rao et al. (2023) report unstable performance variations that are sensitive to different model parameter initialization, hyper-parameter settings ($e.g.,$ clustering parameters) and the diversity and quality of training datasets. Owing to such non-robust performance, many studies opt for the best-trained model with carefully-selected initialization and parameters, while they often fail to reflect the true average performance of corresponding architectures and typically exhibit limited generalization ability. On the other hand, due to training on a single dataset with limited data sizes, views, scenes or conditions, many SRID methods may only perform well on the scenarios that are similar to that of the training data, while they cannot generalize to more challenging data or real-world scenarios ($e.g.,$ RGB-estimated skeleton data). Another important flaw of existing models is their weak interpretability, as they rarely provide human-friendly explanation for the effectiveness of model architectures, skeleton features, and prediction results. Such “black-box” models might increase the risk of wrong prediction, cause the opacity of performance improvement, and further hinder their large-scale application.

Solutions and Directions: To obtain a more robust SRID model, it is necessary to delve into the investigation of effects of model initialization, the most essential hyper-parameters, and data quality ($e.g.,$ extreme data noise or imbalance) on model performance. We can conduct a comprehensive empirical evaluation of them to find key factors and seek an improvement or control of model robustness. Theoretical analyses for the performance variations in terms of model-approximated functions and convergence conditions Rao and Miao (2022a, 2023) can also be provided for a more robust model design. Moreover, fairer multi-faceted performance evaluation metrics such as performance average and standard deviation should be reported to better evaluate the overall robustness of models.

To improve model generality, a promising future direction is to exploit larger-scale SRID datasets containing diverse scenario settings to learn both domain-specific ($e.g.,$ identity-specific) and domain-general ($i.e.,$ identity-shared) gait features, so as to enable the model to be transferred/generalized to different scenarios. It is also feasible to explore domain adaptation or generalization techniques to combine different datasets for model training and transfer Rao and Miao (2022b). More diverse benchmarks such as RGB-estimated skeleton datasets such as CASIA-B-3D Liao et al. (2020) should be explored for generality evaluation of models.

To provide interpretability, different human-friendly explanation including pose/feature visualization and text description can be considered. There also exist various architecture-specific explanation mechanisms, $e.g.,$ class activation maps (CAM) Zhou et al. (2016) for CNN, knowledge graphs Ji et al. (2021) for graph neural networks (GNN), attention visualization for transformers Rao and Miao (2023), which could be applied to explainable skeleton learning. Inspired by the recent success of large language models (LLMs) such as GPT-4V, they can serve as an agent to join the SRID model training to interpret the learning process, and we can transform skeleton representations into text ($e.g.$, time series) or image inputs ($e.g.,$ pose images) and devise prompts to query the pose/feature importance. We provide a further discussion about prompt-based skeletal foundation models in Sec. 5.4.

Multi/Cross-Modal Learning. Combining 3D skeleton data and other modalities such as RGB images, depth images, radio frequency waves ($e.g.,$ Radar waves) is a promising direction, as they can provide pose or gait information from different dimensions ($e.g.,$ appearances, silhouettes) to better identify different persons. Another direction is to transfer and fuse gait representations across skeleton modality and other modality, which can be the key to achieving more general and scalable skeleton learning for more multi-modal tasks.

Unified Evaluation Protocol. As existing SRID studies adopt either identity classification Rao et al. (2021b) or re-ID matching protocol Rao et al. (2023), a more comprehensive unified evaluation protocol that provides not only accuracy-related metrics ($e.g.,$ Rank-1 accuracy, mAP) but also measures of model generality, robustness, and reliability should be devised. It is also imperative to formulate a fair cross-modality evaluation and comparison protocol that standardizes re-ID settings ($e.g.,$ probe/gallery settings, single/multi-shot recognition) for comparing skeleton-based methods and RGB/depth-based methods or multi-modal methods.

Prompt-Based Skeletal Foundation Model. Motivated by the success of LLMs and pose generative models Lucas et al. (2022), we can train or fine-tune them with large-scale skeleton data to build a skeletal foundation model that supports using prompts ($i.e.,$ textual user interaction) for skeleton attribute generation ($e.g.,$ analyzing and summarizing gait attributes), skeleton augmentation, prediction, classification, and customizable applications ($e.g.,$ gait visualization, SRID). Constructing this foundational model is advantageous for investigating the scope of adaptability, universality, and interpretability in 3D skeleton data and SRID.

Privacy Protection of SRID. Although existing SRID models do not utilize or disclose human appearance information, and all publicly-available training skeleton data are completely anonymized, the privacy issue should be kept in mind when developing this emerging technology further ($e.g.,$ combine with RGB images) Ye et al. (2024). As illegally or irresponsibly deploying person re-ID technologies might invade personal privacy, it is still important to establish SRID-relevant laws to protect the privacy.

Future Real-World Applications. (1) Mobile Pattern Recognition: With smaller input data size and lower resource requirement than RGB-based models, SRID models can be promisingly integrated to mobile devices to combine different sensor data ($e.g.,$ camera images) to jointly perform identity-aware action recognition, motion prediction, and gesture recognition tasks. (2) Gait-based Medical Diagnosis: As gait is one of the most important features used for SRID, the pre-trained model can be potentially transferred to different gait classification tasks. A promising direction is to apply SRID models to medical areas to assist in identifying patients and their abnormal gait ($e.g.,$ Parkinsonian gait). (3) Criminal Tracking: With good robustness to appearance changes and environmental variations Han et al. (2017), SRID models can be combined with other bioinformatic features ($e.g.,$ faces) or RGB-based methods Li et al. (2019); Lan et al. (2020) to track criminals and monitor their activities via skeletal poses under more challenging scenarios.