HiH: A Multi-modal Hierarchy in Hierarchy Network for
Unconstrained Gait Recognition

Single-modal gait recognition methods primarily leverage two modalities of input data: appearance-based modalities like silhouettes [5, 9, 27, 7, 53, 8, 46] and model-based modalities like skeletons [1, 41, 42, 13] and 3D meshes [14]. Appearance-based approaches directly extract gait features from raw input data. Earlier template-based methods such as the Gait Energy Image (GEI) [15] and Gait Entropy Image (GEnI) [2] create distinct gait templates by aggregating silhouette information over gait cycles. These techniques compactly represent gait signatures while losing temporal details and being sensitive to viewpoint changes.

Recent silhouette-based methods have excelled by focusing on structural feature learning and temporal modeling. For structure, set-based methods like GaitSet [5] and Set Residual Network [19] treat sequences as unordered sets, enhancing robustness to frame permutation. GaitPart [9] emphasizes unique expressions of different body parts, with 3D Local CNN [22] extracting part features variably. GaitGL [27] and HSTL [46] integrate local and global cues, though HSTL’s pre-defined hierarchical body partitioning may limit its adaptability. Temporally, methods like Contextual relationships [21], second-order motion patterns [4], and meta attention and pooling [7] discern subtle patterns, with advanced techniques exploring dynamic mechanisms [29, 48] and counterfactual intervention learning [8] for robust spatio-temporal signatures. To harness the color and texture information in the original images, recent RGB-based gait recognition techniques aim to directly extract gait features from video frames, mitigating reliance on preprocessing like segmentation [37, 52, 24, 25].

Model-based approaches build gait representations of body joints or 3D structure, then extract features and classify. Recent approaches utilize pose estimation advances to obtain cleaner skeleton input representing joint configurations[41, 13]. Graph convolutional networks help model inherent spatial-temporal patterns among joints [14, 42]. Some techniques incorporate biomechanical or physics priors to learn gait features aligned with human locomotion [26, 14]. Additionally, 3D mesh recovery from video has been explored for pose and shape modeling [50].

Many recent approaches fuse complementary modalities like silhouette, 2D/3D pose, and skeleton to obtain more comprehensive gait representations. TransGait [23] combines silhouette appearance and pose dynamics via a set transformer model. SMPLGait [54] introduces a dual-branch network leveraging estimated 3D body models to recover detailed shape and motion patterns lost in 2D projections. Other works focus on effective fusion techniques, including part-based alignment [6, 32, 20] and refining skeleton with silhouette cues [56]. While fusing modalities like silhouette and pose has demonstrated performance gains in controlled settings, their effectiveness decreases on outdoor benchmarks. This is partly due to inaccurate skeleton pose estimation under unconstrained conditions, which causes difficulty in modality alignment. Unlike existing works, our approach utilizes more reliable 2D joint sequences to apply per-frame spatial-temporal attention correction to the silhouettes, achieving greater consistency across different modalities.

The core of our proposed HiH framework integrates gait silhouettes with pose data to enhance gait recognition. As illustrated in Fig. 2, the framework operates on two input sequences: the silhouette sequence $X_{\text{sil}}\in\mathbb{R}^{C\times T\times H\times W}$ and the pose sequence $X_{\text{pose}}\in\mathbb{R}^{C\times T\times H\times W}$, where $C=1$ for both binary silhouette images and 2D keypoint-based pose representations. $T$ is the number of frames, and $H\times W$ denotes the spatial dimensions.

Building on the input sequences, the $\mathrm{HiH}$ framework utilizes a dual-branch backbone. The main branch processes $X_{\text{sil }}$ for general motion extraction. The initial step involves a 3D convolutional operation to extract foundational spatio-temporal features. This is followed by a series of stage-specific Hierarchical Gait Decomposer (HGD) modules, denoted as $\mathcal{H}_{i}$ for the $i$-th stage. The auxiliary branch leverages $X_{\text{pose }}$ to provide spatial and temporal guidance $\mathcal{G}\left(X_{\text{pose }}\right)$ to the HGD, via either Deformable Spatial Enhancement (DSE) or Deformable Temporal Alignment (DTA) modules. Thus, the output feature $F_{i}$ from each stage $\mathcal{H}_{i}$ is expressed as:

$$F_{i}=\mathcal{H}_{i}\left(X_{\text{sil }},\mathcal{G}\left(X_{\text{pose }}\right)\right).$$

(1)

Following the backbone, our framework applies Temporal Pooling (TP) [27] and Horizontal Pooling (HP) [9] to downsample the spatio-temporal dimensions. The reduced features are then processed through the head layer, which includes separate fully-connected layers and BNNeck [28], effectively mapping them into a metric space. The model is optimized using separate triplet $\mathcal{L}_{\text{tri}}$ and cross-entropy $\mathcal{L}_{\text{ce}}$ losses.

Unlike part-based techniques that typically divide the body into uniform horizontal segments [9, 27], the HGD employs a dual hierarchical approach for gait recognition, as shown in Fig. 2. This approach achieves a depth-wise hierarchy by stacking multiple HGD stages, which captures the gait dynamics from global body movements down to subtle limb articulations. In parallel, the width-wise hierarchy within each HGD stage conducts multi-scale processing to capture a comprehensive set of spatial features. The implementation of the $i$-th stage of the HGD, as illustrated in Fig. 3, can be formalized as follows:

$$F_{i}^{\prime}=\sum_{n=1}^{2^{i-1}}f_{3\times 1\times 1}\left(f_{1\times 3\times 3}^{(n)}\left(F_{i-1}\right)\right),$$

(2)

where $f_{1\times 3\times 3}^{(n)}$ denotes the convolution operation applied to $n$ horizontal strips of $F_{i-1}$ with a kernel size of $1\times 3\times 3$, designed to capture spatial features, while $f_{3\times 1\times 1}$ refines these features over the temporal dimension. Building on this multi-scale feature extraction, the aggregated features $F_{i}^{\prime}$ are further processed through a combination of additional convolutions and a residual connection [16] by

$$F_{i}=f_{3\times 1\times 1}\left(f_{1\times 3\times 3}\left(F_{i}^{\prime}\right)\right)+F_{i-1}.$$

(3)

Silhouettes offer overall shape of gait descriptions but lack structural details. Fusing poses can provide complementary information on joint and limb movements. To achieve this, inspired by [43, 49], we introduce the Deformable Spatial Enhancement (DSE) module to adapt silhouettes using derived pose cues. As shown in Fig. 4, DSE utilizes learned deformable offsets to dynamically warp input silhouettes, emphasizing key spatial gait features and aligning them to corresponding poses. This forms a Spatially Enhanced HGD (SE-HGD) for more discriminative gait analysis.

Within the DSE, offsets are learned from pose input $X_{\text{pose}}$ using a $3\times 3$ convolutional layer. The offsets are organized into a tensor $\mathcal{O}_{\text{DSE}}\in\mathbb{R}^{3\times H\times W}$, prescribing spatial transformations for the silhouette input $X_{\text{sil}}$. $\mathcal{O}_{\text{DSE}}$ contains two components: offsets $\mathcal{O}^{xy}_{\text{DSE}}\in\mathbb{R}^{2\times H\times W}$ representing pixel displacements in $x$ and $y$ directions, constrained by $\tanh$ activation, and offsets $\mathcal{O}_{\text{DSE-s}}^{xy}\in\mathbb{R}^{1\times H\times W}$ as scaling factors, processed via ReLU activation. The pixel-wise update to the silhouette input, utilizing the learned offsets, is conducted as follows:

$$X_{\text{DSE}}=\operatorname{BI}\left(X_{\text{sil }},\tanh\left(\mathcal{O}_{\text{DSE}}^{xy}\right)\odot\operatorname{ReLU}\left(\mathcal{O}_{\text{DSE-s}}^{xy}\right)\right),$$

(4)

where $\mathrm{BI}$ denotes the bilinear interpolation function that applies the spatial adjustments and scaling to $X_{\text{sil }}$, and $\odot$ represents element-wise multiplication.

While silhouette sequences provide a outline of basic body movement, they typically fail to capture the intricate joint dynamics. Building on DSE, the proposed Deformable Temporal Alignment (DTA) module extends pose guidance to the temporal domain, adaptively aligning silhouettes to match gait variations. Combined with HGD, DTA constitutes the Temporally Enhanced HGD (TE-HGD). Moreover, DTA enables per-pixel sampling between frames. This makes temporal downsampling via fixed-stride max pooling more adaptive to motion variations.

As illustrated in Fig. 5, the DTA module employs a 3D convolution $f_{3\times 3\times 3}$ on $X_{\text{pose}}$ to extract 5-channel spatio-temporal offsets $\mathcal{O}_{\text{DTA}}\in\mathbb{R}^{5\times T\times H\times W}$. The first two channels, $\mathcal{O}_{\text{DTA}}^{xy}$, capture spatial displacements in $x$ and $y$ axes, while the third, $\mathcal{O}_{\text{DTA}}^{z}$, quantifies temporal displacement. The fourth channel, $\mathcal{O}_{\text{DTA-s}}^{xy}$, scales spatial offsets, and the fifth, $\mathcal{O}_{\text{DTA-s}}^{z}$, adjusts temporal offsets. Similar to the DSE module, $\mathcal{O}_{\text{DTA}}^{\prime}$ is formed by concatenating the processed offsets. This can be expressed as:

	$\displaystyle\mathcal{O}_{\text{DTA}}^{\prime}=$	$\displaystyle\operatorname{concat}\left(\tanh\left(\mathcal{O}_{\text{DTA}}^{xy}\right)\odot\operatorname{ReLU}\left(\mathcal{O}_{\text{DTA-s}}^{xy}\right),\right.$		(5)
		$\displaystyle\hskip 28.45274pt\left.\tanh\left(\mathcal{O}_{\text{DTA}}^{z}\right)\odot\operatorname{ReLU}\left(\mathcal{O}_{\text{DTA-s}}^{z}\right)\right).$

These modified offsets $\mathcal{O}_{\mathrm{DTA}}^{\prime}$ guide the update of the silhouette features, followed by a MaxPooling operation to reduce redundancy. This process can be formulated as:

$$X_{\mathrm{DTA}}=\operatorname{MaxPool}\left(\operatorname{TI}\left(X_{\mathrm{sil}},\mathcal{O}_{\mathrm{DTA}}^{\prime}\right)\right),$$

(6)

where $\mathrm{TI}$ represents trilinear interpolation for spatio-temporal adjustments of the silhouette sequence, and MaxPool is applied along the temporal dimension.

To effectively train our model, we use the joint losses which include triplet loss $\mathcal{L}_{\text{tri}}$ [17, 29] and cross-entroy loss $\mathcal{L}_{\text{ce}}$ . The $\mathcal{L}_{\text{tri}}$ can be formulated as:

$$\begin{gathered}\mathcal{L}_{\text{tri}}=\frac{1}{N_{\text{tri}}}\overbrace{\sum_{u=1}^{U}}^{\text{stripes}}\overbrace{\sum_{i=1}^{P}\sum_{a=1}^{K}}^{\text{anchors}}\overbrace{\sum_{\begin{subarray}{c}p=1\\ p\neq a\end{subarray}}^{K}}^{\text{positives}}\overbrace{\sum_{\begin{subarray}{c}j=1,\\ j\neq i\end{subarray}}^{P}\sum_{n=1}^{K}}^{\text{negatives}}\left[m+\right.\\ \left.d\left(\phi\left(x_{i,a}^{u}\right),\phi\left(x_{i,p}^{u}\right)\right)-d\left(\phi\left(x_{i,a}^{u}\right),\phi\left(x_{j,n}^{u}\right)\right)\right]_{+},\end{gathered}$$

(7)

where $N_{\text{tri}}$ represents the number of triplets with a positive loss, $U$ is the number of horizontal stripes, $P$ and $K$ are the number of subjects and sequences per subject, respectively, $\left[\gamma\right]_{+}$ equals to $\max\left(\gamma,0\right)$, $m$ is the margin, $d$ denotes the euclidean distance, and $\phi$ is the feature extraction function. The variables $x_{i,a}^{u},x_{i,p}^{u}$, and $x_{j,n}^{u}$ represent the input sequences from anchors, positives, and negatives within the batch, respectively.

The $\mathcal{L}_{\text{ce}}$ can be express as:

$$\mathcal{L}_{\text{ce}}=-\frac{1}{P\times K}\overbrace{\sum_{i=1}^{P}\sum_{j=1}^{K}}^{\text{batch}}\overbrace{\sum_{n=1}^{N}}^{\text{subjects}}q_{i,j}^{n}\log\left(p_{i,j}^{n}\right),$$

(8)

where $N$ is the number of subject categories. In this formulation, $p_{i,j}^{n}$ denotes the predicted probability that the $j$-th sequence of the $i$-th subject in the batch belongs to the $n$-th category, and $q_{i,j}^{n}$ is the ground truth label. Finally, combining Eq. 7 and Eq. 8, the joint loss function $\mathcal{L}_{\text{joint}}$ can be formulated as:

$$\mathcal{L}_{\text{joint}}=\mathcal{L}_{\text{tri}}+\mathcal{L}_{\text{ce}}.$$

(9)

We evaluated our method across four gait recognition datasets: Gait3D [54] and GREW [57] from real-world environments, and OUMVLP [40] and CASIA-B [51] from controlled laboratory settings .

Training details. 1) The margin $m$ in Eq. 7 is set to 0.2, and the number of HP bins is 16; 2) Batch sizes are $(8,8)$ for CASIA-B, $(32,8)$ for OUMVLP, and $(32,4)$ for Gait3D and GREW; 3) Our input modalities include gait silhouettes and pose heatmaps generated by HRNet [39], both resized to $64\times 44$, with a fixed $\sigma$ of 2 for the 2D Gaussian distribution on keypoints. For training CASIA-B and OUMVLP, 30 frames are randomly sampled. For Gait3D, the frame range is $[10,50]$, and for GREW, it is $[20,40]$, following [10, 29]; 4) The optimizer is SGD with a learning rate of 0.1 , training the model for 60K, 140K, 70K, and 200K iterations for CASIA-B, OUMVLP, Gait3D, and GREW, respectively; 5) In the training phase for two real-world datasets, data augmentation strategies (e.g., horizontal flipping, rotation, perspective) are applied as outlined in [10, 30].

We conduct comprehensive comparisons between our proposed HiH method variants, HiH-S (using only silhouette modality) and HiH-M (integrating silhouettes with 2D keypoints for multimodal analysis), and three types of existing gait recognition methods: 1) Pose-based methods including GaitGraph2 [42], PAA [14], and GPGait [13]; 2) Silhouette-based methods such as GaitSet [5], GaitPart [9], GLN [18], GaitGL [27], 3D Local [22], CSTL [21], LagrangeGait [4], MetaGait [7], GaitBase [10], DANet [29], GaitGCI [8], STANet [30], DyGait [48], and HSTL [46]; 3) Multimodal approaches like SMPLGait [54], TransGait [23], BiFusion [32], GaitTAKE [20], GaitRef [56], and MMGaitFormer [6].

To validate the efficacy of each component in HiH, including HGD which provides hierarchical feature learning in depth and width, DSE and DTA for pose-guided spatial-temporal modeling, we conduct ablation studies the Gait3D dataset with results in Tab. 6. The hierarchical depth and width provided by the HGD module lay a foundational baseline for our approach. Adding either DSE or DTA can further improve over this baseline. The best results are achieved when all modules are considered together.

We visualize the heatmaps from the last layer of HiH and other methods on Gait3D in Fig. 6. It can be observed that HSTL focuses on sparse key joints like knees, shoulders and arms, with attention on limited body parts. In comparison, GaitBase attends to more body parts within each silhouette, explaining its superior performance over HSTL. Our HiH-S further outputs denser discriminative regions across multiple views, hence achieving better results. By incorporating pose modality, HiH-M obtains more comprehensive coverage of full-body motion areas.

In our trade-off analysis, shown in Fig. 7, we explore the relationship between model complexity and accuracy. Pose-based methods like GPGait [13] demonstrate parameter efficiency but fall short in performance. Clearly, larger models with higher FLOPs (Floating Point Operations per Second) generally achieve better results. DyGait [48], achieving high accuracy, demands significant computation due to 3D convolutions. In contrast, GaitBase [10] offers better parameter efficiency by utilizing 2D spatial convolutions, but incurs higher FLOPs, likely due to the lack of effective temporal aggregation mechanisms. Our HiH-S finds an optimal balance. HiH-M further enhances performance without substantially increasing computational cost. This is accomplished by using only one 2D and one 3D convolution to extract spatial and temporal dependencies from poses.